Excel Database

[gdimmers]

I have a folder with csv files that I use as a datasource for a powerbi file. The dataset represents about 12M rows of data. A new csv file is added daily and once added the files are static. It works ok, but when i want to make changes or refresh the file it takes 30-60mins presumably because it reloads all the data. I am trying to find if there is a more effcient way to store these data files and connect with BI that makes the file easier to work with.

Thanks.

[gdimmers]

Thanks for the response. I have been going down the route of separating the database and the reports like you suggested. It works ok just having to get used to some of the limitations with the live connection...

[Ritaf1983]

Glad to help 🙂
If this post helps, then please consider Accepting it as the solution to help the other members find it more quickly

[Ritaf1983 · Accepted solution]

Hi @gdimmers 
An incremental refresh is really challenging to deal with CSV because it not availible for query folding.
Please refer to this article, I believe it can be of assistance.

https://pbi-guy.com/2022/07/14/how-to-use-incremental-refresh-on-any-data-source/

Another alternative to avoid dealing with refresh problems in the desktop is to separate the report from the data set.
To publish data sets to the service and then prepare reports from the desktop using the published model as a data source:

If this post helps, then please consider Accepting it as the solution to help the other members find it more quickly