Stop reading all text files again

[efilipe]

Hello!

I have a directory with about 3 thousand text files of 1kb each. I ask PowerBI to open each of them, get the header of the file, and then split by position - voilá.

In this directory new files are added everyday. But it will always have about 3 thousand files, because I have a routine that erases files older than an year.

The files are in the network, so access is slower than working local.

So, what I'd like, is in a way I dont have to update it manually, and that PowerBI doesnt have to read all the files again just because of those new files that came in today. 🙂

Any suggestion?

[v-xicai · Accepted solution]

Hi @efilipe ,

Assuming that you have uploaded your text files into SharePoint Online, then you may use SharePoint Folder connector to combine or consolidate these files into a single table, transform the data in Query Editor, like "get the header of the file, and then split by position - voilá", see how to get SharePoint Online data from the link: Combining Excel Files hosted on a SharePoint folder with Power BI .

Then for the updated data today you may set Incremental refresh , and the Incremental refreshes only data that has changed.  Since incremental refresh is Premium only feature, so you may need to purchase Power BI Premium capacity for your organization, see How to purchase Power BI Premium. Incremental refresh policies are defined in Power BI Desktop and applied when published to the Power BI service.  See more detail steps: Incremental refresh in Power BI Premium , Incremental refresh for files in a Folder or SharePoint – Power BI

Best Regards,

Amy 

Community Support Team _ Amy

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[amitchandak]

@efilipe , refer if this can help

https://blog.crossjoin.co.uk/2020/04/13/keep-the-existing-data-in-your-power-bi-dataset-and-add-new-...

https://insightsoftware.com/blog/power-bi-load-data-from-folder/

[efilipe]

Well, you know it's not that... 🙂