cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
mdlm778
Frequent Visitor

Connect folder with hundreds of files: performance

Hi everybody,

 

I need your help to be sure to choose the best solution for what I have to do.

 

I have a folder on my hard drive.

Frequently (sometimes every day, sometimes every week, sometimes less), I add one or several excel files (xlsm) into that folder.

All the excel files have the same structure (around 20000 rows, 100 columns).

I created a Power BI file connected to the folder.

Each time I refresh the data, the data from the new files are integrated, sorted like I want, indexed like I want, great !

 

Question: one day, I will have hundreds and hundreds of files in the folder. Is it the best solution to proceed ? Or will it take too much time ? If so, is there another way ?

 

Thanks for your help,

Kind regards

4 REPLIES 4
jessi82315
Frequent Visitor

I have the same issue. looks like there's still no quick fix???

mdlm778
Frequent Visitor

Hi everyone,

 

I tried something this morning. In my folder connected to my Power BI file, I added 5 files, they were already 6 files, so when I clicked on refresh data, Power BI has refreshed the data for the 11 files, and it took too much time. But it is not needed for me as the 6 files already included will never be updated. So I tried to delete the files in the folder already imported in my Power BI file and then to import just one new file. Problem: all previous data has been erased.

 

Is there any way to update data with only new file(s) in the folder, not to refresh all data with all the files existing in the folder ?

 

Thank you for your help !

Hi mdlm778,

 

You may click File-> Options and settings-> Options. Then click the options like below to reduce some useless data loading:

1.PNG 

2.PNG 

 

Regards,

Jimmy Tao

Hi @v-yuta-msft

 

Thanks for your reply.

 

I will set my parameters as you said to improve performance. But is there a way (either manual or automatic) to :

- import file 1 into Table f1

- few days later, import file 2 into Table f2 + combine it to Table f1 in order to get MyAllTable + delete files 1 and 2 in the folder

- few days later, import file 3 and file 4 into Table f34 (or Table 3 and Table 4, whatever) + combine them to MyAllTable + delete files 3 and 4 in the folder

- ...

?

 

Because now I don't succeed if I delete the sources. The problem is that I want the users to update the data directly with only the new files added in the folder connected to the power BI file. But they won't wait for update of hundreds of files. That's why I think about deleting the old files once the data has been incorporated into MyAllTable. Is it possible ?

 

Thank you

 

 

Helpful resources

Announcements
MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

July 2021 Update 768x460.png

Check it out!

Click here to read more about the July 2021 Updates

Power Query PA Forum 768x460.png

Check it out!

Did you know that you can visit the Power Query Forum in Power BI and now Power Apps

Urdu Hindi D365 Bootcamp 768x460.png

Urdu Hindi D365 Bootcamp

Dont miss our very own April Dunnam’s The Developer Guide to the Galaxy! Find out what the Power Platform has to offer for the traditional developer.

Top Solution Authors
Top Kudoed Authors