Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
mdlm778
Frequent Visitor

Connect folder with hundreds of files: performance

Hi everybody,

 

I need your help to be sure to choose the best solution for what I have to do.

 

I have a folder on my hard drive.

Frequently (sometimes every day, sometimes every week, sometimes less), I add one or several excel files (xlsm) into that folder.

All the excel files have the same structure (around 20000 rows, 100 columns).

I created a Power BI file connected to the folder.

Each time I refresh the data, the data from the new files are integrated, sorted like I want, indexed like I want, great !

 

Question: one day, I will have hundreds and hundreds of files in the folder. Is it the best solution to proceed ? Or will it take too much time ? If so, is there another way ?

 

Thanks for your help,

Kind regards

4 REPLIES 4
jessi82315
Helper I
Helper I

I have the same issue. looks like there's still no quick fix???

mdlm778
Frequent Visitor

Hi everyone,

 

I tried something this morning. In my folder connected to my Power BI file, I added 5 files, they were already 6 files, so when I clicked on refresh data, Power BI has refreshed the data for the 11 files, and it took too much time. But it is not needed for me as the 6 files already included will never be updated. So I tried to delete the files in the folder already imported in my Power BI file and then to import just one new file. Problem: all previous data has been erased.

 

Is there any way to update data with only new file(s) in the folder, not to refresh all data with all the files existing in the folder ?

 

Thank you for your help !

Hi mdlm778,

 

You may click File-> Options and settings-> Options. Then click the options like below to reduce some useless data loading:

1.PNG 

2.PNG 

 

Regards,

Jimmy Tao

Hi @v-yuta-msft

 

Thanks for your reply.

 

I will set my parameters as you said to improve performance. But is there a way (either manual or automatic) to :

- import file 1 into Table f1

- few days later, import file 2 into Table f2 + combine it to Table f1 in order to get MyAllTable + delete files 1 and 2 in the folder

- few days later, import file 3 and file 4 into Table f34 (or Table 3 and Table 4, whatever) + combine them to MyAllTable + delete files 3 and 4 in the folder

- ...

?

 

Because now I don't succeed if I delete the sources. The problem is that I want the users to update the data directly with only the new files added in the folder connected to the power BI file. But they won't wait for update of hundreds of files. That's why I think about deleting the old files once the data has been incorporated into MyAllTable. Is it possible ?

 

Thank you

 

 

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.