Too long to Refresh - OneDrive for Business

[hatahetahmad]

Hello PBI Community,

I have multiple Excel files (6 files) and Folder (Contain 15 or so XL files) on my OneDrive for Business,

then I import them to PQ using Sharepoint.Content.

- I have a Query "Connection" which establish the OneDrive connection and navigate to the destination Folder

- I reference this query then start mapping to the XLs and the Folder

- Then I make some transformation to my data

When I hit Refresh the Query I'll describe take too long to refresh and here is how I processed it:

- Reference the Connection Query

- Select the XLs inside the folder to combine them

- Combine the files and apply some transformation using custom function I made then expand the tables (query1)

** to this point I refresh less than a minute which is good, but the next steps made the difference**

- I start new query referencing the previous one filter some specific data from external list (query2)

- then append the result from this Query to the previous one to a new query to avoid circular reference error.(result)

This steps made my refresh time more than 2 minutes.

As I see when PQ show you the refresh process, only the new table that has the appended query take long.

It's like with each step connecting to OneDrive to get the data then continue to the next step.

I tried Table.Buffer to the result query1 but it made the refresh longer.

My internet connection is fast and the laptop I use is high end, so no hardware issues.

Any Ideas 🙂

[v-lionel-msft]

Hi @hatahetahmad ,

Maybe you can use this connector:

(Use the root URL of your OneDrive for Business.)

Then we can combine all the files in the OneDrive for business.

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[hatahetahmad]

Thank you for your comment.

I am using the root folder, like this thread 

https://radacad.com/power-bi-get-data-from-multiple-files-in-a-folder-on-onedrive-for-business-no-ga...

[aleayafer_]

I have also faced this problem due to the slowness of that connector and, for the time being, I have solved it as follows:

1. I use one parameter to direct the connection to the OneDrive with all the data or to another that contains very few rows.
2. I perform transformations in PQ by connecting to the OneDrive with few-row files to minimize slowness.
3. Once all the transformations have been done, I modify the parameter to point to the final OneDrive.
4. I'm in the cloud.
5. I'm upgrading from the cloud thereafter, as it's faster than the on-premises upgrade.
6. If I need to make new modifications to PQ I point again to the OneDrive with small files.

It's not the ideal solution, but it can help you while finding a better way.

Best regards.