cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Enileda3
Frequent Visitor

Unable to load data with Azure Data Lake Gen 2

I am working with 30+ Million rows data, loading from azure data lake gen 2. It is loading 160 csv files in total and I have used #"filtered rows in power query to filter rows which are not required. In addition, i have removed columns not required too. The final rows that should be loaded would be less than a million rows.

 

However, my data load was loading for more than 2 hours and around 12GB. It looks like it is still loading but i waited for more than half an hour and it didnt to past 12GB. Tried many different ways a few time and each time is the same, didnt go past 10-12GB.

 

seems like the data filter written in power query is not applied at the source before loading into power bi. Does feels like power bi is loading the full data before filtering.

 

Have search and found most answer are on using advanced query for SQL. Any advise on how to get around this with Azure Data Lake Gen 2?



1 REPLY 1
v-kkf-msft
Community Support
Community Support

Hi @Enileda3 ,

 

Power BI can apply filters to the list obtained by the File System View option, thus restricting the access to only those files you are interested in. However, every time you refresh the data source, the list of all these files is read by Power Query. A better way to manage the process is to get the exact path of the file.

For more information, please refer to Optimizing access to Azure Data Lake Storage (ADLS) Gen 2 in Power Query 

 

If you can use Dremio, Direct Query mode is a good way. Data Lake Analytics with Dremio and Power BI on ADLS Gen2 

 

 

If the problem is still not resolved, please provide detailed error information or the expected result you expect. Let me know immediately, looking forward to your reply.

Best Regards,
Winniz

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
PBI_User Group Leader_768x460.jpg

Manage your user group events

Check out the News & Announcements to learn more.

MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

Get Ready for Power BI Dev Camp

Microsoft named a Leader in The Forrester Wave

Microsoft received the highest score of any vendor in both the strategy and current offering categories.

R2 (Green) 768 x 460px.png

Microsoft Dynamics 365 & Power Platform User Professionals

DynamicsCon is a FREE, 4 half-day virtual learning experience for 11,000+ Microsoft Business Application users and professionals.