Unable to load data with Azure Data Lake Gen 2

[Enileda3]

I am working with 30+ Million rows data, loading from azure data lake gen 2. It is loading 160 csv files in total and I have used #"filtered rows in power query to filter rows which are not required. In addition, i have removed columns not required too. The final rows that should be loaded would be less than a million rows.

However, my data load was loading for more than 2 hours and around 12GB. It looks like it is still loading but i waited for more than half an hour and it didnt to past 12GB. Tried many different ways a few time and each time is the same, didnt go past 10-12GB.

seems like the data filter written in power query is not applied at the source before loading into power bi. Does feels like power bi is loading the full data before filtering.

Have search and found most answer are on using advanced query for SQL. Any advise on how to get around this with Azure Data Lake Gen 2?

[v-kkf-msft]

Hi @Enileda3 ,

Power BI can apply filters to the list obtained by the File System View option, thus restricting the access to only those files you are interested in. However, every time you refresh the data source, the list of all these files is read by Power Query. A better way to manage the process is to get the exact path of the file.

For more information, please refer to Optimizing access to Azure Data Lake Storage (ADLS) Gen 2 in Power Query 

If you can use Dremio, Direct Query mode is a good way. Data Lake Analytics with Dremio and Power BI on ADLS Gen2 

If the problem is still not resolved, please provide detailed error information or the expected result you expect. Let me know immediately, looking forward to your reply.

Best Regards,
Winniz

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.