Implement Power BI within Organization - Azure Data Lake

[andra2]

Hi, 

I need to implement Power BI as reporting tool in my organization that will use Azure Data Lake as source data, but I have the following questions:

- having more than 12TB as data and 10000 columns, which is the approach you are advising us to use? Create a layer for connecting to the data like Azure SQL or Azure Synapse? I am intersted more in terms of performance and cost. If you have some tips and tricks for implementing Power BI using Azure as source data, I would appreciate a lot. 

- beside the best practices that I can find on the website, can you provide me some other having in mind the context I've provided above?

- how you are choosing the proper Power BI license?

Many thanks!

[v-xiaosun-msft]

Hi @andra2 ，

According to your description, here is my suggestion.

First, if you do not have column headers, you will have to add in column names later.

Second, the format of the files being in comma or at least delimited format can be a big help.

You can bring up your Azure Data Lake instance in the Azure Administrative portal. And then connect to data with Power BI Desktop.

Cloud sources are limited to 1 million rows, on-premises sources are limited to about 4 MB of defined load per row (depending on specialized compression algorithms) or 16MB of data size for the entire visual. Some limits may increase when using Premium capacity. This restriction does not affect aggregations or calculations used to create datasets returned using Direct Query. It affects only the rows that are returned. Premium capacity can set the maximum row limit.

If over 1 million rows are returned from Direct Query, Power BI returns an error (unless in Premium capacity, and the row count is under the admin-set limit).

Please refer to the following documents.

Using Azure Data Lake with Power BI - 5MinuteBI

Using DirectQuery in Power BI - Power BI | Microsoft Learn

Azure Data Lake Storage Gen2 - Power Query | Microsoft Learn

Best Regards,
Community Support Team _ xiaosun

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[TomMartens]

Hey @andra2 ,

this article provides a good start into using a large amount of data in combination with Power BI and some other Azure technologies.

Creating a Databricks SQL Dashboard to Analyze NYC Taxi Data - Microsoft Tech Community

Regards,

Tom

[andra2]

Many thanks, Tom, I have seen already this article, but I am more interested in finding out the feedback of someone that has implemented this already in a organization, to understand the challenges they were facing etc and which were the alternatives they were considering when they have chosen that particular source