Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
Anonymous
Not applicable

Maximum performance with large datasets on Azure

Hi all,

 

I am trying to achieve the best performance with large datasets in reports. For example: I have reports that use some visuals with aggregated data, but my users also want to get the details. These details are coming from datasets containing 100M+ rows.

 

Importing this into PowerBI is ofcourse not an option, so I came up with the following ideas:

 

  1. Use Azure Analysis Services
    Putting the datasets in tabular models will consume a lot of memory however, this requires spending a lot on a higher ranged tier (S2 or above, €2,500 per month or higher)

  2. Or use Azure SQL Data Warehouse with DirectQuery
    Did some testing with the basic tier DW100c (€900 per month) and I achieved a performance of 10 seconds searching a 100M rows table. The report only contained a search box and a table. The dataset in SQLDWH was hash distributed and had a columnstore index.
    What I noticed however is that the query PowerBI creates is far more complex than a simple 'SELECT xx,xx FROM WHERE xx = xx' (this executes in 1 second by the way on the server itself). Power BI for example creates some joining, grouping, ordering etc, which are not really necessary for my use-case. Is there a way to optimize this?
  3. Or use a Databricks cluster with Spark and DirectQuery
    Also an option but more complex and a continuous running cluster will also cost at least €730 per month.

I am wondering what the best option is, does anyone have any experience with these options already? Other Azure options are also welcome!

I tend to use SQL DWH though, but can I further optimize this without scaling up?

 

Thanks!

 

1 REPLY 1
parry2k
Super User
Super User

@Anonymous did you explored composite model?



Subscribe to the @PowerBIHowTo YT channel for an upcoming video on List and Record functions in Power Query!!

Learn Power BI and Fabric - subscribe to our YT channel - Click here: @PowerBIHowTo

If my solution proved useful, I'd be delighted to receive Kudos. When you put effort into asking a question, it's equally thoughtful to acknowledge and give Kudos to the individual who helped you solve the problem. It's a small gesture that shows appreciation and encouragement! ❤


Did I answer your question? Mark my post as a solution. Proud to be a Super User! Appreciate your Kudos 🙂
Feel free to email me with any of your BI needs.

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.