Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
MarcinSV
Helper I
Helper I

Refresh in PRO/A1 service much slower then sql procedure

Hi all,

 

I have PBI report with 10 000 000 of rows, 300GB. For tests (to exclude relations measures, conditional colmns etc),  I left only one raw main table (DataSource) which fills by sql procedure (Source in Edit query -> exec procedure DataSource.

 

When I exec this procedure directly on SQL it takes abut 4 minutes (10mln of records).

When I refresh report in PBI service, using gateway it takes about 2 hrs (seen in history of refreshing).

When I refresh report in Embeded A1 - it takes 2hrs, the same like in pro space.

I know that in service will be slower (transfer data) but so many times?

Its strange because this rather big report seems to be unused in service as is (pro or A1 in embeded)?

I don't understand where I should to look for?

Any help?

1 ACCEPTED SOLUTION
v-xicai
Community Support
Community Support

Hi @MarcinSV ,

 

When the refresh is in fact slow or crash, it can be due to several reasons:

  • Insufficient CPU (refresh can be very CPU-intensive).
  • Insufficient memory, resulting in refresh pausing (which requires the refresh to start over when conditions are favorable to recommence).
  • Non-capacity reasons, including data source system responsiveness, network latency, invalid permissions or gateway throughput. You may increase the capacity for workspaces to increase the model refresh parallelism frequency.
  • Data volume - a good reason to configure incremental refresh, Incremental refresh can significantly reduce data refresh duration, especially for large model tables. See more: Incremental refresh in Power BI Premium .

 

See more details: https://docs.microsoft.com/en-us/power-bi/whitepaper-powerbi-premium-deployment#why-are-refreshes-sl....

 

If you use DirectQuery mode to connect data, your report performance depends largely on the performance of the underlying data source.

 

You can optimize your data model using following tips:

 

  • Remove unused tables or columns, where possible. 
  • Avoid distinct counts on fields with high cardinality – that is, millions of distinct values.  
  • Take steps to avoid fields with unnecessary precision and high cardinality. For example, you could split highly unique datetime values into separate columns – for example, month, year, date, and so on. Or, where possible, use rounding on high-precision fields to lower cardinality – (for example, 13.29889 -> 13.3).
  • Use integers instead of strings, where possible.
  • Be wary of DAX functions, which need to test every row in a table – for example, RANKX – in the worst case, these functions can exponentially increase run-time and memory requirements given linear increases in table size.
  • When connecting to data sources via DirectQuery, consider indexing columns that are commonly filtered or sliced again. Indexing greatly improves report responsiveness.  
  • Enable Row-Level Security (RLS) where applicable.
  • Use Microsoft AppSource certified custom visuals where applicable.
  • Do not use hierarchical filters.
  • Provide data categorization for Power BI reports (HBI, MBI, LBI).
  • Use the On-premises data gateway instead of Personal Gateway.
  • Use slicers sparingly.

 

For more information on optimizing data sources for DirectQuery, see DirectQuery in SQL Server 2016 Analysis Services.

 

To minimize the impact of network latency, strive to keep data sources, gateways, and your Power BI cluster as close as possible. If network latency is an issue, try locating gateways and data sources closer to your Power BI cluster by placing them on virtual machines.

To further improve network latency, consider using Azure ExpressRoute, which is able of creating faster, more reliable network connections between your clients and Azure datacenters.

 

You can learn more via the link: https://docs.microsoft.com/en-us/power-bi/power-bi-reports-performance#optimize-your-model.

 

Best Regards,

Amy

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

3 REPLIES 3
v-xicai
Community Support
Community Support

Hi @MarcinSV ,

 

When the refresh is in fact slow or crash, it can be due to several reasons:

  • Insufficient CPU (refresh can be very CPU-intensive).
  • Insufficient memory, resulting in refresh pausing (which requires the refresh to start over when conditions are favorable to recommence).
  • Non-capacity reasons, including data source system responsiveness, network latency, invalid permissions or gateway throughput. You may increase the capacity for workspaces to increase the model refresh parallelism frequency.
  • Data volume - a good reason to configure incremental refresh, Incremental refresh can significantly reduce data refresh duration, especially for large model tables. See more: Incremental refresh in Power BI Premium .

 

See more details: https://docs.microsoft.com/en-us/power-bi/whitepaper-powerbi-premium-deployment#why-are-refreshes-sl....

 

If you use DirectQuery mode to connect data, your report performance depends largely on the performance of the underlying data source.

 

You can optimize your data model using following tips:

 

  • Remove unused tables or columns, where possible. 
  • Avoid distinct counts on fields with high cardinality – that is, millions of distinct values.  
  • Take steps to avoid fields with unnecessary precision and high cardinality. For example, you could split highly unique datetime values into separate columns – for example, month, year, date, and so on. Or, where possible, use rounding on high-precision fields to lower cardinality – (for example, 13.29889 -> 13.3).
  • Use integers instead of strings, where possible.
  • Be wary of DAX functions, which need to test every row in a table – for example, RANKX – in the worst case, these functions can exponentially increase run-time and memory requirements given linear increases in table size.
  • When connecting to data sources via DirectQuery, consider indexing columns that are commonly filtered or sliced again. Indexing greatly improves report responsiveness.  
  • Enable Row-Level Security (RLS) where applicable.
  • Use Microsoft AppSource certified custom visuals where applicable.
  • Do not use hierarchical filters.
  • Provide data categorization for Power BI reports (HBI, MBI, LBI).
  • Use the On-premises data gateway instead of Personal Gateway.
  • Use slicers sparingly.

 

For more information on optimizing data sources for DirectQuery, see DirectQuery in SQL Server 2016 Analysis Services.

 

To minimize the impact of network latency, strive to keep data sources, gateways, and your Power BI cluster as close as possible. If network latency is an issue, try locating gateways and data sources closer to your Power BI cluster by placing them on virtual machines.

To further improve network latency, consider using Azure ExpressRoute, which is able of creating faster, more reliable network connections between your clients and Azure datacenters.

 

You can learn more via the link: https://docs.microsoft.com/en-us/power-bi/power-bi-reports-performance#optimize-your-model.

 

Best Regards,

Amy

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Thank you very much for detailed information!

I have one more question you don't mind, which I don't understand. My laptop is in LAN with SQL from where I take data. Refreshing on my laptop (desktop PBI) takes 1-2hrs. During this I've checked performance my laptop - 50% using RAM, 20% CPU. When I exec procedure exactly on SQL server - it takes 2-4 min. I don't understand why in local there are so huge differents.

 

MarcinSV
Helper I
Helper I

"I have PBI report with 10 000 000 of rows, 300GB." - sorry, should to be 300MB

Any ideas?

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors