cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
shrikant23
Regular Visitor

​power bi spark connector poor performance

​power bi spark connector performance is extremely slow. it is taking 4 to 5 hours to process 6 gigs azure databricks delta tables into power bi premium nodes (p3). Both power bi premium capacity and azure databricks workspace are in same azure data center. Even after adjusting maxresultset and batch size performance is poor. I exported the delta tables into CSV files on ADLS Gen2 and was able to process it in 3X faster. What is wrong with power bi spark connector, we heard from databricks team that microsoft is working on the fix. Any updates / workarounds? 

5 REPLIES 5
shrikant23
Regular Visitor

Dear Team, 

I have tested power bi refresh using new spark connector available in recent sept 2020 power desktop release (version 2.85.681.0). I see the significant improvement in one the small scale model (<1 gig). In the past using old connector - model refreshed in 14 min while current refresh completed within 4 min and 11 sec. Using synapse as source - same model is refreshing in 3 min and 4 sec. Can someone explain what was the problem in the past and it was resolved in the new connector?     

Anyways - thank you for excellent work to resolve the performance issue. 

 

v-easonf-msft
Community Support
Community Support

Hi, @shrikant23 
Is it caused by  an incompatibility between Databricks version 6.4 (and above) and connector used by Power BI service?

Take a try to downgrading  Databricks to 5.5 LTS.

 

Best Regards,
Community Support Team _ Eason

hello Team, 

Still no improvements. Do you have benchmark numbers for tpch dataset refresh from ADB Delta table to Premium capacity node? Also do you believe that we will have to live with expcation that spark connector is slow until its re-enginnered in the next iteration / fix or shall we open support ticket to check if this is enviornmental issue? 

 

Best Reagrds

Shrikant Kulkarni 

 

Hi , @shrikant23 

Sorry I can't fix it on my side currently.

It is recommended to create a support ticket to investigate this issue.Once you create a support ticket and get any workaround, please kindly share it here for reference.
https://powerbi.microsoft.com/en-us/support/

 

Best Regards,
Community Support Team _ Eason
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Thank you!! support ticket was raised that time and microsoft power bi support was involved. In fact we did reached out to Adam saxson as well during that time.     

Helpful resources

Announcements
UG GA Amplification 768x460.png

Launching new user group features

Learn how to create your own user groups today!

November Power BI Update 768x460.png

Check it Out!

Click here to read more about the November 2021 Updates!

M365 768x460.jpg

Microsoft 365 Collaboration Conference | December 7–9, 2021

Join us, in-person, December 7–9 in Las Vegas, for the largest gathering of the Microsoft community in the world.