cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
rexii23 Helper I
Helper I

incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

We have Premium and I've setup a dataflow that grabs data from an SQL DB. One of the entities with 30+ million rows of data has been configured for incremental refresh.

 

The incremental refresh is working as expected and the dataflow refreshes in ~2minutes.

 

However, now when the dataset that's connected to the dataflow refreshes, it is taking around 30minutes to refresh as it is essentially loading ALL the partitioned flat files off ADLS and there is no Query Folding at all.

 

I understand that my dataflow is much quicker to refresh but what is the point if my dataset will still take forever to refresh? I can of course just connect to the SQL DB directly but then what would be the point of using dataflows? I've implemented dataflows as the dataflow can be utilised by others so there is no need to hit the DB anymore.

 

Can anyone help me understand this?

5 REPLIES 5
Community Support
Community Support

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Hi @rexii23 ,

 

You may refer to the documents:

https://docs.microsoft.com/en-us/power-bi/service-dataflows-overview,

https://docs.microsoft.com/en-us/power-bi/service-dataflows-create-use,

https://docs.microsoft.com/en-us/power-bi/service-dataflows-incremental-refresh.

 

Best Regards,

Amy

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

rexii23 Helper I
Helper I

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Hi Amy

 

I've already read through the documentation. I do not see the scenario I outlined covered them.

keerthig21
Frequent Visitor

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Hi,

 

We have the same issue with incremental refresh from dataflow to dataset. Please let me know if you found any solution.

 

Thanks,

Keerthi

TheBeeGuy
Frequent Visitor

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

I'm running into the same problem.  Original source is SQL DB that gets loaded into dataflow with incremental refresh (2 to 3 minutes to refresh).  The incremental refresh works great for this step.  Next step is consuming the dataflow entity into a dataset with incremental refresh.  This is terribly slow (1 hour to refresh).  I'm considering removing the dataflow from the equation.  How have others handled this situation?

mnorstedt
Frequent Visitor

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Digging up an old topic here but I have the same issue. I love the idea of shared and incremental refreshed dataflows, but not at the expense of exponential dataset refresh times. In my case I have a dataset that takes about 6-8 mins to refresh using direct SQL but almost 30 mins when calling the dataflows that contain the same logic.

 

Has anyone found a fix for this? I'm about ready to ditch the dataflows for this project, which would be shame because they have so much promise but 30min refreshes for no reason are hard to sell to the business.

Helpful resources

Announcements
New Ranks Launched March 24th!

New Ranks Launched March 24th!

The time has come: We are finally able to share more details on the brand-new ranks coming to the Power BI Community!

‘Better Together’ Contest Finalists Announced!

‘Better Together’ Contest Finalists Announced!

Congrats to the finalists of our ‘Better Together’-themed T-shirt design contest! Click for the top entries.

Arun 'Triple A' Event Video, Q&A, and Slides

Arun 'Triple A' Event Video, Q&A, and Slides

Missed the Arun 'Triple A' event or want to revisit it? We've got you covered! Check out the video, Q&A, and slides now.

Community Summit North America

Community Summit North America

Innovate, Collaborate, Grow. The top training and networking event across the globe for Microsoft Business Applications

Top Kudoed Authors