cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
rexii23 Advocate I
Advocate I

incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

We have Premium and I've setup a dataflow that grabs data from an SQL DB. One of the entities with 30+ million rows of data has been configured for incremental refresh.

 

The incremental refresh is working as expected and the dataflow refreshes in ~2minutes.

 

However, now when the dataset that's connected to the dataflow refreshes, it is taking around 30minutes to refresh as it is essentially loading ALL the partitioned flat files off ADLS and there is no Query Folding at all.

 

I understand that my dataflow is much quicker to refresh but what is the point if my dataset will still take forever to refresh? I can of course just connect to the SQL DB directly but then what would be the point of using dataflows? I've implemented dataflows as the dataflow can be utilised by others so there is no need to hit the DB anymore.

 

Can anyone help me understand this?

5 REPLIES 5
Highlighted
Community Support
Community Support

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Hi @rexii23 ,

 

You may refer to the documents:

https://docs.microsoft.com/en-us/power-bi/service-dataflows-overview,

https://docs.microsoft.com/en-us/power-bi/service-dataflows-create-use,

https://docs.microsoft.com/en-us/power-bi/service-dataflows-incremental-refresh.

 

Best Regards,

Amy

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

rexii23 Advocate I
Advocate I

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Hi Amy

 

I've already read through the documentation. I do not see the scenario I outlined covered them.

keerthig21
Frequent Visitor

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Hi,

 

We have the same issue with incremental refresh from dataflow to dataset. Please let me know if you found any solution.

 

Thanks,

Keerthi

TheBeeGuy
Frequent Visitor

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

I'm running into the same problem.  Original source is SQL DB that gets loaded into dataflow with incremental refresh (2 to 3 minutes to refresh).  The incremental refresh works great for this step.  Next step is consuming the dataflow entity into a dataset with incremental refresh.  This is terribly slow (1 hour to refresh).  I'm considering removing the dataflow from the equation.  How have others handled this situation?

mnorstedt
Frequent Visitor

Re: incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

Digging up an old topic here but I have the same issue. I love the idea of shared and incremental refreshed dataflows, but not at the expense of exponential dataset refresh times. In my case I have a dataset that takes about 6-8 mins to refresh using direct SQL but almost 30 mins when calling the dataflows that contain the same logic.

 

Has anyone found a fix for this? I'm about ready to ditch the dataflows for this project, which would be shame because they have so much promise but 30min refreshes for no reason are hard to sell to the business.

Helpful resources

Announcements
Announcing the New Spanish Forum

Announcing the New Spanish Forum

Do you need help in Spanish? Check out our new Spanish community section.

MBAS Gallery 2020

MBAS Gallery 2020

Watch Microsoft Business Applications Summit sessions on-demand.

‘Better Together’ Integration Forum Launch

‘Better Together’ Integration Forum Launch

We've launched a how-to forum where you can learn about how Power BI integrates with other Power Platform products.