Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
rexii23
Advocate I
Advocate I

incremental refresh of SQL DB->Dataflow->Dataset concept flawed?

We have Premium and I've setup a dataflow that grabs data from an SQL DB. One of the entities with 30+ million rows of data has been configured for incremental refresh.

 

The incremental refresh is working as expected and the dataflow refreshes in ~2minutes.

 

However, now when the dataset that's connected to the dataflow refreshes, it is taking around 30minutes to refresh as it is essentially loading ALL the partitioned flat files off ADLS and there is no Query Folding at all.

 

I understand that my dataflow is much quicker to refresh but what is the point if my dataset will still take forever to refresh? I can of course just connect to the SQL DB directly but then what would be the point of using dataflows? I've implemented dataflows as the dataflow can be utilised by others so there is no need to hit the DB anymore.

 

Can anyone help me understand this?

5 REPLIES 5
TheBeeGuy
Frequent Visitor

I'm running into the same problem.  Original source is SQL DB that gets loaded into dataflow with incremental refresh (2 to 3 minutes to refresh).  The incremental refresh works great for this step.  Next step is consuming the dataflow entity into a dataset with incremental refresh.  This is terribly slow (1 hour to refresh).  I'm considering removing the dataflow from the equation.  How have others handled this situation?

Anonymous
Not applicable

Digging up an old topic here but I have the same issue. I love the idea of shared and incremental refreshed dataflows, but not at the expense of exponential dataset refresh times. In my case I have a dataset that takes about 6-8 mins to refresh using direct SQL but almost 30 mins when calling the dataflows that contain the same logic.

 

Has anyone found a fix for this? I'm about ready to ditch the dataflows for this project, which would be shame because they have so much promise but 30min refreshes for no reason are hard to sell to the business.

v-xicai
Community Support
Community Support

Hi @rexii23 ,

 

You may refer to the documents:

https://docs.microsoft.com/en-us/power-bi/service-dataflows-overview,

https://docs.microsoft.com/en-us/power-bi/service-dataflows-create-use,

https://docs.microsoft.com/en-us/power-bi/service-dataflows-incremental-refresh.

 

Best Regards,

Amy

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Hi Amy

 

I've already read through the documentation. I do not see the scenario I outlined covered them.

Hi,

 

We have the same issue with incremental refresh from dataflow to dataset. Please let me know if you found any solution.

 

Thanks,

Keerthi

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors