Can Someone Clarify Refresh Interactions of Datasets and Dataflows?

[dvl_ctaul]

I've been reading some articles about dataflows and refreshes but I'm still a little bit confused, can anyone clarify something for me?

I set up dataflows that I am using for my dataset that resides on the same app workspace. I set up a scheduled refresh for both the dataflows and the dataset that uses those data flows. I'm just wondering if I'm refreshing more than I need to? Does the scheduled dataset refresh also refresh the dataflow? Are there conditions? Why does the dataflow refresh take 8 minutes, but the dataset refresh take 15 minutes? All of the data in that dataset is coming from my dataflow.

Any direction is appreciated!

[v-rongtiep-msft]

Hi  @dvl_ctaul ,

1、Does the scheduled dataset refresh also refresh the dataflow? 

The answer to your question is no as dataflow and dataset refreshes are completely separate. Refreshing one does not automatically refresh the other, if you want to trigger one refresh when the other is done you'll have to set that up using APIs.

This could help : https://community.powerbi.com/t5/Service/Dataflow-vs-Dataset-refresh/m-p/699601#M68755

2、Why does the dataflow refresh take 8 minutes, but the dataset refresh take 15 minutes.

• Power BI Dataflow is the data transformation component in Power BI. It is a Power Query process that runs in the cloud, independent from Power BI report and dataset, and store the data into CDM: Common Data Model inside Azure Data Lake storage.
• Power BI Dataset is the object that contains the connection to the data source, data tables, the data itself, the relationship between tables, and DAX calculations. Usually, Power BI dataset is hidden from the Power BI Desktop view, but easily can be seen in the Power BI service.

A Power BI dataflow can run Power Query transformations, and load the output into Azure Data Lake storage for future usage. And that is exactly, how it can help with reducing your Power BI dataset refresh time.

More details: Dataflow Vs. Dataset: What are the Differences of these two Power BI Components 

How to Use Dataflow to Make the Refresh of Power BI Solution FASTER! 

Best Regards

Community Support Team _ Polly

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[dvl_ctaul]

That's the problem though, since I started migrating my data sources to data flow my dataset refreshes started taking longer. Very weird. 

[v-rongtiep-msft]

Hi @dvl_ctaul ,

The reason for refreshing taking too much time：

Too many datasets to refresh at the same time occupy CPU and memory, Insufficient gateway or underlying data source performance (busy on network / refresh), To refresh the model to be loaded into memory, it takes up 2 times the size of the data set, and you need to wait for memory resources.

You need to scheduled the refresh of dataset at different time.

I have found a similar post, please refer to it to see if it helps you.

https://community.powerbi.com/t5/Service/Scheduled-refreshes-are-slow-manual-refresh-is-quick/m-p/10...

Best Regards

Community Support Team _ Polly

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[GilbertQ]

Hi @dvl_ctaul 

A scheduled dataset refresh will NOT refresh the dataflow.

You will need to ensure that you first refresh your dataflow and only once that is finished then refresh your dataset.

The reason it often takes longer to refresh a dataset compared to a dataflow could be due to what else is being refreshed and the steps taken in Power Query?