One Dataflow vs Multiple dataflows

[matteoumbri]

Hi everyone, 

I have several dataflows, for simplicity say 1, 2 and 3 (all of them getting data from a SQL Server database with native queries) having the same refresh frequency needs - 8 times a day starting from 7:30am. 

They query different data, and are not linked/dependent each other.

What if I merge them all into one main dataflow: would overall total refresh time be higher or lower than keeping them separated? That is, will the refreshes occur sequential entity by entity, or will the engine run start them all at the same time and run in parallel?

Any possible concerns / drawbacks of such approach? 

What if one entity refresh fails, will there be any automatic retry? Will the succeeded entity be anyway saved and accessible, or will 5he entire dataflow "failed" and the accessible data will be the one of the previous successful refresh? 

Many thanks

[v-eqin-msft · Accepted solution]

Hi @matteoumbri ,

Dataflows, which require different refresh timings, can all be scheduled individually.To better understand how a dataflow refresh operation performs, review the Refresh History for the dataflow by navigating to Dataflow > Settings > Refresh History. You can also select the dataflow in the Workspace > context menu (…) > Refresh History.

Refer to:

Understanding and optimizing dataflows refresh - Power BI | Microsoft Docs

Develop solutions with dataflows - Power BI | Microsoft Docs

Best Regards,
Eyelyn Qin
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[matteoumbri]

Hi @v-eqin-msft ,

thanks for reply, I have checked the Refresh History csv files where you can see individual refresh duration for the single entities of a dataflow, and found some interesting data:

- even very small entities (based on SELECT statements on very small SQL Server tables) always last at least 30 seconds, never lower than 30 seconds

- a query (exactly the same) when executed for refreshing an entity inside a dataflow with multiple entities, takes always less time than the the same query which is run in a dataflow with 1 entity only. Weird, I would assume their duration should be the same

Anyway, if this is how it works, I am going to merge into 1 dataflow all those 7-8 queries that have same refresh schedule needs, and are all part of the same dataset

Thanks

Matteo