Solved: Dataflows Architecture Question

alexpbi88 · ‎10-20-2022

Hi everyone,

I'm thinking to include in our reporting solutions the dataflows and datamarts.

My idea is to have in dataflows a 1:1 copy of the tables stored in Azure Databricks and then consume them with PowerBI Datamarts creating datamarts dedicated to each project.

There are multiple reasons why we're choosing to have a copy of the DB in dataflows instead of connecting directly to Azure DBK so please let's not focus on this point.

My question is: is there any best practice on how to setup dataflows? How many tables should a dataflow include? For now we have diverging opinions in the team, ranging from having 1 dataflow per 1 table (allowing max flexibility and different refreshes times per table), to 1 dataflow per 1 source system, to 1 dataflow for all (around 40 tables, all <1.5mln rows).

What should we take into consideration to choose and prevent issues in the future, when more tables might be included?

Thanks in advance for the help

v-rongtiep-msft · ‎10-20-2022

Hi @alexpbi88 ,

Deleted datasources will still appear in the Setting page in the gateway drop-down.
Depth equates to dataflows linked to other dataflows. The current maximum depth is 32.
Breadth equates to entities within a dataflow.
- There's no guidance or limits for the optimal number of entities is in a dataflow, however, shared dataflows have a refresh limit of two hours per entity, and three per dataflow. So if you have two entities, and each takes two hours, you shouldn't put them in the same dataflow.
- For Power BI Premium, guidance and limits are driven by individual use cases rather than specific requirements. The only limit for Power BI Premium is a 24-hour refresh per dataflow.
A Power BI Premium subscription is required in order to refresh more than 10 dataflows cross workspace
PowerQuery limitations are found in the PowerQuery Online usage limits article.
When using Computed entities with gateway data sources, the data ingestion should be performed in different data sources than the computations. The computed entities should build upon entities that are only used for ingestion, and not ingest data within their own mash-up steps.
In Power BI dataflows, you can use parameters but you can't edit them unless you edit the entire dataflow. In this regard, parameters in dataflows behave similar to declared constants.

Dataflows Limitations, restrictions and supported connectors and features - Power BI | Microsoft Lea...

I have also found asimilar post, please refer to it to see if it helps you.

Solved: All tables to dataflows - Microsoft Power BI Community

Best Regards

Community Support Team _ Polly

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

v-rongtiep-msft · ‎10-20-2022

Hi @alexpbi88 ,

Deleted datasources will still appear in the Setting page in the gateway drop-down.
Depth equates to dataflows linked to other dataflows. The current maximum depth is 32.
Breadth equates to entities within a dataflow.
- There's no guidance or limits for the optimal number of entities is in a dataflow, however, shared dataflows have a refresh limit of two hours per entity, and three per dataflow. So if you have two entities, and each takes two hours, you shouldn't put them in the same dataflow.
- For Power BI Premium, guidance and limits are driven by individual use cases rather than specific requirements. The only limit for Power BI Premium is a 24-hour refresh per dataflow.
A Power BI Premium subscription is required in order to refresh more than 10 dataflows cross workspace
PowerQuery limitations are found in the PowerQuery Online usage limits article.
When using Computed entities with gateway data sources, the data ingestion should be performed in different data sources than the computations. The computed entities should build upon entities that are only used for ingestion, and not ingest data within their own mash-up steps.
In Power BI dataflows, you can use parameters but you can't edit them unless you edit the entire dataflow. In this regard, parameters in dataflows behave similar to declared constants.