We have a dataflow that's a bit large, with around 55 million records and we use it as a data source for four datasets.
All datasets use Import mode to get the data from this dataflow.
Three of those, use only this dataflow as a data source. The fourth uses this same dataflow plus another dataflow which is small with ~100k records only.
We have noticed that when we trigger those four datasets to refresh concurrently, the performance is much worse then if we trigger them individually, one at a time. At first we thought it could be a matter of server resources, but we tried running those on an almost dedicated premium P1 capacity and we still found issues, including an error with the following message:
| Data source error: | The operation was throttled by Power BI Premium because there were too many datasets being processed concurrently. Please try again later. |
To put that to the test, we tried refreshing 5 big datasets, all having big dataflows as sources as well, but different dataflows, not the same one. We did not see the same performance issues in that test, refresh times weren't far away from the expected.
I know dataflows are CSV files in Azure Data Lake in the background, so I'm wondering if anybody knows if there's some good practice to follow here, with regards to sharing a dataflow to be imported by many datasets. Is there such a thing as a dataset locking down a dataflow? For example, making it unavailable for other datasets?
If that is the case, then the more concurrent dataset refreshes are executed using the same dataflow, the slower the performance would be. If that is so, I'd be surprised since one of the main advantages of a dataflow is exactly reusability.
Are we missing something here? Is there some setting in the premium capacity that can enhance parallel reading of a dataflow for datasets?
