Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!
We had a little bit of an emergency yesterday concerning our dataflows. Here's the chain of events:
1. We recently got Power BI Premium, and I put one of our workspaces containing many dataflows in a Premium capacity. Everything was fine for a couple days.
2. I went into the capacity settings and adjusted Max Memory in the capacity settings for dataflows and datasets.
3. A few hours later, datasets using these dataflows failed to refresh. We also noticed that we could not connect to any of these dataflows in a new .pbix file (Error: Dataflow Internal Error).
4. I changed the capacity settings back to what they had been. The error persisted.
5. I took the workspace off Premium capacity. The error persisted.
6. I tried refreshing the dataflows once the workspace was no longer premium, and they all failed to refresh (Error: There was a problem refreshing your dataflow).
7. Since the Premium capacity was only a test capacity, I deleted the capacity itself. When we tried to refresh the dataflow, it threw an error message saying we needed to assign a capacity to the workspace.
8. We created a new premium capacity and assigned it to the workspace. Everything worked fine.
Why did this happen??? It makes me nervous to use Power BI Premium, and especially to adjust the workloads in the capacity settings. Has this happened to anyone else?
Are you using the Enhanced Dataflows Compute Engine? I've had issues with large queries failing so I've disabled this preview feature.
You can use this calculator to see how many containers you can get with allocated memory to dataflows and to containers: http://petcu40.blogspot.com/2019/06/premium-dataflow-workload-parameters.html
We're not using many datasets in Premium, so I've dialed that down to 80%, increased dataflows to 80%, disabled EDCE and set container size to 2400 Mb for (hopefully) 8 max containers.
Certain dataflow features aren't supported when dataflow storage is in Azure Data Lake Storage Gen2:
Power BI Pro, Premium and Embedded workspaces:
There are some additional considerations as well, described in the following list:
Thanks for the reply, Lewis-H. Some of your suggestions make sense, but none of them fit exactly:
1. We aren't using any linked entities.
2. I am the owner of the dataflows, and during this period I could not even open the dataflow to see the list of columns in each table.
3. What is meant by the dataflow storage location cannot be changed? I don't think this is the issue, because the dataflows were fine for a couple days after putting the workspace in a premium capacity, but broke only after I adjusted the dataflow settings in the capacity.
4. Could you clarify your last point? Our dataflows do connect to an on-prem data warehouse, but they worked completely fine when on a shared capacity and a premium capacity until I adjusted the dataflow workload settings.