Refresh of Dataset fails inconsistently because of too little memory

[Eric7]

The dataset is located in a worskpace with Embedded (Premium) capacity. The Large Dataset option is turned OFF.

What is strange is that quite often, but not always, the refresh fails with an error message saying "wasn’t enough memory" in the capacity (see full message at the bottom). The data being refreshed is exactly the same every time (coming from 3 Dataflows where 2 in Power Platform and 1 in Power BI Service), and the dataset itself is also unchanged. And neither the dataset nor its report is accessed by any users at all.

The capacity scale/size is normally A2. But when we get this memory error, we increase to A3. And sometimes the refresh fails even on A3, so we have to set it to A4.

As far as we have understood, on Premium capacities, each artifact (e.g. Dataset?) will have its own memory limit being the max that is set in the capacity scale/size, and will not share this memory with other artifacts using the same capacity.

How can the memory usage vary this much on different refreshes of the very same dataset with the very same data?? To us, it seems that either our dataset does NOT have its own separated memory limit, or the refresh logic in Power BI Service is unstable.

The full error message:

Data source error: Resource Governing: This operation was canceled because there wasn’t enough memory to finish running it. Either increase the memory of the Premium capacity where this dataset is hosted or reduce the memory footprint of your dataset by doing things like limiting the amount of imported data. More details: consumed memory MB, memory limit MB, database size before command execution MB. Learn more, see https://go.microsoft.com/fwlink/?linkid=2159753.

[GilbertQ]

Hi @Eric7 

Yeah that does not make sense... What happens when you look at the log analytics and view the metrics?

[GilbertQ]

Hi @Eric7 

You have to remember that when you are refreshing the following all contributes to your total memory:

- Dataset size

- Refreshing

- Queries

If you are refreshing the entire dataset and the dataset consumes lets say 10 GB, that means you will need another 10 GB when the dataset is refreshing. If you then have queries running at the same time that consume a lot of memory that could consume say another 2GB which is more than 20GB in total, even though your dataset is only 8GB.

I would suggest using incremental refreshing which means your refreshing could drop to say 1GB?

[Eric7]

Hi @GilbertQ 

Yes, but in this case, no queries run against the dataset and the dataset size does not change. So why it sometimes refreshes with A2 (5 GB) and sometimes needs A4 (25GB) is still a mystery?

I assume that workspaces with Embedded do make available to each dataset the full amount of RAM that the capacity allows when refreshing?

[v-chenwuz-msft]

Hi @Eric7 

This article states that Capacity operations are classified as either interactive or background, and interactive operations are always prioritized over background operations to ensure the best possible user experience. https://docs.microsoft.com/en-us/power-bi/admin/service-premium-what-is#how-capacities-function

And when refreshing the data set, you need twice the size of the dataset to refresh as @GilbertQ said. When your user interactive with other reports ,this will consume the capacity.

Please refer this article to know what content consume capacity and optimizing your models.

https://docs.microsoft.com/en-us/power-bi/admin/service-premium-capacity-optimize#what-content-is-us...

Best Regards

Community Support Team _ chenwu zhu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.