Refresh Failed - PPU - This operation was canceled because there wasn’t enough memory to finish

[mmaraz]

The scheduled refresh for a dataset in PPU (Premiunm per User) workspace is failing with a memory error:

Resource Governing: This operation was canceled because there wasn’t enough memory to finish running it. Either increase the memory of the Premium capacity where this dataset is hosted or reduce the memory footprint of your dataset by doing things like limiting the amount of imported data. More details: consumed memory 51399 MB, memory limit 51351 MB, database size before command execution 51048 MB. Learn more, see https://go.microsoft.com/fwlink/?linkid=2159753.

The dataset is setup for incremental refresh (largest tables), contains about 6 months of data with last 3 days being refreshed incrementally.

While dataset is large (50014.31 MB), it is well below the 100GB PPU limit. 

While I have seen post regarding needing a double memory for refresh+rendering, my understanding was that incremental refresh would be the exactly the right solution for this.

Sometimes just executing a manual refresh right after a scheduled one fails with no change whatsoever succeeds.

Any help/clarity into this would be appreciated.

Best,

Miro

[ElKharchi]

Same problem. Someone with solution??

[corvinnicolae]

Hi Miro,

From my recent experience, the triggered memory error is not about the data itself, but the memory allocated to the queries run in parallel. In the PBI Desktop client one must make sure that the total memory allocated for parrallel queries does not exceed the PBI cloud capacity limit. File/Options and Settings/Options/Data Load/Parallel loading of tables.

Corvin

[LukanowiczpRY]

Have exactly same problem, even more riddiculous as my full data-set is only 1,5-2,0 GB. Full load works most/all of the times, 3 days incremental fails alot, with same mentioned error. Still looking for solution or at least understanding, how 2 GB dataset can generate 50 GB of memory consumption. I saw alot of articles that consuption could be arround 2x Dataset size, not 25x.

[mmaraz]

Unfortunately we were not able to determine the root cause. Went to some lenghts to reduce the model size -(both total volume and reduce incremental interval) but even then it was failing frequently (but not always which is strange by itself - maybe depends on shared resources or collocation or other factor?). We had to abandon PowerBI as a visualization layer due to this for the project and look for an alternative.

[GilbertQ]

Hi @mmaraz 

Can you confirm that the partitions are correctly showing data for the incremental processing?

You could also change the processing configuration from being say yearly to monthly and see if that helps. And also the last N days to process what happens if you make that fewer days?

[GilbertQ]

Hi @mmaraz 

I am pretty confident that there are some background limitations.

From the error it is saying that it is consuming 51,048MB which is 50GB, this means that your refreshing of your dataset is consuming 50GB for the last 3 days which is quite a lot. 

I would look at your dataset to see which columns could be made smaller, to consume less memory, and allow the refresh to work.

[mmaraz]

Hello @GilbertQ, thanks for the response.

It's exactly those internal limits that I'm trying to understand. What I'm most confused about is that the entire dataset which is 6 months of data is 50GBs, so how is it that incremental refresh of 3 days which is a fraction of that is consuming so much memory ?

[Anonymous]

Did you end up determining the high memory consumption? I'm having the same problem as well and the weird thing is that it seems the incremental refresh is the one that's failing while the initial full refresh is succeeding. If anything I would've thought it should be the opposite.

[GilbertQ]

Hi @Anonymous 

Can you have a look at the Power BI Premium Metrics App to see how much memory your capacity is using?

Monitor Power BI Premium capacities with the Premium Capacity Metrics app. - Power BI | Microsoft Docs

[Power_user_]

Same problem. Someone with solution??