Dataset refresh failures - recent issue

[gmd_23]

Hi,

Is anyonwe else facing multiple dataset filaures on previously working datasets. Problem occurs regardless of datasource, size, complexity of query etc. This occuring since late 3rd Feb. Same error across the board - 

{"error":{"code":"ModelRefresh_ShortMessage_ProcessingError","pbi.error":{"code":"ModelRefresh_ShortMessage_ProcessingError","details":[{"code":"ModelRefresh_DataSourceErrorLabel","detail":{"type":1,"value":"The M evaluation exceeded the memory limit. To address the issue consider optimizing the M expressions, reducing the concurrency of operations that are memory intensive or upgrading to increase the available memory. Container exited unexpectedly with code 0x0000DEAD. PID: 5888.\r\nUsed features: (none).;Container exited unexpectedly with code 0x0000DEAD.

Regards

[lbendlin]

If it's any consolation, unnecessary refreshes are something we also see all the time. It's a tricky thing though - how should the service know that the developer did due diligence and updated the data too when they mucked with the report and re-uploaded to the workspace.  Worst case the service does an unneeded refresh, best case it catches the developer napping and inadvertently uploading a really old data snapshot.

[lbendlin]

We have switched to Gen2, and did not regret it yet.  Getting rid of the rendering shackles was totally worth losing a few metrics temporarily. IMHO the only metric that really counts is cumulative query duration, and we still get that from the audit logs.

[gmd_23]

Not going to discount it as an option, not yet.

Seeing other odd behaviour - refreshes that are reported as finished before they've even started. Dataset refreshes after a publish. Refreshes happening even when scheduled refresh is switched off.

Our region is North Europe.

[gmd_23]

Its a P1 capacity, which has been running fine with the same workload for quite some time. Now, all of a sudden its failing spectacularly. The metrics from the capacity app indicate that all is well - nothing hitting memory limits etc.

Also reports of odd behaviour where once a dataset is publiched, the system tries to immediately refresh it. Nothing set on schedule, no refresh on demand requested, still it tries to refresh (which typically fails).

Problem occurs even with very low level of refresh -  again the metrics indicate low levels of refresh concurrency and 150+ minuites of capacity refresh capacity available, and at a time when there would be little to no user activity.

[nickyvv · Accepted solution]

Hi @gmd_23,

You could try enabling Premium Gen 2 on your capacity, that will for sure get rid of any concurrency or memory refresh problem. Do remember that it is in preview, and that you will not have refresh metrics in the Premium Capacity App anymore. Have a look at all the current limitations.

But you can always switch back if it doesn't solve your problems.

If that doesn't help, I suggest you create a support ticket (which is free with a Pro license). Go to the support page to file a support ticket.

[gmd_23]

Hi Nicky,

Thought of that, not having the capacity metrics a huge drawback.

What is alarming is that refreshes are failing even at times of low user and refresh activity.

This does not feel like "our" problem, but a problem in the data centre, like the recent advisory about log files searching for a "D" drive.

[v-cazheng-msft]

Hi, @gmd_23 

According to the error message you provided, this issue is caused by exceeding the memory limit. The size of the datasets will grow multiply when refreshing. And If multiple datasets need refresh at the same time, they will competing for memory resources. If the memory is insufficient, even if there is only one model in the capacity, data refresh will not complete.

You can try these steps to solve this problem

1 Reduce the number of datasets refreshed concurrently

2 If the capacity is Premium capacity, analyze the refresh history and metrics related with refresh to find the datasets needing optimizing, including removing data unused and reducing queries

3 Scale up the capacity if necessary

You can also refer Why are data refreshes not completing.

Best Regards,

Caiyun Zheng

Is that the answer you're looking for? If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[lbendlin]

Needs more details. Which SKU are you on? What's the utilization on the capacity?