Reopened: Gateway Pipeline Error [GatewayPipelineErrorCode]=DM_GWPipeline_Gateway_CanceledError

[LasseMr]

Dear community,

This is a reopening of the issue I posted some time ago as it has started reocurring, find related post here: https://community.powerbi.com/t5/Service/Gateway-Pipeline-Error-GatewayPipelineErrorCode-DM-GWPipeli...

In short, I have currently one particular dataflow that keeps failing and always after a certain amount of seconds, see snip from history, e.g. after 1:14 or after 14seconds:

The short error message from the refresh log is: Gateway Pipeline Error [GatewayPipelineErrorCode]=DM_GWPipeline_Gateway_CanceledError

Since last I have enabled additional logging on the gateway, I am a bit lost in the log, but I believe it is around this section I can find details, a series of 5 errors shortly after each other looking like this:

DM.EnterpriseGateway Error: 0 : 2021-06-03T06:30:07.7360208Z DM.EnterpriseGateway ac634d11-6f78-4637-aab7-c3b8bf78dab4 2d53fc88-e80c-49d9-92c9-96c12dc41041 MGPL 60b3c3e9-873a-493c-b94c-6a68a076ecb9 06cd82d2-55a3-4811-9b83-f21ea5703f21 3319b036-df62-4890-9e39-b6b820757b02 03F51593 [DM.Pipeline.Common.TracingTelemetryService] Event: FireActivityCompletedWithFailureEvent (duration=1, err=AsyncOperationExpiredException, rootcauseErrorEventId=0)

It mentions FireActivityCompletedWithFailureEvent and err=AsyncOperationExpiredException and ends up throwing:

System.AggregateException: One or more errors occurred. ---> Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.AsyncOperationExpiredException: The async operation id 8b132291-d644-477b-9d18-a18062314939 was already expired.
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.GatewayOperationResultSpooler.SpooledOperationController.EnsureAlive(Boolean extendAutomaticCancellation, String callerMemberName)

Does anyone have experience with these sorts of errors and/or ideas on how to troubleshoot?

The dataflow still runs through with success occasionally, lately there has just been more errors than success.

Worth mentioning, running on a embedded capacity, have around 5 gateways setup on different environments, and on this particular one there are approx 8 dataflows and overall it is only this one dataflow that currently fails, and I have not experienced any similar errors on the 4 other gateways.

@v-kkf-msft , I know you looked into my post from earlier, maybe you have experience in this field?

[v-henryk-mstf]

Hi @LasseMr ,

It may be the overload caused by too many datasets being refreshed at the same time. Try to modify the scheduled refresh time or close other refreshes at a certain time, and only perform this scheduled refresh, whether it can be refreshed successfully.

Or check the common refresh problems in the official website documents for troubleshooting: Troubleshooting refresh scenarios - Power BI | Microsoft Docs

If the problem is still not resolved,let me know immediately. Looking forward to your reply.

Best Regards,
Henry

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[LasseMr]

Hi @v-henryk-mstf ,

Thank you for the support.

I have tried to modify scheduled refresh to a timeslot that is not overlapping any other dataflows/datasets, I have even tried to disable all other and only schedule this, and I have tried to manually trigger the refresh during weekend daytime when I know there is no useractivity - all with same result, 9 out of 10 fails. There is a tendency however, but not always the case, that if the server on which the standard gateway is installed is restarted and I subsequently manually trigger the refresh it goes through succesfully.

I have looked into the troubleshooting scenarios and associated logs - they haven't been of any use in this case, unfortunately, which is also why I was adviced to enable additional logging, from which you can see some of the error-details in the first post.

Might there be any MS engineers with greater experience in these advanced logs following in on the community?

Best regards,

Lasse