Hello all,
We are using PowerBI Pro and we have an on-premise gateway installed on a VM. We created our dashboards and set up the data refresh, it ran some days and failed some days. As we progressed through creating more dashboards, it started to fail more. We tried to switch the scheduled run time making sure it never runs at peak hours when the overall load is high, nothing worked. The refresh is frequently failing now. We checked the log files and this is what we see.
2 hours after the data refresh kicks off:
DM.EnterpriseGateway Information: 0 : 2018-04-18T13:18:23.3488507Z DM.EnterpriseGateway a2426269-6c98-468c-a080-dd78d2b56afe d4df02bd-8be4-4223-9013-e2b8ec245a48 MGPL 3c92e719-bf94-4c28-87fd-ca9f5b60129b B07501EB [DM.GatewayCore] Spooling packet index 387 for async operation 85b80b60-2fc3-47e2-a15a-6a20a492681c
DM.EnterpriseGateway Information: 0 : 2018-04-18T13:18:23.4738628Z DM.EnterpriseGateway 4e8b8538-7060-4b3e-9c47-ba2be7ab11e6 d4df02bd-8be4-4223-9013-e2b8ec245a48 MGPP 3c92e719-bf94-4c28-87fd-ca9f5b60129b 0C45C463 [DM.GatewayCore] Async operation bb09f5e0-0b00-44e5-b65d-367a4be0c65c transitioned into Cancelled state after 120 seconds timeout
DM.EnterpriseGateway Information: 0 : 2018-04-18T13:18:23.4738628Z DM.EnterpriseGateway 4e8b8538-7060-4b3e-9c47-ba2be7ab11e6 d4df02bd-8be4-4223-9013-e2b8ec245a48 MGPP 3c92e719-bf94-4c28-87fd-ca9f5b60129b 13725449 [DM.GatewayCore] CleanupPackets - Cancelling in-progress streaming. Current job ID: bb09f5e0-0b00-44e5-b65d-367a4be0c65c
DM.EnterpriseGateway Error: 0 : 2018-04-18T13:18:23.4738628Z DM.EnterpriseGateway 255fbe83-bc15-41f7-98b0-4a8f5745260b d4df02bd-8be4-4223-9013-e2b8ec245a48 MGAS 3c92e719-bf94-4c28-87fd-ca9f5b60129b D8C401C1 [DM.Pipeline.Diagnostics] Exception object created [IsBenign=False]: Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.AsyncOperationStreamingCancelledException: The async operation id bb09f5e0-0b00-44e5-b65d-367a4be0c65c streaming was cancelled.; ErrorShortName: AsyncOperationStreamingCancelledException
DM.EnterpriseGateway Error: 0 : 2018-04-18T13:18:23.4738628Z DM.EnterpriseGateway 0971d3ff-9491-4abc-ba6f-499434434f98 d4df02bd-8be4-4223-9013-e2b8ec245a48 MGPS 3c92e719-bf94-4c28-87fd-ca9f5b60129b 865A0AC3 [DM.Pipeline.Common.TracingTelemetryService] Event: FireActivityCompletedWithFailureEvent (duration=7267414, err=AsyncOperationStreamingCancelledException, rootcauseErrorEventId=0)
DM.EnterpriseGateway Error: 0 : 2018-04-18T13:18:23.4738628Z DM.EnterpriseGateway 553ae6ad-3ffa-4df7-96e4-eaa1babbac50 d4df02bd-8be4-4223-9013-e2b8ec245a48 MDSR 3c92e719-bf94-4c28-87fd-ca9f5b60129b 1CE0F329 [DM.GatewayCore] Error serializing response: [0]Microsoft.PowerBI.DataMovement.Pipeline.Diagnostics.AsyncOperationStreamingCancelledException: The async operation id bb09f5e0-0b00-44e5-b65d-367a4be0c65c streaming was cancelled.
M.EnterpriseGateway Error: 0 : 2018-04-18T13:22:34.0063973Z DM.EnterpriseGateway 4cb29abd-e497-4061-a25f-7c2550a9dadb d4df02bd-8be4-4223-9013-e2b8ec245a48 MGST 3c92e719-bf94-4c28-87fd-ca9f5b60129b 95CDDA27 [DM.Pipeline.Common] Non-gateway exception encountered in activity scope: System.ServiceModel.CommunicationException: An error occurred while transmitting data. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndSend(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndWrite(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
I can see that the streaming is being cancelled after 120 seconds of timeout throwing a socket communication exception while trying to transmit data because the host is forcibly closing the connection. I don't understand why this is happeneing and what could be the possible solution? On the days when the refresh happens successfully, the task is completed within 20mins.
Could you please help me understand this issue and guide me through troubleshooting the problem?
Thank you.
