We have publised a number of reports which use Direct Query (to SQL server) on a premium capacity. In the last couple of weeks there have been several occasions where all these report fail to load - All visualizations in the report fail to load with the message either "Unable to load data model" or "Database operation was cancelled due to a previous error".
The problem corrects itself after a couple of hours.
The On-Prem Data Gateway logs the following exception during this time.
Any idea what could be wrong?
DM.EnterpriseGateway Error: 0 : 2017-12-28T03:16:37.9703712Z DM.EnterpriseGateway 78bfbdc4-b932-46f9-8df4-c9ab2bd28f15 e339d9df-e1fa-4af2-a40d-2f5dd806dc27 MGSR e339d9df-e1fa-4af2-a40d-2f5dd806dc27 DB4431EE [DM.Pipeline.Common] Non-gateway exception encountered in activity scope: System.ServiceModel.CommunicationException: The communication object '{0}' is in the {1} state. Communication objects cannot be used for communication unless they are in the Opened state.
Server stack trace:
at Microsoft.ServiceBus.ClientWebSocketConnection.ThrowIfNotOpen()
at Microsoft.ServiceBus.ClientWebSocketConnection.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.BufferedConnection.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.SendAsyncResult.WriteCore()
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.SendAsyncResult..ctor(FramingDuplexSessionChannel channel, Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.OnBeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.OutputChannel.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.SocketConnectionChannelListener`2.DuplexSessionChannel.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.StartSend(Boolean completedSynchronously)
at System.ServiceModel.Channels.ServiceChannel.BeginCall(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, TimeSpan timeout, AsyncCallback callback, Object asyncState)
at System.ServiceModel.Channels.ServiceChannel.BeginCall(ServiceChannel channel, ProxyOperationRuntime operation, Object[] ins, AsyncCallback callback, Object asyncState)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2,TArg3](Func`6 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, TArg3 arg3, Object state, TaskCreationOptions creationOptions)
at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.CreateTask(ServiceChannel channel, ProxyOperationRuntime operation, Object[] inputParameters)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeTaskService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.PowerBI.DataMovement.Pipeline.InternalContracts.IGatewayTransferCallback.TransferCallbackAsync(Byte[] packet)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.ServiceModel.GatewayTransferService.<>c__DisplayClass5_0.<<TransferImplAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.Diagnostics.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
DM.EnterpriseGateway Error: 0 : 2017-12-28T03:16:37.9859976Z DM.EnterpriseGateway 78bfbdc4-b932-46f9-8df4-c9ab2bd28f15 e339d9df-e1fa-4af2-a40d-2f5dd806dc27 MGSR e339d9df-e1fa-4af2-a40d-2f5dd806dc27 C416A628 [DM.Pipeline.Common.TracingTelemetryService] Event: FireActivityCompletedWithFailureEvent (duration=37, err=CommunicationException, rootcauseErrorEventId=0)
DM.EnterpriseGateway Error: 0 : 2017-12-28T03:16:37.9859976Z DM.EnterpriseGateway 9c014220-0d4f-4948-98d2-12ca02cbb203 e339d9df-e1fa-4af2-a40d-2f5dd806dc27 MGRF e339d9df-e1fa-4af2-a40d-2f5dd806dc27 C9C615D0 [DM.Pipeline.Common] Non-gateway exception encountered in activity scope: System.ServiceModel.CommunicationException: The communication object '{0}' is in the {1} state. Communication objects cannot be used for communication unless they are in the Opened state.
Server stack trace:
at Microsoft.ServiceBus.ClientWebSocketConnection.ThrowIfNotOpen()
at Microsoft.ServiceBus.ClientWebSocketConnection.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.BufferedConnection.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.SendAsyncResult.WriteCore()
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.SendAsyncResult..ctor(FramingDuplexSessionChannel channel, Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.OnBeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.OutputChannel.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.SocketConnectionChannelListener`2.DuplexSessionChannel.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.StartSend(Boolean completedSynchronously)
at System.ServiceModel.Channels.ServiceChannel.BeginCall(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, TimeSpan timeout, AsyncCallback callback, Object asyncState)
at System.ServiceModel.Channels.ServiceChannel.BeginCall(ServiceChannel channel, ProxyOperationRuntime operation, Object[] ins, AsyncCallback callback, Object asyncState)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2,TArg3](Func`6 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, TArg3 arg3, Object state, TaskCreationOptions creationOptions)
at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.CreateTask(ServiceChannel channel, ProxyOperationRuntime operation, Object[] inputParameters)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeTaskService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.PowerBI.DataMovement.Pipeline.InternalContracts.IGatewayTransferCallback.TransferCallbackAsync(Byte[] packet)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.ServiceModel.GatewayTransferService.<>c__DisplayClass5_0.<<TransferImplAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.Diagnostics.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.ServiceModel.GatewayTransferService.<>c__DisplayClass5_0.<<TransferImplAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.Diagnostics.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
DM.EnterpriseGateway Error: 0 : 2017-12-28T03:16:37.9859976Z DM.EnterpriseGateway 9c014220-0d4f-4948-98d2-12ca02cbb203 e339d9df-e1fa-4af2-a40d-2f5dd806dc27 MGRF e339d9df-e1fa-4af2-a40d-2f5dd806dc27 F911D718 [DM.Pipeline.Common.TracingTelemetryService] Event: FireActivityCompletedWithFailureEvent (duration=358, err=CommunicationException, rootcauseErrorEventId=0)
DM.EnterpriseGateway Error: 0 : 2017-12-28T03:16:37.9859976Z DM.EnterpriseGateway c9286486-48ef-42d7-acbc-db6fcc8999d5 e339d9df-e1fa-4af2-a40d-2f5dd806dc27 MGST e339d9df-e1fa-4af2-a40d-2f5dd806dc27 0E92BE0E [DM.Pipeline.Common] Non-gateway exception encountered in activity scope: System.ServiceModel.CommunicationException: The communication object '{0}' is in the {1} state. Communication objects cannot be used for communication unless they are in the Opened state.
Server stack trace:
at Microsoft.ServiceBus.ClientWebSocketConnection.ThrowIfNotOpen()
at Microsoft.ServiceBus.ClientWebSocketConnection.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.BufferedConnection.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.SendAsyncResult.WriteCore()
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.SendAsyncResult..ctor(FramingDuplexSessionChannel channel, Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.FramingDuplexSessionChannel.OnBeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.Channels.OutputChannel.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at Microsoft.ServiceBus.SocketConnectionChannelListener`2.DuplexSessionChannel.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Dispatcher.DuplexChannelBinder.BeginSend(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.StartSend(Boolean completedSynchronously)
at System.ServiceModel.Channels.ServiceChannel.BeginCall(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, TimeSpan timeout, AsyncCallback callback, Object asyncState)
at System.ServiceModel.Channels.ServiceChannel.BeginCall(ServiceChannel channel, ProxyOperationRuntime operation, Object[] ins, AsyncCallback callback, Object asyncState)
at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2,TArg3](Func`6 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, TArg3 arg3, Object state, TaskCreationOptions creationOptions)
at System.ServiceModel.Channels.ServiceChannelProxy.TaskCreator.CreateTask(ServiceChannel channel, ProxyOperationRuntime operation, Object[] inputParameters)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeTaskService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.PowerBI.DataMovement.Pipeline.InternalContracts.IGatewayTransferCallback.TransferCallbackAsync(Byte[] packet)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.ServiceModel.GatewayTransferService.<>c__DisplayClass5_0.<<TransferImplAsync>b__1>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.Diagnostics.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.ServiceModel.GatewayTransferService.<>c__DisplayClass5_0.<<TransferImplAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.Diagnostics.PipelineTelemetryService.<ExecuteInActivity>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.ServiceModel.GatewayTransferService.<TransferImplAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.GatewayCore.ServiceModel.GatewayTransferService.<>c__DisplayClass4_0.<<TransferAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.Pipeline.Common.Diagnostics.PipelineTelemetryService.<ExecuteInActivity>d__7`1.MoveNext()
