cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
burny Frequent Visitor
Frequent Visitor

New OP data gateway unreachable

Our data gateway has been occasionally failing scheduled refreshes with 'gateway unreachable' errors which I'm fairly confident are because the machine was taxed by other data processing tasks, so to remedy I've added a second data gateway to the same cluster and set the cluster to distribute queries. Unfortunately I'm still seeing the 'gateway unreachable' errors and I am not sure if the secondary gateway is participating in the cluster. However now the pbi service gateway page now reports this error:

 

Not all gateway instances within this cluster are connected.

 

Please find more details below about specific errors for each gateway instance.

Activity ID: d5e98a0b-4ddb-42f2-9321-824569cde7a3

Request ID: 7582bc4f-04fa-74f1-fd35-3605dd6e21f6

Cluster URI: https://df-msit-scus-redirect.analysis.windows.net

Status code:

Time: Wed Nov 21 2018 14:27:28 GMT-0800 (Pacific Standard Time)

Version: 13.0.7529.129

 

ossku-azure-vm2 data gateway: Gateway on endpoint 'sb://df-msit-scus-relay3.servicebus.windows.net/9d5d25de-xxxx-xxxx-xxxx-1012e92e2735/' is unreachable.

 

There is no indication which gateways are working or failing to connect. Both gateways are running, status says online and ready, version 3000.0.155.1 Nov 2018. Network port tests succeed on the first gateway but fail (Incomplete failed to connect to network) on the new second gateway.

13 REPLIES 13
burny Frequent Visitor
Frequent Visitor

Re: New OP data gateway unreachable

Oops, clarifying.. I mean there is no indication on the new on prem data gateway software that it's handling queries or not. There is the port test failure and error message indicating its the new cluster member. Both gateways are on the same network and are virtually identical machines.

 

Any help troubleshooting is appreciated!

Community Support Team
Community Support Team

Re: New OP data gateway unreachable

Hi @burny,

 

I try to trace your request id and found it seems like a common gateway connection issue:

The endpoint was not found. There are no listeners connected for the endpoint. Please ensure that you can connect to the internet using HTTP port 80 and TCP port 9350.

 

I'd like to suggest you to Please check power bi gateway service is keep running on your device and firewall white list to confirm it hasn't blocked ports and httprequest.

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin Sheng
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.



For learning resources/Release notes, please visit: | |
burny Frequent Visitor
Frequent Visitor

Re: New OP data gateway unreachable

This machine has no problem reaching the internet. For good measure I've added allow rules for all traffic on port 80 and 9350 and the windows service exe itselft but there is no change in behavior:

1. Data gateway application says online but fails its port test

2. Service reports the same error

3. Another identicaly machine on the same network has no problem

 

Are there troubleshooting steps or logs that could help? Is there a manual analog to the port test?

burny Frequent Visitor
Frequent Visitor

Re: New OP data gateway unreachable

The service appears to be running. 

 

I did notice that my first data gateway machine reports the service starting and stopping to it's own eventlog (On-premises data gateway service.evtx) whereas the new one does not. I dumped logs from the gateway application and found these errors:

DM.EnterpriseGateway Error: 0 : 2018-11-26T20:11:09.9732940Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 806193F3 [DM.EnterpriseGateway] Error getting or creating Windows event log: The source was not found, but some or all event logs could not be searched.  Inaccessible logs: Security.
DM.EnterpriseGateway Error: 0 : 2018-11-26T20:12:40.1324321Z DM.EnterpriseGateway 2bf976d1-a83d-410a-9eb8-33318a12dfe1 00000000-0000-0000-0000-000000000000 MFUC 00000000-0000-0000-0000-000000000000 53D65D6E [DM.Analytics] Performance counters not installed, application performance metrics will not be traced (category = Performance counters specific to the on-premises data gateway)

 

Reinstalling and recovering the gateway did not resolve or change the behavior. Is there a clean install process? How to ensure counters and application logs are installed properly and accessible to the application?

 

I also found these errors in the Configurator log:

=============================

EnterpriseGatewayConfigurator.exe Information: 0 : Initialize EnterpriseGatewayMainViewModel
EnterpriseGatewayConfigurator.exe Information: 0 : GetConfiguratorStateAsync
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SOFTWARE\Microsoft\.NETFramework\v4.0.30319 is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319 is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Exception thrown during GetConfiguration: 'Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClientException: Error retrieving gateway configuration. ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.pipe://localhost/powerbi/gatewayconfiguration/service that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.IO.PipeException: The pipe endpoint 'net.pipe://localhost/powerbi/gatewayconfiguration/service' could not be found on your local machine.
   --- End of inner exception stack trace ---
Server stack trace:
   at System.ServiceModel.Channels.PipeConnectionInitiator.GetPipeName(Uri uri, IPipeTransportFactorySettings transportFactorySettings)
   at System.ServiceModel.Channels.NamedPipeConnectionPoolRegistry.NamedPipeConnectionPool.GetPoolKey(EndpointAddress address, Uri via)
   at System.ServiceModel.Channels.CommunicationPool`2.TakeConnection(EndpointAddress address, Uri via, TimeSpan timeout, TKey& key)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.PowerBI.DataMovement.Pipeline.InternalContracts.IGatewayConfigurationService.GetConfiguration()
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClient.GetConfiguration()
   --- End of inner exception stack trace ---
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClient.GetConfiguration()
   at Microsoft.PowerBI.DataMovement.GatewayUiCommon.ConfiguratorInitializationContextFactory.<GetConfiguratorStateAsync>d__2.MoveNext()'
EnterpriseGatewayConfigurator.exe Information: 0 : Exception thrown during GetConfiguration: 'Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClientException: Error retrieving gateway configuration. ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.pipe://localhost/powerbi/gatewayconfiguration/service that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.IO.PipeException: The pipe endpoint 'net.pipe://localhost/powerbi/gatewayconfiguration/service' could not be found on your local machine.
   --- End of inner exception stack trace ---

 

 

 

=============================

EnterpriseGatewayConfigurator.exe Information: 0 : Local Version 3000.0.155.1
EnterpriseGatewayConfigurator.exe Information: 0 : Create web request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Information: 0 : Create httpClient request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '1f362f8e-d630-4451-9f51-7431f70e0696', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:12:48 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '85c2fed6-4a0c-47f9-a55f-c633e6dc5301', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:14:07 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()

burny Frequent Visitor
Frequent Visitor

Re: New OP data gateway unreachable

I noticed the new data gateway wasn't writing to its own eventlog. I dumped the logs from the data gateway application and found in the error logs this:

 

DM.EnterpriseGateway Error: 0 : 2018-11-26T20:42:10.1361032Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 53D65D6E [DM.Analytics] Performance counters not installed, application performance metrics will not be traced (category = Performance counters specific to the on-premises data gateway)

 

Uninstall and reinstall does not resolve this. Is there a clean up process for a fresh install?

Community Support Team
Community Support Team

Re: New OP data gateway unreachable

HI @burny,

 

According to your error message, it mentioned endpoint listen not found. Can you please go to system 'service' menu to check on-premise gateway service status?

4.PNG

 

BTW, is there any proxy settings apply on your device? You can also check the gateway status(you setup to handle data refresh operation of particular dataset) to confirm if it still online.

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin Sheng
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.



For learning resources/Release notes, please visit: | |
burny Frequent Visitor
Frequent Visitor

Re: New OP data gateway unreachable

@v-shex-msftthanks for helping

 

The windows service has been running without problems reported in the eventlogs:
Annotation 2018-12-05 110105.jpg

 

In fact the gateway eventlog itself appears busted, no messages at all unlike the first gateway:
Annotation 2018-12-05 110312.jpg

 

There are no proxies applied to either gateway:
Annotation 2018-12-05 110533.jpg

 

Community Support Team
Community Support Team

Re: New OP data gateway unreachable

Hi @burny,


I'd like to suggest you submit a support ticket to get better support form power bi team, I can't find the what cause the issue from your description.

Support Ticket.gifsubmit a support ticket

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin Sheng
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.



For learning resources/Release notes, please visit: | |
Anonymous
Not applicable

Re: New OP data gateway unreachable

Hi @burny,

 

Did you solve this problem? I'm facing same problem, where our first gateway does a succesful network ports test and the second newer gateway failes this test. I can't figure out why.

 

Thanks,

Niels

Helpful resources

Announcements
Community Kudopalooza

Win Power BI Swag with Community Kudopalooza!

Each week, complete activities and be qualified in the drawing for cool Power BI Swag.

Summit North America

Power Platform Summit North America

Register by September 5 to save $200

Virtual Launch Event

Microsoft Business Applications Virtual Launch Event

Watch the event on demand for an in-depth look at the new innovations across Dynamics 365 and the Microsoft Power Platform.

MBAS Gallery

Watch Sessions On Demand!

Continue your learning in our online communities.

Top Kudoed Authors
Users Online
Currently online: 258 members 2,576 guests
Please welcome our newest community members: