Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
burny
Frequent Visitor

New OP data gateway unreachable

Our data gateway has been occasionally failing scheduled refreshes with 'gateway unreachable' errors which I'm fairly confident are because the machine was taxed by other data processing tasks, so to remedy I've added a second data gateway to the same cluster and set the cluster to distribute queries. Unfortunately I'm still seeing the 'gateway unreachable' errors and I am not sure if the secondary gateway is participating in the cluster. However now the pbi service gateway page now reports this error:

 

Not all gateway instances within this cluster are connected.

 

Please find more details below about specific errors for each gateway instance.

Activity ID: d5e98a0b-4ddb-42f2-9321-824569cde7a3

Request ID: 7582bc4f-04fa-74f1-fd35-3605dd6e21f6

Cluster URI: https://df-msit-scus-redirect.analysis.windows.net

Status code:

Time: Wed Nov 21 2018 14:27:28 GMT-0800 (Pacific Standard Time)

Version: 13.0.7529.129

 

ossku-azure-vm2 data gateway: Gateway on endpoint 'sb://df-msit-scus-relay3.servicebus.windows.net/9d5d25de-xxxx-xxxx-xxxx-1012e92e2735/' is unreachable.

 

There is no indication which gateways are working or failing to connect. Both gateways are running, status says online and ready, version 3000.0.155.1 Nov 2018. Network port tests succeed on the first gateway but fail (Incomplete failed to connect to network) on the new second gateway.

13 REPLIES 13
v-shex-msft
Community Support
Community Support

Hi @burny,

 

I try to trace your request id and found it seems like a common gateway connection issue:

The endpoint was not found. There are no listeners connected for the endpoint. Please ensure that you can connect to the internet using HTTP port 80 and TCP port 9350.

 

I'd like to suggest you to Please check power bi gateway service is keep running on your device and firewall white list to confirm it hasn't blocked ports and httprequest.

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

This machine has no problem reaching the internet. For good measure I've added allow rules for all traffic on port 80 and 9350 and the windows service exe itselft but there is no change in behavior:

1. Data gateway application says online but fails its port test

2. Service reports the same error

3. Another identicaly machine on the same network has no problem

 

Are there troubleshooting steps or logs that could help? Is there a manual analog to the port test?

burny
Frequent Visitor

I noticed the new data gateway wasn't writing to its own eventlog. I dumped the logs from the data gateway application and found in the error logs this:

 

DM.EnterpriseGateway Error: 0 : 2018-11-26T20:42:10.1361032Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 53D65D6E [DM.Analytics] Performance counters not installed, application performance metrics will not be traced (category = Performance counters specific to the on-premises data gateway)

 

Uninstall and reinstall does not resolve this. Is there a clean up process for a fresh install?

HI @burny,

 

According to your error message, it mentioned endpoint listen not found. Can you please go to system 'service' menu to check on-premise gateway service status?

4.PNG

 

BTW, is there any proxy settings apply on your device? You can also check the gateway status(you setup to handle data refresh operation of particular dataset) to confirm if it still online.

 

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.

@v-shex-msftthanks for helping

 

The windows service has been running without problems reported in the eventlogs:
Annotation 2018-12-05 110105.jpg

 

In fact the gateway eventlog itself appears busted, no messages at all unlike the first gateway:
Annotation 2018-12-05 110312.jpg

 

There are no proxies applied to either gateway:
Annotation 2018-12-05 110533.jpg

 

Hi @burny,


I'd like to suggest you submit a support ticket to get better support form power bi team, I can't find the what cause the issue from your description.

submit a support ticketsubmit a support ticket

Regards,

Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider accept as solution to help other members find it more quickly.
Anonymous
Not applicable

Hi @burny,

 

Did you solve this problem? I'm facing same problem, where our first gateway does a succesful network ports test and the second newer gateway failes this test. I can't figure out why.

 

Thanks,

Niels

@burny, @Niels: quite the same problem here:

-2 virtual servers, Win2016 DataCenter, .Net 4.7.2, latest updates;

-GatewayInstaller V3000.0.265, Cluster installation according to Microsoft Docs

 

First node is fine for the network test, second node fails network test. Both have problems with the event log and with the performance counters. From a functional perspective the cluster seems ok (tests using PowerBI and an Oracle connector report successfull connection to both nodes of the cluster).
Just opened a support ticket.

Any news or hints from your side are welcome. Thanks,
Mike

Update:

Answer from product group regarding the network test issues:

“We have a bug in the gateway network port test and it will be resolved in the next gateway release (Mid Feb)

This failure will not have any impact on the communication and processing of request for Logic Apps. “

 

Still waiting some clarification regarding performance counter errors.

Mike

Update:

Answer from product group regarding performance counter errors:

 

"...failure to create perf counter won't affect gateway functionality. we already have an item to trace this to get it fixed. but customer shouldn't be affected by this."

 

Mike

I am encountering same issue.  The gateway program shows port test failure, but functionally I can successfully refresh PBI Service datasets using the gateway.  I wonder if MS has applied a fix so far?  I am using the March version of gateway.  

burny
Frequent Visitor

The service appears to be running. 

 

I did notice that my first data gateway machine reports the service starting and stopping to it's own eventlog (On-premises data gateway service.evtx) whereas the new one does not. I dumped logs from the gateway application and found these errors:

DM.EnterpriseGateway Error: 0 : 2018-11-26T20:11:09.9732940Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 806193F3 [DM.EnterpriseGateway] Error getting or creating Windows event log: The source was not found, but some or all event logs could not be searched.  Inaccessible logs: Security.
DM.EnterpriseGateway Error: 0 : 2018-11-26T20:12:40.1324321Z DM.EnterpriseGateway 2bf976d1-a83d-410a-9eb8-33318a12dfe1 00000000-0000-0000-0000-000000000000 MFUC 00000000-0000-0000-0000-000000000000 53D65D6E [DM.Analytics] Performance counters not installed, application performance metrics will not be traced (category = Performance counters specific to the on-premises data gateway)

 

Reinstalling and recovering the gateway did not resolve or change the behavior. Is there a clean install process? How to ensure counters and application logs are installed properly and accessible to the application?

 

I also found these errors in the Configurator log:

=============================

EnterpriseGatewayConfigurator.exe Information: 0 : Initialize EnterpriseGatewayMainViewModel
EnterpriseGatewayConfigurator.exe Information: 0 : GetConfiguratorStateAsync
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SOFTWARE\Microsoft\.NETFramework\v4.0.30319 is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Value SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319 is not defined in registry.
EnterpriseGatewayConfigurator.exe Information: 0 : Exception thrown during GetConfiguration: 'Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClientException: Error retrieving gateway configuration. ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.pipe://localhost/powerbi/gatewayconfiguration/service that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.IO.PipeException: The pipe endpoint 'net.pipe://localhost/powerbi/gatewayconfiguration/service' could not be found on your local machine.
   --- End of inner exception stack trace ---
Server stack trace:
   at System.ServiceModel.Channels.PipeConnectionInitiator.GetPipeName(Uri uri, IPipeTransportFactorySettings transportFactorySettings)
   at System.ServiceModel.Channels.NamedPipeConnectionPoolRegistry.NamedPipeConnectionPool.GetPoolKey(EndpointAddress address, Uri via)
   at System.ServiceModel.Channels.CommunicationPool`2.TakeConnection(EndpointAddress address, Uri via, TimeSpan timeout, TKey& key)
   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.PowerBI.DataMovement.Pipeline.InternalContracts.IGatewayConfigurationService.GetConfiguration()
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClient.GetConfiguration()
   --- End of inner exception stack trace ---
   at Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClient.GetConfiguration()
   at Microsoft.PowerBI.DataMovement.GatewayUiCommon.ConfiguratorInitializationContextFactory.<GetConfiguratorStateAsync>d__2.MoveNext()'
EnterpriseGatewayConfigurator.exe Information: 0 : Exception thrown during GetConfiguration: 'Microsoft.PowerBI.DataMovement.Pipeline.GatewayClient.GatewayConfigurationClientException: Error retrieving gateway configuration. ---> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.pipe://localhost/powerbi/gatewayconfiguration/service that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.IO.PipeException: The pipe endpoint 'net.pipe://localhost/powerbi/gatewayconfiguration/service' could not be found on your local machine.
   --- End of inner exception stack trace ---

 

 

 

=============================

EnterpriseGatewayConfigurator.exe Information: 0 : Local Version 3000.0.155.1
EnterpriseGatewayConfigurator.exe Information: 0 : Create web request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Information: 0 : Create httpClient request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '1f362f8e-d630-4451-9f51-7431f70e0696', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:12:48 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '85c2fed6-4a0c-47f9-a55f-c633e6dc5301', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:14:07 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()

burny
Frequent Visitor

Oops, clarifying.. I mean there is no indication on the new on prem data gateway software that it's handling queries or not. There is the port test failure and error message indicating its the new cluster member. Both gateways are on the same network and are virtually identical machines.

 

Any help troubleshooting is appreciated!

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors