Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!
Our data gateway has been occasionally failing scheduled refreshes with 'gateway unreachable' errors which I'm fairly confident are because the machine was taxed by other data processing tasks, so to remedy I've added a second data gateway to the same cluster and set the cluster to distribute queries. Unfortunately I'm still seeing the 'gateway unreachable' errors and I am not sure if the secondary gateway is participating in the cluster. However now the pbi service gateway page now reports this error:
Not all gateway instances within this cluster are connected.
Please find more details below about specific errors for each gateway instance.
Activity ID: d5e98a0b-4ddb-42f2-9321-824569cde7a3
Request ID: 7582bc4f-04fa-74f1-fd35-3605dd6e21f6
Cluster URI: https://df-msit-scus-redirect.analysis.windows.net
Status code:
Time: Wed Nov 21 2018 14:27:28 GMT-0800 (Pacific Standard Time)
Version: 13.0.7529.129
ossku-azure-vm2 data gateway: Gateway on endpoint 'sb://df-msit-scus-relay3.servicebus.windows.net/9d5d25de-xxxx-xxxx-xxxx-1012e92e2735/' is unreachable.
There is no indication which gateways are working or failing to connect. Both gateways are running, status says online and ready, version 3000.0.155.1 Nov 2018. Network port tests succeed on the first gateway but fail (Incomplete failed to connect to network) on the new second gateway.
Hi @burny,
I try to trace your request id and found it seems like a common gateway connection issue:
The endpoint was not found. There are no listeners connected for the endpoint. Please ensure that you can connect to the internet using HTTP port 80 and TCP port 9350.
I'd like to suggest you to Please check power bi gateway service is keep running on your device and firewall white list to confirm it hasn't blocked ports and httprequest.
Regards,
Xiaoxin Sheng
This machine has no problem reaching the internet. For good measure I've added allow rules for all traffic on port 80 and 9350 and the windows service exe itselft but there is no change in behavior:
1. Data gateway application says online but fails its port test
2. Service reports the same error
3. Another identicaly machine on the same network has no problem
Are there troubleshooting steps or logs that could help? Is there a manual analog to the port test?
I noticed the new data gateway wasn't writing to its own eventlog. I dumped the logs from the data gateway application and found in the error logs this:
DM.EnterpriseGateway Error: 0 : 2018-11-26T20:42:10.1361032Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 53D65D6E [DM.Analytics] Performance counters not installed, application performance metrics will not be traced (category = Performance counters specific to the on-premises data gateway)
Uninstall and reinstall does not resolve this. Is there a clean up process for a fresh install?
HI @burny,
According to your error message, it mentioned endpoint listen not found. Can you please go to system 'service' menu to check on-premise gateway service status?
BTW, is there any proxy settings apply on your device? You can also check the gateway status(you setup to handle data refresh operation of particular dataset) to confirm if it still online.
Regards,
Xiaoxin Sheng
@v-shex-msftthanks for helping
The windows service has been running without problems reported in the eventlogs:
In fact the gateway eventlog itself appears busted, no messages at all unlike the first gateway:
There are no proxies applied to either gateway:
Hi @burny,
I'd like to suggest you submit a support ticket to get better support form power bi team, I can't find the what cause the issue from your description.
Regards,
Xiaoxin Sheng
Hi @burny,
Did you solve this problem? I'm facing same problem, where our first gateway does a succesful network ports test and the second newer gateway failes this test. I can't figure out why.
Thanks,
Niels
@burny, @Niels: quite the same problem here:
-2 virtual servers, Win2016 DataCenter, .Net 4.7.2, latest updates;
-GatewayInstaller V3000.0.265, Cluster installation according to Microsoft Docs
First node is fine for the network test, second node fails network test. Both have problems with the event log and with the performance counters. From a functional perspective the cluster seems ok (tests using PowerBI and an Oracle connector report successfull connection to both nodes of the cluster).
Just opened a support ticket.
Any news or hints from your side are welcome. Thanks,
Mike
Update:
Answer from product group regarding the network test issues:
“We have a bug in the gateway network port test and it will be resolved in the next gateway release (Mid Feb)
This failure will not have any impact on the communication and processing of request for Logic Apps. “
Still waiting some clarification regarding performance counter errors.
Mike
Update:
Answer from product group regarding performance counter errors:
"...failure to create perf counter won't affect gateway functionality. we already have an item to trace this to get it fixed. but customer shouldn't be affected by this."
Mike
I am encountering same issue. The gateway program shows port test failure, but functionally I can successfully refresh PBI Service datasets using the gateway. I wonder if MS has applied a fix so far? I am using the March version of gateway.
The service appears to be running.
I did notice that my first data gateway machine reports the service starting and stopping to it's own eventlog (On-premises data gateway service.evtx) whereas the new one does not. I dumped logs from the gateway application and found these errors:
DM.EnterpriseGateway Error: 0 : 2018-11-26T20:11:09.9732940Z DM.EnterpriseGateway 00000000-0000-0000-0000-000000000000 00000000-0000-0000-0000-000000000000 0000 00000000-0000-0000-0000-000000000000 806193F3 [DM.EnterpriseGateway] Error getting or creating Windows event log: The source was not found, but some or all event logs could not be searched. Inaccessible logs: Security.
DM.EnterpriseGateway Error: 0 : 2018-11-26T20:12:40.1324321Z DM.EnterpriseGateway 2bf976d1-a83d-410a-9eb8-33318a12dfe1 00000000-0000-0000-0000-000000000000 MFUC 00000000-0000-0000-0000-000000000000 53D65D6E [DM.Analytics] Performance counters not installed, application performance metrics will not be traced (category = Performance counters specific to the on-premises data gateway)
Reinstalling and recovering the gateway did not resolve or change the behavior. Is there a clean install process? How to ensure counters and application logs are installed properly and accessible to the application?
I also found these errors in the Configurator log:
=============================
=============================
EnterpriseGatewayConfigurator.exe Information: 0 : Local Version 3000.0.155.1
EnterpriseGatewayConfigurator.exe Information: 0 : Create web request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Information: 0 : Create httpClient request for Uri: http://www.msftncsi.com/ncsi.txt
EnterpriseGatewayConfigurator.exe Information: 0 : Response status: OK
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '1f362f8e-d630-4451-9f51-7431f70e0696', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:12:48 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()
EnterpriseGatewayConfigurator.exe Error: 0 : Unable to Get ServiceBusRelaysInfo, ex: Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIHttpRequestException: Request error: Id: '85c2fed6-4a0c-47f9-a55f-c633e6dc5301', Status code: 'Unauthorized', Timestamp: '11/26/2018 20:14:07 +00:00', ErrorCode: 'PowerBINotAuthorizedException', Detail: ''
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<EnsureSuccessfulDMMSRequest>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.ExternalClient.PowerBIDataMovementClientExtensions.<GetServiceBusRelaysInfoAsync>d__56.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.PowerBI.DataMovement.GatewayCommon.ServiceBusPortsTest.ServiceBusPortsTestNetwork.<RequestRelays>d__0.MoveNext()
Oops, clarifying.. I mean there is no indication on the new on prem data gateway software that it's handling queries or not. There is the port test failure and error message indicating its the new cluster member. Both gateways are on the same network and are virtually identical machines.
Any help troubleshooting is appreciated!