All our refreshes failing this monrning in central US at 8 am

[MarkPalmberg]

Every single one. MSFT Power BI status page says, of course, that everything is up and running (as of this writing).

[nicg]

Having a similar problem on our Gen2 P1 Premium node in North Central US. Dozens of datasets have been stuck in a "hung" state where the UI just shows them as spinning endlessly. Issue appears to have started around 6AM eastern time.

[MarkPalmberg]

Here's the error code from one of the refreshes:

Data source error{"error":{"code":"DM_GWPipeline_Gateway_InvalidConnectionCredentials","pbi.error":{"code":"DM_GWPipeline_Gateway_InvalidConnectionCredentials","parameters":{},"details":[{"code":"DM_ErrorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"-2147467259"}},{"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"The credentials provided for the SQL source are invalid. (Source at SERVER.)"}},{"code":"DM_ErrorDetailNameCode_UnderlyingHResult","detail":{"type":1,"value":"-2147467259"}},{"code":"Microsoft.Data.Mashup.CredentialError.DataSourceKind","detail":{"type":1,"value":"SQL"}},{"code":"Microsoft.Data.Mashup.CredentialError.DataSourcePath","detail":{"type":1,"value":"SERVER"}},{"code":"Microsoft.Data.Mashup.CredentialError.Reason","detail":{"type":1,"value":"AccessUnauthorized"}},{"code":"Microsoft.Data.Mashup.MashupSecurityException.DataSources","detail":{"type":1,"value":"[{\"kind\":\"SQL\",\"path\":\"SERVER\"}]"}},{"code":"Microsoft.Data.Mashup.MashupSecurityException.Reason","detail":{"type":1,"value":"AccessUnauthorized"}}],"exceptionCulprit":1}}} Table: UserPermission.
Cluster URIWABI-US-NORTH-CENTRAL-G-PRIMARY-redirect.analysis.windows.net
Activity ID7326e067-1cfb-452d-b9f2-4a1ff2947d61
Request IDfc304105-6184-2ca7-ac93-d240a00f4752
Time2022-02-03 16:12:51Z

[nicg]

Updating the gateway from Dec 2021 release 1 to Dec 2021 release 2 appears to have resolved the issue for us.

[MarkPalmberg]

More details: We're wondering now if maybe this is a system user credentials error on our end. All the refreshes that were pointing to one of our SQL servers failed, but those pointing to a different server -- using the same gateway -- refreshed OK. Which is to say, in part, that the gateway version is probably not causing our issue, @nicg . I'm glad you got your issue figured out!

[MarkPalmberg]

Our issue was resolved by resetting the user credentials for the data source that was giving us the errors. But now we're at a loss to determine how or why those credentials "went bad" overnight. I'm not the gateway admin so don't know the means by which we can see the user credentials for each data source. All I know from my chair is "(admin has granted access, credentials are not required)"

[v-cazheng-msft]

Issue is resolved with updating credential of data source.