Custom Connector OAuth2 Scheduled Refresh Issue

[hofax]

I have developed a custom connector that uses OAuth2 "authorization_code" flow. Everything works fine in Power BI Desktop and I can get authorized and load data.

I then installed the On-premise data gateway (personal mode) and enabled custom connectors. My connector shows up and after publishing a report that uses this connector I was able to edit the data source credentials and signin via the connector. The sign in completed successfully and everything appears to be okay.

I then click refresh now and it attempts to refresh the data set, but after only 10-20 seconds it has failed with the following (It wouldn't let me insert an image to I copied the relevant information):

There was an error in the data gateway.

Underlying error message: Object reference not set to an instance of an object. Table: Selected Properties.
DM_ErrorDetailNameCode_UnderlyingHResult: -2147467261

Selected Properties is one of my queries, but which query it fails on varies between refresh attempts so I assume it is failing on whichever query executes/completes first.

Any help as to what this could be would be greatly appreciated!

UPDATE: For anyone still having issues with scheduled refreshes of custom connectors using OAuth, I wrote a quick piece here that may be useful.

[janinebp]

I'm dealing with the same issues. Here it says: "OAuth isn't a supported authentication scheme with the on-premises data gateway. You can't add data sources that require OAuth. If your dataset has a data source that requires OAuth, you can't use the gateway for scheduled refresh."

[hofax]

@janinebp  I believe you may need to use the Personal Gateway for custom connectors to work. Presumably because it needs access to the connector which is located on your personal machine.

[Anonymous]

I managed to make it work, it was a problem in one of my functions (I will check if I remember which one)
And I can confirm that Custom Connectors do refresh through the Gateway (no need to be the personal one).

EDIT: Found it. I implemented a logic developed by Matt Mason in the TokenMethod function. After this change, it worked.

TokenMethod = (grantType, code) =>
      let
                query = [
                           grant_type = grantType,
                           client_id = client_id,
                           client_secret = client_secret,
                           redirect_uri = redirect_uri
                             ],

// Code from Matt Mason. Check https://www.mattmasson.com/
                  queryWithCode = if (grantType = "refresh_token") then [ refresh_token = code ] else [code = code],

                  response = Web.Contents(token_uri, [
                  Content = Text.ToBinary(Uri.BuildQueryString(query & queryWithCode)),
                  Headers=[#"Content-type" = "application/x-www-form-urlencoded",#"Accept" = "application/json"],                      ManualStatusHandling = {400}]),
                  body = Json.Document(response),
                  result = if (Record.HasFields(body, {"error", "error_description"}))

                               then error Error.Record(body[error], body[error_description], body)
                              else body
      in
                  result;

Refresh = (resourceUrl, refresh_token) => TokenMethod("refresh_token", refresh_token);

[vsingh23]

@Anonymous ,
Could you please help me with the entire code that worked in your case? 

[hofax]

@Anonymous Thanks for the reponse, the issue I'm experiencing is actually after I get authenticated and attempt to refresh the data set.

However!! 

You did help me identify a bug in my token refresh, so thank you very much. I was doing something very similar in regards to switching the parameter name depending on whether it was an access token request or token refresh. Turns out the documentation was lacking and I had the wrong name for the refresh token tokenField parameter.

Refresh = (resourceUrl, refresh_token) => TokenMethod(resourceUrl, "refresh_token", "refresh_token", refresh_token);

TokenMethod = (resourceUrl, grantType, tokenField, code) =>
    let
       queryString = [
           grant_type = grantType,
           redirect_uri = redirect_uri,
           client_id = GetEnvironmentVariable(resourceUrl, "client_id"),
           client_secret = GetEnvironmentVariable(resourceUrl, "client_secret")
    ],
    queryWithCode = Record.AddField(queryString, tokenField, code),

    base_url = GetAuthUrl(resourceUrl),
    tokenResponse = Web.Contents(base_url & "connect/token", [
               Content = Text.ToBinary(Uri.BuildQueryString(queryWithCode)),
               Headers = [
                    #"Content-type" = "application/x-www-form-urlencoded",
                    #"Accept" = "application/json"
     ],
     ManualStatusHandling = {400}
   ]),
    body = Json.Document(tokenResponse),
    result = if (Record.HasFields(body, {"error", "error_description"})) then
            error Error.Record(body[error], body[error_description], body)

    else if (Record.HasFields(body, {"error"})) then
            error Error.Record(body[error], "Failed to login.", body)
    else
            body
in
     result;

Cheers!

[hofax]

For anyone still having issues with scheduled refreshes of custom connectors using OAuth, I wrote a quick piece here that may be useful.

[Anonymous]

Glad I could help somehow!
Cheers.

[Anonymous]

Any luck with this?

[hofax]

No luck, I tried again today and now after clicking "Edit Credentials" on the customer custom the window opens and I can sign in. But after the redirect the window closes and just sits in the edit credentials window.

Closing that window and clicking edit credentials again breifly pops open our sign in page which immediately closes, which I assume means that I already have a valid token and am being redirected.