cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
ChadC
Frequent Visitor

Help with List.Generate() in making API calls

Hi All, 

 

I'm building a custom connector to utilize an API. I'm attempting to handle pagination, but am having a hard time. Here's what I'm trying to do:

  1. An call is made to the API.
  2. A response is returned.
  3. The connector needs to check if the response contains a header called "Continuation."
  4. If true, another call needs to be made with the previous "Continuation" header included.
  5. If false, return the results.

I'm trying to use List.Generate because it appears to be my best option for a Do...While/Until loop. Here's what I have:

 

Main = (Scope) =>
    let
        HasContToken = (responseHeaders) as logical => if (Record.HasFields(responseHeaders, {"Continuation"})) then 
                    true
                 else
                    false,
        GetPage = (Scope, ContinuationToken) =>
            let
                url = api_source_uri & Scope,
                RawData = Web.Contents(url, [
                    Headers = [
                        #"Continuation" = ContinuationToken
                   ]])
            in
                RawData,
        Pages = List.Generate(() => Web.Contents(api_source_uri & Scope),
                each HasContToken(Value.Metadata(_)[Headers]),
                each GetPage(Scope, Value.Metadata(_)[Headers][Continuation])),
        Pages_JSON = Json.Document(Pages),
        Table = Table.FromList(Pages_JSON, Splitter.SplitByNothing())
    in
        Table;

 

 I receive this error when trying to run this code: "The parameter is expected to be of type Text.Type or Binary.Type."

 

I don't know which parameter this is referring to, but I assume it is related to how I've set up List.Generate. Any help is greatly appreciated.

 

Thank you!

1 ACCEPTED SOLUTION
ChadC
Frequent Visitor

Hi @v-xicai ,

 

Thank you for your response. The issue was indeed related to null values being returned in subsequent calls. However, this was because the API I'm using required an "accept" header to be included, which I had overlooked. Below is my updated code:

 

let
        HasContToken = (responseHeaders) as logical => if (Record.HasFields(responseHeaders[Headers], {"Continuation"})) then 
                    true
                 else
                    false,
        GetPage = (Scope, ContinuationToken) =>
            let
                RawData = Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Query = [
                        limit = "1000"
                    ],
                    Headers = [
                        #"Continuation" = ContinuationToken,
                        accept = "application/json"
                   ]])
            in
                RawData,
        Pages = List.Generate(() => Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Headers = [
                        accept = "application/json"
                    ],
                    Query = [
                        limit = "1000"
                    ]]),
                each HasContToken(Value.Metadata(_)),
                each GetPage(Scope, Value.Metadata(_)[Headers][Continuation])),
        JSON = List.Transform(Pages, (_) => Json.Document(_)),
        List = List.Combine(JSON),
        Table = Table.FromList(List, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
    in
        Table;

View solution in original post

2 REPLIES 2
v-xicai
Community Support
Community Support

Hi @ChadC ,

 

When combining files, be sure to choose "Skip files with errors." see more: https://www.sqlchick.com/entries/2018/5/6/querying-data-in-azure-data-lake-store-with-power-bi.

 

There may be some nulls in your dataset which generated the error when the parsing was attempted, replace the null values with an empty structure "<div></div>" which was then parsed . See The parameter is expected to be of type Text.Type or Binary.Type.

 

If you still have this issue for Power BI, you'd better create a support ticket in Power BI Support , Scroll down and click "CREATE SUPPORT TICKET", to get further help.

 

Best Regards,

Amy

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

ChadC
Frequent Visitor

Hi @v-xicai ,

 

Thank you for your response. The issue was indeed related to null values being returned in subsequent calls. However, this was because the API I'm using required an "accept" header to be included, which I had overlooked. Below is my updated code:

 

let
        HasContToken = (responseHeaders) as logical => if (Record.HasFields(responseHeaders[Headers], {"Continuation"})) then 
                    true
                 else
                    false,
        GetPage = (Scope, ContinuationToken) =>
            let
                RawData = Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Query = [
                        limit = "1000"
                    ],
                    Headers = [
                        #"Continuation" = ContinuationToken,
                        accept = "application/json"
                   ]])
            in
                RawData,
        Pages = List.Generate(() => Web.Contents(api_source_uri, [
                    RelativePath = "/reporting/v1/" & Scope,
                    Headers = [
                        accept = "application/json"
                    ],
                    Query = [
                        limit = "1000"
                    ]]),
                each HasContToken(Value.Metadata(_)),
                each GetPage(Scope, Value.Metadata(_)[Headers][Continuation])),
        JSON = List.Transform(Pages, (_) => Json.Document(_)),
        List = List.Combine(JSON),
        Table = Table.FromList(List, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
    in
        Table;

View solution in original post

Helpful resources

Announcements
MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

July 2021 Update 768x460.png

Check it out!

Click here to read more about the July 2021 Updates

Power Query PA Forum 768x460.png

Check it out!

Did you know that you can visit the Power Query Forum in Power BI and now Power Apps

Urdu Hindi D365 Bootcamp 768x460.png

Urdu Hindi D365 Bootcamp

Dont miss our very own April Dunnam’s The Developer Guide to the Galaxy! Find out what the Power Platform has to offer for the traditional developer.

Top Solution Authors
Top Kudoed Authors