How to overcome API call row limit?

[h4tt3n]

Hello,

I am attempting to extract measurement data from an API with a limit of 1000 rows per call. Each row consists of a data value and a timestamp. I have set up a Power Query script that connects to the API and can retrieve any 1000 concecutive rows of data. 

What I need is an iterative script that retrieves the data I need in chunks of 1000 rows and merges them ino a single table. Below is my current script. The Source line is the one of interest here.

let
    Source = Json.Document(Web.Contents(""&url&"/TagsData/"&tagid&"/After?subscription-key="&key&"&after="&after&"&count="&count&"")),
    #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"tag", "data"}, {"tag", "data"}),
    #"Removed Columns" = Table.RemoveColumns(#"Expanded Column1",{"tag"}),
    #"Expanded data" = Table.ExpandListColumn(#"Removed Columns", "data"),
    #"Expanded data1" = Table.ExpandRecordColumn(#"Expanded data", "data", {"d", "v"}, {"d", "v"}),
    #"Changed Type" = Table.TransformColumnTypes(#"Expanded data1",{{"d", type datetime}, {"v", type number}})
in
    #"Changed Type"

 The body parameter "after" sets the datetime after which you want data from. The parameter "count" sets the number of rows, and is limited to 1000.

One challenge is that the API is not paginated, so I will need to keep track of timestamps and number of returned rows for each call.

I have been working with PowerBI professionally for the last year, but my Power Query scripting skills are not good enough for this challenge. Thanks in advance for any help.

Cheers, Mike

[JosephDodd]

@mahoneypat Hey Pat, can't you use the theories and code in this community post to update the Salesforce Report connector to automatically call the next set of rows until it returns blank?

[mahoneypat]

Instead of pagination, you can also use List.Numbers along with Skip or Offset in the web calls to overcome the row limit.  This video walks through it.

Power BI - Tales From The Front - REST APIs - YouTube

Regards,

Pat

[DeeJo]

@mahoneypat Thanks for that wonderful video. I have implemented skip and offset method while pulling data from API. Earlier it used to limit 100 records but using this I am able to load complete data. Next challenge is refresh doesn't work in Power BI Service. Returns error saying This dataset contains dynamic data source. 

Any suggestions so that refresh works..Please note I already tried approach mentioned in below link

https://blog.crossjoin.co.uk/2016/08/23/web-contents-m-functions-and-dataset-refresh-errors-in-power...

but then after adding skip and top as query parameters.. it pulls same set of records everytime.. so this is not working for me. Do you suggest any other option which will resolve dynamic data source issue in Service?

[Jimmy801]

Hello @h4tt3n 

there are a lot of examples of how to do in this forum.

They use all List.Generate where you have a parameter when to stop to generate to api calls like giving an error or empty page.

just start out to search for api List.Generate.

If you have any specific question feel free to come back

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy