Paginated API calls for Workfront with an unknown number of pages (looping until no more data)

[Anonymous]

Hi,

I've read quite a few threads, on this topic but due to the variety of approaches and different APIs and their pagination methods, it's hard to work out what will best apply to my scenario with Workfront. Hoping that a guru out there can help!
 
In summary, Workfront's API operates where you specify the starting record number, and the number of records to return.  The limit is 2000 records per query.  
 
An example query for the first 2000 task records is:  

 
The next two pages of records are as follows:

 
It is also possible to do a count, in case this is useful in building a loop-based query solution.
 
Up until now, I have created the power-query the dumb way by pre-defining (e.g.) a list of 50 queries of 2000 records per query (because I know that 100000 is enough to safely retrieve all records), each with incremented starting positions, and it runs through all these and converts them into tables, combines the tables, then I discard any duplicates and I have usable data to report on.  This works OK, but I have to periodically check the counts on the various queries and make sure there are enough lines in the query so that I am not missing data.  I am sure the last x number of queries which return no data also slow down the refresh process and waste time.  This is not sustainable long term and I would like to have a query that will keep looping through until it receives no data.
 
1000 points of good karma and many thanks to anyone who can help!!!

Cheers, David

[mahoneypat]

Here is a video that shows a good way to approach this one.

Power BI - Tales From The Front - REST APIs - YouTube

Pat

[apatkar]

If someone stumbles upon this while searching for a solution, you can do a separate API call with /task/report to get a total count. Then divide that total by 2000 to figure out how many times you need to loop.

[Anonymous]

HI david, did you find a better solution? I am working on the same requirement. I need to get a solution asap. kindly let me know if you have any solution. Thanks in advance

[Anonymous]

No, I haven't had time to experiment yet. Still hoping someone can give us some tips in the meantime.

[Anonymous]

Hey David,

Did you ever figure this out?  I have a similar set of requirements to get data from Workfront and would appreciate a starting place.

Thanks,

Josh

[Anonymous]

HI david, did you find a better solution? I am working on the same requirement. I need to get a solution asap. kindly let me know if you have any solution. Thanks in advance

[quentin_vigne]

Hi @Anonymous

We can't acess your link

"Not to worry Workfront is currently active. You have typed in an invalid domain. To get the login screen please use the correct domain name for your company. If you need assistance in finding out which domain you are supposed to use please "

Quentin

[Anonymous]

Hi - I've recreated the example queries.  Hoping for any assistance. Thanks