Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
jkbrookes
Frequent Visitor

Getting data from web api in parallel

Hello, first time asking a question here so thank you in advance for any help.

 

I've got a custom m-script that is authenticating to our monitoring platform's api using oauth.  It uses their search api to pull a list of every device we are monitoring.  Following some hints I got online I am able to pull the multiple pages of items one at a time.

 

The platform then has a "get device" api where I can get further details about each individual device.  I want to do this for each device that is returned in the above search but ideally that will be done in parallel as that search can return 8000-10000 devices (and thus 8000-10000 calls to the get device api).

Assuming I could do this in parallel there would be nothing stopping me from pulling the search pages in parallel as well.

How can I run these api calls in parallel with each other?  I believe I could set up separate queries if I knew the number of devices and pages that would be returned, but those things will be very dynamic.

2 REPLIES 2
Eric_Zhang
Employee
Employee

@jkbrookes

 

I think you can save the device api url in one column and add an extra column with Web.Contents.

Check this similar thread.

So I added the extra column calling the function which calls Web.Contents and I pass the url to that function:

#"Added Custom" = Table.AddColumn(DeviceTable, "Records", each GetAPI(#"Base URL" & [uniqueId] & "/devices/" & [id])),

 

DeviceTable is generated from some other api calls and can contain anywhere from 8000-10000 rows in it.  It still looks like the GetAPI function is being called one at a time, row by row, waiting for each row's GetAPI function call to finish before moving on to the next row.

What I'd like to happen is have all of those rows run their respective GetAPI function at the same time (or at least call them in larger chunks than one at a time).  Is this possible?  Am I calling something wrong?

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors