Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!
We are reading data from a REST API. The process is fairly common: do one call to get a list of IDs, then do one or more calls for each ID to get the details. For a small data set, everything works fine. With a large number of IDs, the number of REST calls arriving at the server simulateously is extremely high. This causes the server to return 502/503 errors and the data loading fails.
With other languages, a typical solution is to either make the calls sequentially instead of in parallel, or to rate-limit the calls in some way. The reduces the onslaught of requests to the server and allows the server to answer the requests instead of rejecting them. Is there a way to accomplish this with Power Query?
Regards,
G
Solved! Go to Solution.
You can use Function.InvokeAfter to introduce a delay between requests. This code has a 2 second delay between web requests
let
id_list = {1 .. 5},
GetWeb = (id) => Json.Document(Web.Contents("https://jsonplaceholder.typicode.com/todos/" & Text.From(id))),
Output = List.Transform(id_list, each Function.InvokeAfter( ()=>GetWeb(_), #duration(0,0,0,2)))
in
Output
regards
Phil
If I answered your question please mark my post as the solution.
If my answer helped solve your problem, give it a kudos by clicking on the Thumbs Up.
Proud to be a Super User!
You can use Function.InvokeAfter to introduce a delay between requests. This code has a 2 second delay between web requests
let
id_list = {1 .. 5},
GetWeb = (id) => Json.Document(Web.Contents("https://jsonplaceholder.typicode.com/todos/" & Text.From(id))),
Output = List.Transform(id_list, each Function.InvokeAfter( ()=>GetWeb(_), #duration(0,0,0,2)))
in
Output
regards
Phil
If I answered your question please mark my post as the solution.
If my answer helped solve your problem, give it a kudos by clicking on the Thumbs Up.
Proud to be a Super User!
Beautiful!
User | Count |
---|---|
140 | |
113 | |
104 | |
77 | |
65 |
User | Count |
---|---|
135 | |
116 | |
101 | |
71 | |
61 |