Implementing "Value.WaitFor" in PowerBI DataFlow

[Anonymous]

Hi experts!

In my dataflow that consumes a REST API I come across HTTP 429 "Too many requests" errors which break the dataflow refresh.

I have copied all queries to my PowerBI Desktop and implemented the 'Value.WaitFor' to overcome the issues with the 'HTTP 429' Status.

Finally, I used the code supplied here: https://gist.github.com/CurtHagenlocher/68ac18caa0a17667c805 

So, now I have moved the queries back to the dataflow, The preview of the Queries works fine. 

But when I am Saving the DataFlow, the message

Can't save dataflow

One or more entities references a dynamic data source.

I really hope that someone here has a workaround or a proper solution. This is a major blocker for us.

Thanks for reading!

Best,

Denis

[iBusinessBI]

Have you find a solution? Same issue.

[v-easonf-msft]

Hi , @Anonymous 

Please check whether the solution in this document coild help:

Issues-with-using-a-parameter-in-web-content-url-in-dataflow 

Best Regards,
Community Support Team _ Eason

[Anonymous]

Hi @v-easonf-msft !

Thanks but I am already using the RelativePath and Query options together with Web.Contents. 

That's my big frustration. The base URL is always the same (server name) but still, the error suggests that I am using a dynamic URL.

As you can see in my screenshot, the Web.Contents method always gets the same parameter value:

(url as text, optional options as record) => 

let
    WaitFor = 
        (producer as function, interval as function, optional count as number) as any =>
                let
                    list = List.Generate(
                        () => {0, null},
                        (state) => state{0} <> null and (count = null or state{0} < count),
                        (state) => if state{1} <> null
                            then {null, state{1}}
                            else {1 + state{0}, Function.InvokeAfter(() => producer(state{0}), interval(state{0}))},
                        (state) => state{1})
                in
                    List.Last(list)

in

WaitFor(
        (i) =>
            let
                options2 = if options = null then [] else options,
                options3 = if i=0 then options2 else options2 & [IsRetry=true],
                result = Web.Contents(#"Jira Endpoint", options3 & [ManualStatusHandling={429}]),
                buffered = Binary.Buffer(result), /* avoid risk of double request */
                status = if buffered = null then 0 else Value.Metadata(result)[Response.Status],
                actualResult = if status = 429 then null else buffered
            in
                actualResult,
        (i) => #duration(0, 0, 0, i*0.1), 10
        )

Best, Denis