Dataflows refresh for linked entities failing

[uberdube]

Hi,

Does anyone know if it's possible to refresh dataflows in parallel (lets call those dataflows 'A'), when they are connected to another dataflow ('B') which contains linked entities from 'A'? I can only seem to have one dataflow refesh at a time, the other will fail. There is also a minimum 30 min interval between scheduling refreshes, which would take too long to schedule all the dataflows I need. There's currently 9 dataflows set to incremental refresh so they only take about 5 mins each, but would need to be staged over 4.5 hours which isn't acceptable for my case (I have about a 1 hour window available to have refreshes completed) so currently the only way is to do this manually every day :(.

Alternatively I'm also wondering if this is actually/partially consequent of other current recognised issues around scheduled refreshes failing, and that the issue might soon just go away with a fix coming?

Any help appreciated!

[uberdube]

Hi,

Does anyone know if it's possible to refresh dataflows in parallel (lets call those dataflows 'A'), when they are connected to another dataflow ('B') which contains linked entities from 'A'? I can only seem to have one dataflow refesh at a time, the other will fail. There is also a minimum 30 min interval between scheduling refreshes, which would take too long to schedule all the dataflows I need. There's currently 9 dataflows set to incremental refresh so they only take about 5 mins each, but would need to be staged over 4.5 hours which isn't acceptable for my case (I have about a 1 hour window available to have refreshes completed) so currently the only way is to do this manually every day :(.

Alternatively I'm also wondering if this is actually/partially consequent of other current recognised issues around scheduled refreshes failing, and that the issue might soon just go away with a fix coming?

Any help appreciated!

[jeffshieldsdev · Accepted solution]

I originally used this architecture too.

As you've noticed, when your first A Level dataflow starts refreshing, it puts a lock on downstream dataflows in the same workspace...so if the second A Level dataflow tries to refresh, it won't be able to get a lock on the B Level dataflow and will fail.

My workaround has been to create another workspace--my "publish" workspace--that has your B Level dataflow with all the A Level entities contained as Linked Entities.

B won't automatically refresh with the A Level dataflows, but since all B contains is Linked Entities, a scheduled refresh should run quickly.

With the B Level dataflow removed from your inital workspace, your A Level dataflows will be able to refresh concurrently.

Also, you could look at using MS Flow/Power Automate or Logic Apps to orchestrate a more granular refresh schedule--you can trigger Power BI refresh from both 🙂

[andreamastrini]

Thanks @jeffshieldsdev ! I have the same exact issue too.

Anyway what should be the correct behaviour assuming all issues about scheduled refrehes will be fixed? Is the best practice to

 1) create the "publish" workspace separating A and B level Dataflows OR

 2) leave both A and B Level dataflows in the same workspaces (assuming the refresh will complete correclty)?

Thanks

[jeffshieldsdev]

If you want your A dataflows to be able to refresh concurrently, your B level cannot be in the same workspace--so option 1.

Techincally it could be--if you disabled load on the linked entities in it--but that would make a copy of the A Level dataflow data, which is redundant.

[uberdube]

Thank you so much @jeffshieldsdev !! Great to get a definitive answer on this.

Sadly (for me!) what you have described (ie. option 1) is exactly how I did this originally..  however I was encountering a strange error and behaviours with entities in the 'publish' workspace being 'locked' and unable to save changes/refresh etc... prompting me to attempt this 'single' dataflow method instead. I logged an MS SUpport call and this turned out to be some kind of weird 'glitch' it seems.. but endeavoured to blow the original model away and attempt rebuilding in an alternative style.

I'm definately able (and happy!) to go back to the drawing board now feeling confident this isn't all the result of a development bug!

Thanks so much for your response.