Deployment Pipeling Dataflows and linked entities

[CoffeeAddictMk2]

Im trying to utilise deployment pipelines however having an issue where a master dataflow (fed from other dataflows in the pipeline) isnt updating to the new workspaces as it is promoted unless the dataflow entities have enable load ticked.

This is a problem as there are 12 dataflows going into this master flow and I do not want 12 refreshes triggered on it.

Is there a way to disable the triggered refresh without disabling enable load?

[agusmba]

I started playing around recently with dataflows and deployment pipelines.

From what I've experienced, the auto-bind feature of deployment pipelines won't work unless the linked entities are loaded.

I'm following the recommendation of having an extraction dataflow that simply loads the source tables, and a second dataflow connected (with linked tables) to the first in order to transform the tables.

If I don't load the linked tables on the second dataflow, they won't be auto-bound to their new workspace equivalents when deployed, and will keep referencing the entities in the original workspace (not what we normally intend). [Should this be an improvement request to allow auto-bind on not-loaded-linked-tables-from-other-dataflows?]

As for your comment regarding cascading refreshes, I think I read somewhere that the refreshes only affect the parts of the dataflow that depend on the triggering refresh. The cascading refreshes are not full-refresh. But I have not tested this so I'm not sure if that is correct.

[CoffeeAddictMk2 · Accepted solution]

I have a solution though the use of Parameters to update the workspace to the different workspaces in the pipeline as a dataflows are progressed.

The issue was I need to combine Years worth of excel extracts from sharepoint for multiple data sources as I cannot access the servers directly due to lack of gateway.

Dataflow A  - 24 excel files for Y data

Dataflow B - 24 Excel files for Z data

(there were more sources but simplifying it) These 2 dataflows, append and transform their data so they match field names and are appended to each other in dataflow C

in the dev space i only want to append 2 files for each data set and in the test and prod space i wanted all the data.

By Setting up a workspace parameter and changing the Navigation1 step to 

Navigation{[workspaceName=""&Workspace&""]}[Data]

This enables me to change the workspace the unloaded dataflow refreshes it data from as it moved through the pipeline.

[agusmba]

What I meant, trying to adapt it to your example, is that if your dataflow C is linking tables from dataflows A and B, but not loading them, when you deploy C on your TEST environment, the linked-but-not-loaded tables will point to A and B on the DEV environment. They are not auto-bound to A and B on the TEST environment unless the "load" tick is set.

[CoffeeAddictMk2]

I know, that was the problem I intially posted. 

having them set to load would cause them all to fail as both dataflows A and B would try and refresh C at the same time.

[agusmba]

Exactly, you'd need to set up different times for the refresh on A and B so that the cascading refreshes on C won't overlap.

If you don't load the linked tables, you can refresh A and B at the same time, and trigger C only once (manual/schedule). However in this last case you need to make sure that TEST-C is taking data from your TEST A and B instead of from DEV. You've probably already solved this with the use of parameters if I read you correctly, which is the way to go in this case.

In my case the dataflows in C were referenced via Id, so I'm not sure using a parameter for the workspace would be enough. Are you using the PowerPlatform dataflows connector or the legacy PowerBI one? For whatever reason I was reluctant to switch from the legacy to the new one.

[lbendlin]

You are supposed to use scaled down data sources in the DEV and ITG workspaces. Using the same source in all workspaces defeats the purpose somewhat. You can also choose to manually refresh the dataflow on demand.

[CoffeeAddictMk2]

Thats exactly what im trying to do via use of parameters to bring in less files from sharepoint for each of the 12 dataflows.

To utilise different sharepoint locations would be couter intuitive as this would be duplication the data storage, It would also not alleviate the problem of combining the datasources into a single entity.

Each Dataflow represents a different system providing eqivilent data, Each systems data needs to be transformed and cleansed so they can be appended together to create a dataset which can report across all systems to give a buisness wide view.  There is no Direct database access possible