Deployment Pipelines and Dataflows

[chris__1]

Hi,

I have a workspace that contains all my Dataflows. This Dataflow-Workspace has the deployment Pipeline enabled.

The other workspaces contain reports and datasets that make use of the dataflows.

Those workspaces have also deployment Pipelines enabled.

I chose this setup before I enabled the deployment pipelines because I make extensive use of dataflows and having them in one single workspace seemed like a good idea to keep things organized (as one dataflows gets used by multiple reports).

Naturally I would like to have my DEV-Dataflows to be the data source for my DEV-Datasets & Reports (same for TEST and PROD). However if I am not mistaken, rules for Dataflows in Pipelines are not supported (atm?) (https://docs.microsoft.com/en-us/power-bi/create-reports/deployment-pipelines-get-started#deployment...). This means that I can't set different dataflows as a reports data source depending on the stage the report is in.

Is this correct? Would it be a much better setup to have the Dataflows in every workspace where they are being used (this would mean to have duplicates)? I'm using PBI Premium PU if this makes any difference!

Thanks!

[Nimrod_Shalit]

@chris__1 we've addressed this issue exactly in an even easier way-it will just work for you automatically!

no need to configure anything.

Please follow the instructions here to make sure it works for you.

[chris__1]

@Nimrod_Shalit I don't think that the method you described works in my case.

Let me explain the setup I am using:

Pipeline A: Contains the Dataflow A. Dataflow A is deployed in DEV, TEST and PROD of Pipeline A

Pipeline B: Contains the Report B and the Dataset B! Report B uses Dataset B with uses the Dataflow A as its (only) Datasource.

Trying to deploy the Report B and Dataset B from DEV to TEST gives me this error message:

I am doing something wrong here? The Dataflows do exist in the the DEV-Stage of Pipeline A! If I am not doing something wrong, I guess the error message means that I need to have the Dataset in the same Pipeline as the Dataflows. However from a maintenance perspective it would be so much easier it we could have the Reports and Datasets in one Pipeline and Datasets in a different Pipeline!

Thanks for any advice!

[Nimrod_Shalit]

You don't need to change anything. Looks like the dataflow in pipeline A just doesn't exist in Test stage. Since we are trying to maintain the same structure as in Dev stages of pipelines A & B, we first check if the dataflow also exists in Test of pipeline A, so we can connect the dataset to it.
Please make sure the dataflow exists in Test of pipeline A, and then redeploy dataset and report in pipeline B from Dev to Test.

[chris__1]

@Nimrod_Shalit please, have a look at the steps I take to deploy the report from DEV to TEST:

1. The Dataflow Pipeline is equal in DEV and TEST:

2. The Report&Dataset Pipeline: DEV has the Report&Dataset, TEST will be deployed now:

3. I hit "Deploy to Test"

4. I get the same error as posted before:

EDIT

The Pipeline for the Reports&Datasets and the Pipeline for the Dataflows are different Workspaces! Might this be the cause for the problem? Do all Ojects (Report, Dataset, Dataflow) need to be in the same WORKSPACE??

[Nimrod_Shalit]

Looks like you are doing it right.
Please verify that the dataset is indeed connected to that dataflow in the other pipeline, and has no dependencies on other pipelines anywhere else.

If the check is OK, then it look like a technical issue we will need to explore. Please open a support ticket with all the technical info and IDs of pipelines and artifacts.

[chris__1]

@Nimrod_Shalit this is the lineage for the dataset. The only source are various dataflow from this Pipeline:

so I need to create an issue, however I did not find the right category for Pipeline Issues on this page: https://admin.powerplatform.microsoft.com/newsupportticket/powerbipro

[MichaD]

Hi chris__1,
has your issue been resolved? What was the solution?

Thanks

Michael

[chris__1]

Hi Michael,

unfortunatly this issue has not been resolved for me - hence no solution has been marked.

I suspect, that my workspace setup has not been fully understood here - most likely I did not explain very well.

Do you have the same setup as I do: 

- One Workspace containing ALL dataflows. This Workspace has deployment Pipelines activated.

- Multiple other workspaces containing reports and corresponding Datasets. Those Workspaces also have Deployment Pipelines activated.

- Trying to have the datasets in the stages "TEST", "DEV", "PROD" automatically connect to the datasets in the corresponding stage of the Dataflow-Workspace.

Right now I have all my datasets in *all* stages connected to the PROD stage of the dataflow workspace... which is not the ideal solution.

Please let me know if you manage to find a solution!

Thanks, Chris

[LeeBenjamin]

Hi  @chris__1 

As for your last point of:
"Trying to have the datasets in the stages "TEST", "DEV", "PROD" automatically connect to the datasets in the corresponding stage of the Dataflow-Workspace."

Please advise that the above-mentioned datasets in the stages "TEST", "DEV", "PROD", ARE NOT connected to the dataset on your Dataflows workspaces. they are connected to the dataflows themselves. With that, in order for things to work, please ensure the following:

On your Datasets and Reports pipeline - Dev stage, select the dataset you'd like to deploy to Test environment and view its lineage. we expect to see that:

1. It is connected to dataflows on your Dataflows pipeline Dev stage's workspace (I can't see on the lineage view screenshot you sent if this is the case because it is a lineage view of a different dataset than the one you wish to deploy. I can tell by the name). 
2. Those connected dataflows are also on your Dataflows pipeline Test stage's workspace and are in sync (which is the case per your screenshot of your Dataflows pipeline).

Once all the above are in place the deployment should be completed successfully. Otherwise, please open a support ticked do we could assist further.
BR,

Lee

[v-rongtiep-msft]

Hi @chris__1 ,

Now Power BI Pipelines support Dataflows.

The article says: 

• To create a deployment rule, you must be the owner of the dataflow, dataset or paginated report you're creating a rule for.

• Rules for dataflows that have other dataflows as sources, are not supported.

• Data source rules for common data model (CDM) folders in a dataflow, are not supported.

• Rules for datasets that use dataflows as their source, are not supported.

Each workspace (or environment) is available only for members of that workspace. If you build all your dataflows in one workspace, you're minimizing the reuse of your dataflows. You can have some generic workspaces for dataflows that are processing company-wide entities. You can also have some workspace for dataflows to process entities across multiple departments. And you can also have some workspaces for dataflows to be used only in specific departments.

More details:Best practices for reusing dataflows across environments and workspaces 

If it does not help, please provide more details with your desired output.

If it doesn't help, please provide a simple sample with the same issue.

Best Regards

Community Support Team _ Polly

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[chris__1]

I'll try to break down the core of my question:

Is it possible to use parameters to have the datasets of reports point to different dataflows as datasource if those dataflows are in a different workspace? The DEV, TEST and PROD datasets should use the DEV, TEST, PROD dataflows.

Thanks!