Dataflows deployed from a Deployment Pipeline - invalid data source

[xhead]

I have created a dataflow that has a single data source (a SQL database). Here it is in lineage view.

I am using a Power BI Deployment Pipeline to deploy that dataflow from Workspace Dev to Workspace Test. There is a rule applied that changes the data source from the Dev database and server specified in Workspace Dev to a different Test database and server in Workspace Test. (In various labels on these screenshots, you'll see "rt" in the naming convention which represents Dev, and "pp" which represents Test).

This is the rule I've defined:

When I look at the deployed dataflow in Workspace Test, it now shows lineage for two data sources, both Dev and Test. 

OCA Dataflow lineage in Test workspace

The top SQL Server database is the Dev connection (unused and invalid), the bottom one is the Test connection.

When I export the json of the dataflow in Workspace Test, it only shows one single data source, and it correctly points to the Test database. 

This looks like a bug in the deployment pipeline somehow? 

The problem for me is that I need to configure the gateway connection for the SQL database, and, due to company security policies, there is a gateway configured for the Test environment that should only have access to the Test Azure SQL Server instances in a virtual network, and not the Dev Azure SQL Server instances, but the data source configurations for the Test dataflow require me to map *both* those data sources to the same gateway. Since the dev database isn't actually used, I can configure a dummy datasource in the gateway for the dev instance and since it never gets queried, it won't fail. But it looks like I have both Dev and Test data sources in the Test gateway.

OCA dataflow gateway config TEST

I shouldn't have the first data source if the dataflow lineage was correct, but I need to configure this (invalid and unused) datasource in this gateway in order to map both these data sources to the same gateway.

I think I'll also create a support request for this issue, but I wanted to post it here in case anyone had expierenced this and figured out a better way or maybe I'm doing something wrong. 

Mike

[xhead · Accepted solution]

I have reported this issue through a Power BI Support ticket, and it has been acknowledged as a bug which involves the Data Flow team, the Lineage team, and the Deployment team. The estimated time the bug will be resolved is Feb 2022.

I have been able to work around this issue partially by parameterizing the database and server names in the connection, and using deployment rules on the parameter values in the deployment. In the target workspace, the old data source dependency is shown in the lineage view, but that "phantom" data source does not need to be mapped to a data gateway in the settings of the data flow. 

When I didn't parameterize the data source server and database, I needed to map the "phantom" data source to a data gateway (and the same data gateway) as the valid data source. 

[v-kkf-msft]

Hi @xhead ,

This is by design. When I export the dataflow from workspace A as a .json file, and then use that model in workspace B and modify the data source, then the old data source will still appear as lineage for the dataflow. And it will still appear in the Setting page in the gateway drop-down.

  

This is mentioned in the documentation.

If the problem is still not resolved, please provide detailed error information or the expected result you expect. Let me know immediately, looking forward to your reply.

Best Regards,
Winniz

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.