cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
xhead
Helper II
Helper II

Dataflows deployed from a Deployment Pipeline - invalid data source

I have created a dataflow that has a single data source (a SQL database). Here it is in lineage view.

 

OCA Dataflow lineage DEV.PNG

 

 

I am using a Power BI Deployment Pipeline to deploy that dataflow from Workspace Dev to Workspace Test. There is a rule applied that changes the data source from the Dev database and server specified in Workspace Dev to a different Test database and server in Workspace Test. (In various labels on these screenshots, you'll see "rt" in the naming convention which represents Dev, and "pp" which represents Test).

 

This is the rule I've defined:

 

OCA data source rule.PNG

 

When I look at the deployed dataflow in Workspace Test, it now shows lineage for two data sources, both Dev and Test. 

OCA Dataflow lineage in Test workspaceOCA Dataflow lineage in Test workspace

The top SQL Server database is the Dev connection (unused and invalid), the bottom one is the Test connection.

 

When I export the json of the dataflow in Workspace Test, it only shows one single data source, and it correctly points to the Test database. 

 

This looks like a bug in the deployment pipeline somehow? 

 

The problem for me is that I need to configure the gateway connection for the SQL database, and, due to company security policies, there is a gateway configured for the Test environment that should only have access to the Test Azure SQL Server instances in a virtual network, and not the Dev Azure SQL Server instances, but the data source configurations for the Test dataflow require me to map *both* those data sources to the same gateway. Since the dev database isn't actually used, I can configure a dummy datasource in the gateway for the dev instance and since it never gets queried, it won't fail. But it looks like I have both Dev and Test data sources in the Test gateway.

 

OCA dataflow gateway config TESTOCA dataflow gateway config TEST

I shouldn't have the first data source if the dataflow lineage was correct, but I need to configure this (invalid and unused) datasource in this gateway in order to map both these data sources to the same gateway.

 

I think I'll also create a support request for this issue, but I wanted to post it here in case anyone had expierenced this and figured out a better way or maybe I'm doing something wrong. 

 

Mike

 

 

1 ACCEPTED SOLUTION
xhead
Helper II
Helper II

I have reported this issue through a Power BI Support ticket, and it has been acknowledged as a bug which involves the Data Flow team, the Lineage team, and the Deployment team. The estimated time the bug will be resolved is Feb 2022.

 

I have been able to work around this issue partially by parameterizing the database and server names in the connection, and using deployment rules on the parameter values in the deployment. In the target workspace, the old data source dependency is shown in the lineage view, but that "phantom" data source does not need to be mapped to a data gateway in the settings of the data flow. 

 

When I didn't parameterize the data source server and database, I needed to map the "phantom" data source to a data gateway (and the same data gateway) as the valid data source. 

 

View solution in original post

2 REPLIES 2
xhead
Helper II
Helper II

I have reported this issue through a Power BI Support ticket, and it has been acknowledged as a bug which involves the Data Flow team, the Lineage team, and the Deployment team. The estimated time the bug will be resolved is Feb 2022.

 

I have been able to work around this issue partially by parameterizing the database and server names in the connection, and using deployment rules on the parameter values in the deployment. In the target workspace, the old data source dependency is shown in the lineage view, but that "phantom" data source does not need to be mapped to a data gateway in the settings of the data flow. 

 

When I didn't parameterize the data source server and database, I needed to map the "phantom" data source to a data gateway (and the same data gateway) as the valid data source. 

 

v-kkf-msft
Community Support
Community Support

Hi @xhead ,

 

This is by design. When I export the dataflow from workspace A as a .json file, and then use that model in workspace B and modify the data source, then the old data source will still appear as lineage for the dataflow. And it will still appear in the Setting page in the gateway drop-down.

 

vkkfmsft_1-1631522530511.png  vkkfmsft_2-1631522617820.png

 

This is mentioned in the documentation.

 

vkkfmsft_3-1631522704616.png

 

 

If the problem is still not resolved, please provide detailed error information or the expected result you expect. Let me know immediately, looking forward to your reply.

Best Regards,
Winniz

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
June 2022 update 768X460.jpg

Check it out!

Click here to read more about the June 2022 updates!

Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Power BI Dev Camp Session 23 768x460.jpg

Check it Out!

Mark your calendars and join us on Thursday, June 30 at 11a PDT for a great session with Ted Pattison!

June 20 episode 7 with aka link 768x460.jpg

The Power BI Community Show

Join us on June 20 at 11 am PDT when Kim Manis shares the latest on Azure Synapse analytics, the Microsoft Intelligent Data Platform, and notable Power BI Updates from Microsoft Build 2022.

Top Solution Authors