Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
itchyeyeballs
Impactful Individual
Impactful Individual

Deployment Pipeline Error - Changes made to the source dataset are preventing the deployment.

Hi All,

 

We have hit an issue with deployment pipelines, everything was working well and we went through several cycles of deployment from Dev to Test to Prod.

After the most recent changes to the model we published to Dev but had the following error when trying to deploy to test:

Changes made to the source dataset are preventing the deployment. Make sure the dataset is configured properly, then try deploying it again.

This is the second time this has occurred, last time I deleted the test version and redeployed from Dev, however, I then ended up with two separate versions of the model in Prod with the same name (so I don't want to do that again)

Any ideas on how to resolve? I can independently refresh the Dev and Test models successfully so they both seem to be valid. 


1 ACCEPTED SOLUTION

Hi,

 

The dataset that caused the issue was an SQL Server (On-Prem) database table. We had made many changes (add, remove, modify fields) and successfully deployed multiple times. Not sure what it was that broke the Pipeline. 

 

The model itself has several SQL server data tables, dataflows from within the workspace and dataflows from a separate workspace.

I think I have solved the issue (still testing). I used ALM toolkit to sync the two datasets and it has just let me deploy to test, I had to then use ALM toolkit again as I got the same error Test --> Dev 

I have just kicked off the PowerAutomate process that refreshes everything, hopefully will work!

View solution in original post

10 REPLIES 10
willayd
Advocate I
Advocate I

@itchyeyeballssuggestion to look at the diff using the ALM Toolkit was immensely helpful. In our particular case we ran into this issue because we set a table in our DEV environment to "Mark as Date Table". We had to revert that change in DEV for the pipelines to work again. My guess is that there are a multitude of other reasons why this may happen

itchyeyeballs
Impactful Individual
Impactful Individual

Hi all,

 

Additional update in case anyone finds it useful as I hit another deployment error today 😞

 

Again terrible error messages with no clue on what the issue was or how to resolve. This time ALM toolkit did not manage to solve the problem (although it pointed me in the right direction)

The issue occurred when I modified an import process to remove a format function from the underlying SQL (it was resulting in invalid dates) then converted the resulting text field to date, I was able to deploy the revised model to test as that dataset was empty, deployment to prod then failed, I'm assuming this was because the values already in the dataset could not be converted to date as they were invalid. As I couldn't modify or clear the contents of the prod dataset directly I was stuck.

The fix was:

- format the field back to text in dev version

- redeploy (all stages)

- refresh the data (all stages) (to remove the invalid dates)

- Format the field back to date in dev version

-redeploy (all stages)

- Refresh the data.

It feels like the pipeline process needs an option to clear existing data before applying schema changes (and better error warnings!)

@itchyeyeballs,

For the scenario you described we provided, on the error message, the ability to continue deployment while removing data, or stop the deployment.

I'm interested to understand why this didn't happen in your case.

It would be very helpful if you can try and repro what you described and provide screenshots and technical details in a support ticket so we can take a look internally and investigate.

Hi, I will try to replicate the issue.

Is there any available documentation that describes this deployment behavior? I have looked here but can't see anything. 

Thank you, I'll test to see if it was me missing the obvious instructions or if something else happened.

Anonymous
Not applicable

Any updates on this? I'm getting the same error and can't figure out what causes this?

I'm still not sure what causes the issue, I would love to know what process happens behind the scenes as I had assumed that when deploying the dev version would simply overwrite the test version. I have no idea what testing happens during deployment or why (and why there is no way to explore and fix issues that the testing identifies).

My suspicion is the error happens when changing links between tables (possibly when combined with column changes/additions).

ALM toolkit has solved the problem for me every time but it's a bit of a pain to have to use it, especially as the error from PowerBI is so unhelpful.

npergand
Helper I
Helper I

If you don't mind me asking, what was the exact change you made to the dataset that triggered this error?

I am interested in understanding what types of dataset modifications may cause limitations in with the deployment pipelines.

Also what types of data sources are in your dataset? I wonder if that may be part of it.

Hi,

 

The dataset that caused the issue was an SQL Server (On-Prem) database table. We had made many changes (add, remove, modify fields) and successfully deployed multiple times. Not sure what it was that broke the Pipeline. 

 

The model itself has several SQL server data tables, dataflows from within the workspace and dataflows from a separate workspace.

I think I have solved the issue (still testing). I used ALM toolkit to sync the two datasets and it has just let me deploy to test, I had to then use ALM toolkit again as I got the same error Test --> Dev 

I have just kicked off the PowerAutomate process that refreshes everything, hopefully will work!

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors