Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
VishalSukhija
Frequent Visitor

Deployment pipeline - Update on table set for incremental refresh

Hello,

 

I am using Deployment pipeline on a dataset which is using incremental refresh. I understand the limitations which are posted in the docs:

  • Republishing a dataset that doesn't use incremental refresh, to replace a dataset that has incremental refresh enabled.

  • Renaming a table that has incremental refresh enabled.

  • Renaming non-calculated columns in a table with incremental refresh enabled.

We get below error whenever we try to deploy the using deployment pipeline while using incremental refresh and changing something in same fact:

dep_pipeline.png

 

 

 
 

We are ok with loosing data in destination. Is there an option to just overwrite the dataset in destination even if there is change in the fact with incremental refresh enabled? We can refresh complete data again.

 

Thanks,

Vishal

1 ACCEPTED SOLUTION
Nimrod_Shalit
Power BI Team
Power BI Team

We have the option to choose to continue the deployment,

even if it means to drop the data.

 

See more info here.

View solution in original post

17 REPLIES 17
Argyle311
Advocate I
Advocate I

This is still an issue for us today. Has there been any resolution? It basically makes development with Incremental Refresh tables & Deployment Pipelines an impossible task.

@Argyle311 can you explain what exactly is the problem?
there are many threads here so its hard to understand what problem you mean.

Also note that we do support incremental refresh, and also support deploying breaking changes by letting the user choose to continue the deployment and drop the data.

Absolutely, thanks for reaching out @Nimrod_Shalit. I think what I'm looking for is how to enable what you say "letting the user choose to continue the deployment and drop the data" because I don't seem to have that option. The popup message I get doesn't have the option to proceed:

 

Argyle311_0-1638377721881.png

How do I bypass this error/warning and push the deployment through, knowing and understanding (and wanting) to rebuild all incremental partitions? Is there an admin setting or toggle I'm missing somewhere? For instance, its very common for us to add new columns to our incrementally refreshed fact tables, warranting a full reload in these instances.

No response so far huh? We have the same issue...

Yes, still an issue for us as well. @Nimrod_Shalit can you help us out? We're still not getting the option to continue deployment even if it results in data loss, its not an option in the popup pasted in my original message.

 

@RobinNeven our annoying workaround is simply to manually publish the PBIX forward to each workspace, update our deployment pipeline parameters manually (for Dev/Test server names and date filters in our case) then trigger a full refresh (incremental tables get rebuilt). Very lame and time consuming, especially since we have numerous huge fact tables that get rebuilt and typically we'd only need to update one at a time if we could actually use the pipeline properly.

Nimrod_Shalit
Power BI Team
Power BI Team

We have the option to choose to continue the deployment,

even if it means to drop the data.

 

See more info here.

Great, thanks for the update!

 

As I understand it, if a table or column gets a breaking change, the entire target dataset will be emptied and need a full refresh, not just the affected table, correct?

 

Is there documentation that describes what counts as a breaking change?

 

Thanks again!

Since this is eventually an AS model, you can try and search in the AS documentation.

However, there are so many options and so many different situations that can cause that, that it might be impossible to fully document it. 

Ok, then there are some other issue. Because we have an situation that we limited the decimal numbers from 9 to 4 (column type from decimal numer to fixed decimal number) and tried to deploy the new version using deployment pipeline but faced the can't stat deployment -error message. What I understand we should get this continue deployment dialog but we just get the can't start deployment dialog and because of that we can't do the deployment at all.

Can you share the exact error message you get, as well as the tech details in the message? we can then take a look and better help out.

Nimrod_Shalit
Power BI Team
Power BI Team

Hi @VishalSukhija,

we are working right now to enable this scenario exactly.

should be live within 1-2 months. 

Any news related to this? We have just encountered the similar situation and we are eager to hear if this is now possible to solve.

Just encountered this same issue. I can't republish as the changes are in a Tabular Editor .bim file, not in Power BI Desktop. Is this feature still in the works?

 

Thanks.

v-jingzhang
Community Support
Community Support

Hi @VishalSukhija 

 

According to the docs about deployment pipeline, it seems this is by design to avoid a data loss. I haven't found any workaround to this.

 

You may post an idea about this feature and the development team will know about it.

 

Reference: https://powerbi.microsoft.com/en-us/blog/announcing-deployment-pipelines-general-availability-ga/

 

Best Regards,
Community Support Team _ Jing Zhang

It can't be true (that it's by design) because if I use SSMS and clear the data completely (ie. no data exists any longer in the table) the error is still given.  I even try clearing (deleting) all data for all tables including the date tables and it still gives this same error (that the deployment could result in loss of data)!!   It's absolutely illogical and absurd.

After a few seconds I've determined that the error message is inaccurate / a red herring - which is always frustrating for a user.  The loss of data (even when there is none) is not an issue.  It appears the way to solve it (at least for me), is to remove the Incremental Refresh config on the table and publish.  That at least allows for a successful deployment and for the column name to change.  Later, I guess we need to reapply the Incremental Refresh configs.  All very frustrating for this month's release!  

Obviously, the error message isn't necessary if devs allowed the renaming when no records exist in the tables.

Nice workaround, but jeez, that's super annoying and shouldn't be necesary. I have dozens of incremental refresh tables each with highly customized policies, its unrealistic for me to toggle them on and off just to deploy with the pipeline. I've been just publishing the dataset directly to the upstream workspaces, then circling back to the deployment pipeline in order to invoke my custom deployment parameters, then refreshing the dataset and carrying on with life.

 

I've gone through an entire ticket cycle on this with Microsoft and have been told its still an issue. Here's the most recent quote from my support tech:

 

"After circling internally, I figured that we still don’t support this action for Incremental Refresh datasets and tables unfortunately. Apologies for the confusion.

Our engineering team is working now to see how we can add this support for future cases; however we don’t have an ETA yet."

 

Please make it happen, Microsoft. You're our only hope.

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors