cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
harshdhawan Frequent Visitor
Frequent Visitor

Handling of Deleted data in incremental refresh

Hi All

 

I wanted to understand how the rows which are deleted from table are handled in incremental refresh dataset.

Does it identify the deleted rows and delete from dataset OR Do we need to do full refresh in PBI desktop and then publish?

 

Thanks & regards

Harsh Dhawan

5 REPLIES 5
Super User IV
Super User IV

Re: Handling of Deleted data in incremental refresh

Hi there

When using Incremental refresh it is done in the backend in the Power BI Service where the model is hosted.

The process actually drops the partitions which are considered to be outside of the data window of data to keep.





Did I answer your question? Mark my post as a solution!

Proud to be a Super User!






Power BI Blog
harshdhawan Frequent Visitor
Frequent Visitor

Re: Handling of Deleted data in incremental refresh

Thanks Gilbert for your reply

 

I have a table which contains 12 million rows. and it is modelled based on row level permissions from source system

This table is truncated and loaded on daily basis, Now I am being asked to look at the possibility to do incremental refresh for dataset which is loading this table

 

In case user permissions are removed from any record in source table, that data rows will not be loaded in the next etl run

 

I am not sure how that row will be removed from power bi dataset during the next incremental refresh if it no longer exists in source table now

 

Hope my question makes sense!

 

Cheers

 

 

Super User IV
Super User IV

Re: Handling of Deleted data in incremental refresh

Hi there

If I understand your question correctly let me try and explain it by like this.

When using incremental refreshing in Power BI, older data will not be refreshed or imported again into Power BI.

Based on your explanation above that would mean that certain data must be deleted or changed because the underlying data has changed.

You have 2 options, the first one is to just reload all the data. I currently have a project where I am uploading 15 million rows and it refreshes in about 20minutes.

Option number 2 is that you can use the detect data changes when using Incremental refresh with Power BI. Here is a great video explaining how to do it by GuyInaCube: https://www.youtube.com/watch?v=JsJWBr1_ktQ




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!






Power BI Blog
ndesousa Regular Visitor
Regular Visitor

Re: Handling of Deleted data in incremental refresh

Handling deletes is just what I am after. Thanks for posting that link  🙂

 

The approach outlined by Patric is not ideal but it's the next best thing - until that capability is added to incremental refresh.

Highlighted
Super User IV
Super User IV

Re: Handling of Deleted data in incremental refresh

Yeah for sure, I am hopeful that over time the incremental refresh story will improve.




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!






Power BI Blog

Helpful resources

Announcements
Exclusive LIVE Community Event #3 – Ask Arun Anything

Exclusive LIVE Community Event #3 – Ask Arun Anything

Join us in the third Triple A event!

Meet the 2020 Season 1 Power BI Super Users!

Meet the 2020 Season 1 Power BI Super Users!

It’s the start of a new Super User season! Learn all about the new Super Users and brand-new tiered recognition system.

January 2020 Community Highlights

January 2020 Community Highlights

Make sure you didn't miss any of the things that happened in the community in January!

Top Solution Authors