cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
PowerBIRookie
Regular Visitor

Incremental Refresh with Dataflow is duplicating

I have a dataflow that I'm trying to setup incremental refresh. I have set it up but no the dataflow has some rows that have duplicate of my primary key from the base fact table. I can see the 2 rows for the same primary key number in the dataflow using power query. So my question is how is this happening?

my refresh is set up like

PowerBIRookie_0-1656090956768.png

 

The UPDATED column is a datetime column in the fact table that changes every time the record is updated. 

 

My Goal is to load 2 years of data (fact more then that... like 10 years... but I just want a rolling 2 years) and I want to reload or refresh records in the dataflow that have been modified within the last 5 days or so. But if the record exists already in the dataflow the modified record should overwrite the existing record that has the same pk. How do I do that? Cause currently is loads the original record then if that record was modified it loads that one too as a second record in the dataflow.

2 REPLIES 2
otravers
Super User
Super User

Make sure that one of your date filters doesn't have the = sign, i.e. use >RangeStart and <= RangeEnd or >=RangeStart and < RangeEnd. Here's a good entry to guide you:

http://biinsight.com/implementing-incremental-refresh-in-power-bi-part-1/

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals

Yes I made sure about that.. I read that article here is the filter 

Table.SelectRows(#"Removed columns", each DateTime.From([UPDATED]) >= RangeStart and DateTime.From([UPDATED]) < RangeEnd)
 
but this only applies to the desktop not the service... so I'm checking the dataflow using power query to see the duplicated row with the primary key (first column)
PowerBIRookie_0-1656093057466.png

 



Helpful resources

Announcements
Carousel_PBI_Wave1

2023 Release Wave 1 Plans

Power BI release plans for 2023 release wave 1 describes all new features releasing from April 2023 through September 2023.

Power BI Summit Carousel 2

Global Power BI Training

Make sure you register today for the Power BI Summit 2023. Don't miss all of the great sessions and speakers!

Thank you 2022 Review

2022 Monthly Feature Releases

We had a great 2022 with a ton of feature releases to help you drive a data culture.

Top Solution Authors
Top Kudoed Authors