cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
PowerBIRookie
Regular Visitor

Incremental Refresh with Dataflow is duplicating

I have a dataflow that I'm trying to setup incremental refresh. I have set it up but no the dataflow has some rows that have duplicate of my primary key from the base fact table. I can see the 2 rows for the same primary key number in the dataflow using power query. So my question is how is this happening?

my refresh is set up like

PowerBIRookie_0-1656090956768.png

 

The UPDATED column is a datetime column in the fact table that changes every time the record is updated. 

 

My Goal is to load 2 years of data (fact more then that... like 10 years... but I just want a rolling 2 years) and I want to reload or refresh records in the dataflow that have been modified within the last 5 days or so. But if the record exists already in the dataflow the modified record should overwrite the existing record that has the same pk. How do I do that? Cause currently is loads the original record then if that record was modified it loads that one too as a second record in the dataflow.

2 REPLIES 2
otravers
Super User
Super User

Make sure that one of your date filters doesn't have the = sign, i.e. use >RangeStart and <= RangeEnd or >=RangeStart and < RangeEnd. Here's a good entry to guide you:

http://biinsight.com/implementing-incremental-refresh-in-power-bi-part-1/

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals

Yes I made sure about that.. I read that article here is the filter 

Table.SelectRows(#"Removed columns", each DateTime.From([UPDATED]) >= RangeStart and DateTime.From([UPDATED]) < RangeEnd)
 
but this only applies to the desktop not the service... so I'm checking the dataflow using power query to see the duplicated row with the primary key (first column)
PowerBIRookie_0-1656093057466.png

 



Helpful resources

Announcements
Winner of T-Shirt Design

Power BI T-Shirt Design Challenge 2023

Find out who won the T-Shirt Design Challenge and the top 3 finalists.

March 2023 Update3

Power BI March 2023 Update

Find out more about the March 2023 update.

Top Solution Authors