cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
PowerBIRookie
Regular Visitor

Incremental Refresh with Dataflow is duplicating

I have a dataflow that I'm trying to setup incremental refresh. I have set it up but no the dataflow has some rows that have duplicate of my primary key from the base fact table. I can see the 2 rows for the same primary key number in the dataflow using power query. So my question is how is this happening?

my refresh is set up like

PowerBIRookie_0-1656090956768.png

 

The UPDATED column is a datetime column in the fact table that changes every time the record is updated. 

 

My Goal is to load 2 years of data (fact more then that... like 10 years... but I just want a rolling 2 years) and I want to reload or refresh records in the dataflow that have been modified within the last 5 days or so. But if the record exists already in the dataflow the modified record should overwrite the existing record that has the same pk. How do I do that? Cause currently is loads the original record then if that record was modified it loads that one too as a second record in the dataflow.

2 REPLIES 2
otravers
Super User
Super User

Make sure that one of your date filters doesn't have the = sign, i.e. use >RangeStart and <= RangeEnd or >=RangeStart and < RangeEnd. Here's a good entry to guide you:

http://biinsight.com/implementing-incremental-refresh-in-power-bi-part-1/

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals

Yes I made sure about that.. I read that article here is the filter 

Table.SelectRows(#"Removed columns", each DateTime.From([UPDATED]) >= RangeStart and DateTime.From([UPDATED]) < RangeEnd)
 
but this only applies to the desktop not the service... so I'm checking the dataflow using power query to see the duplicated row with the primary key (first column)
PowerBIRookie_0-1656093057466.png

 



Helpful resources

Announcements
Vote for T-Shirt Design

Power BI T-Shirt Design Challenge 2023

Vote for your favorite t-shirt design now through March 28.

March 2023 Update3

Power BI March 2023 Update

Find out more about the March 2023 update.

March Events 2023A

March 2023 Events

Find out more about the online and in person events happening in March!

Top Solution Authors