Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
Anonymous
Not applicable

Incremental Refresh with Dataflow is duplicating

I have a dataflow that I'm trying to setup incremental refresh. I have set it up but no the dataflow has some rows that have duplicate of my primary key from the base fact table. I can see the 2 rows for the same primary key number in the dataflow using power query. So my question is how is this happening?

my refresh is set up like

PowerBIRookie_0-1656090956768.png

 

The UPDATED column is a datetime column in the fact table that changes every time the record is updated. 

 

My Goal is to load 2 years of data (fact more then that... like 10 years... but I just want a rolling 2 years) and I want to reload or refresh records in the dataflow that have been modified within the last 5 days or so. But if the record exists already in the dataflow the modified record should overwrite the existing record that has the same pk. How do I do that? Cause currently is loads the original record then if that record was modified it loads that one too as a second record in the dataflow.

2 REPLIES 2
otravers
Community Champion
Community Champion

Make sure that one of your date filters doesn't have the = sign, i.e. use >RangeStart and <= RangeEnd or >=RangeStart and < RangeEnd. Here's a good entry to guide you:

http://biinsight.com/implementing-incremental-refresh-in-power-bi-part-1/

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals
Anonymous
Not applicable

Yes I made sure about that.. I read that article here is the filter 

Table.SelectRows(#"Removed columns", each DateTime.From([UPDATED]) >= RangeStart and DateTime.From([UPDATED]) < RangeEnd)
 
but this only applies to the desktop not the service... so I'm checking the dataflow using power query to see the duplicated row with the primary key (first column)
PowerBIRookie_0-1656093057466.png

 



Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors