Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more.
Get startedGrow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.
I have set up incremental refresh on a flow as follows:
I am pulling from a SQL database table and it's last update date as above.
I have pulled the table into my dataset and I am getting duplicate keys. I'm not sure what I'm missing? Why am I getting duplicates?
Hi. That might be because the "Detect data changes". As the name says this is incremental. That means that if you modify a row from a year ago but you are only specifing the refresh should be from last day, then you will have last day + changes from history. The changes are a new row that looks almost the same as the one before but with the change.
The detect data changes is NOT recommended unless you know how to handle it as data perspective. You usually use a change or modify date and things like that.
Incremental refresh cannot update rows from old history, it can add new rows with the change. If your warehouse or source is build for that changes you can add columns like "last update" flag to filter it in the power query.
I hope that helps,
Happy to help!
Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.
Ask questions in Eventhouse and KQL, Eventstream, and Reflex.