Incremental Data Load Options in Microsoft Fabric

[jashfabric]

Hi everyone,
I'm building a Medallion Architecture in Microsoft Fabric and am exploring options for implementing incremental data loads. Does Fabric offer built-in functionalities to achieve this? If so, what are the available approaches (e.g., Data Factory pipelines, Spark notebooks)?

Any insights or best practices for implementing incremental loads in Fabric would be highly valuable.
Thanks!
Tagging @v-nikhilan-msft 

[v-nikhilan-msft · Accepted solution]

Hi @jashfabric 
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.

For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-datafl...

Hope this helps. Please let me know if you have any further queries.

[AndyDDC · Accepted solution]

Hi @jashfabric from a Spark/Notebook/Delta perspective, you could look into using Change Data Feed:

Delta Lake Change Data Feed in Fabric Lakehouses (serverlesssql.com)

[v-nikhilan-msft · Accepted solution]

Hi @jashfabric 
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.

For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-datafl...

Hope this helps. Please let me know if you have any further queries.

[jovianaditya]

Hello, i want to ask some question. Is this method (from the link that you provided) is only works for append condition? If i have 100 rows from day one, and for the next day there are 2 rows that have been modified will that method also automatically update my 2 rows?

[jashfabric]

Thanks for your help @v-nikhilan-msft  @AndyDDC 
That really helps.

[v-nikhilan-msft]

Hi @jashfabric 
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.