Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.
Hi everyone,
I'm building a Medallion Architecture in Microsoft Fabric and am exploring options for implementing incremental data loads. Does Fabric offer built-in functionalities to achieve this? If so, what are the available approaches (e.g., Data Factory pipelines, Spark notebooks)?
Any insights or best practices for implementing incremental loads in Fabric would be highly valuable.
Thanks!
Tagging @v-nikhilan-msft
Solved! Go to Solution.
Hi @jashfabric
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.
For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-datafl...
Hope this helps. Please let me know if you have any further queries.
Hi @jashfabric from a Spark/Notebook/Delta perspective, you could look into using Change Data Feed:
Delta Lake Change Data Feed in Fabric Lakehouses (serverlesssql.com)
Hi @jashfabric from a Spark/Notebook/Delta perspective, you could look into using Change Data Feed:
Delta Lake Change Data Feed in Fabric Lakehouses (serverlesssql.com)
Hi @jashfabric
Thanks for using Fabric Community.
You can use a Dataflow gen2 for incremental load. Incrementally amassing data in a data destination requires a technique to load only new or updated data into your data destination. This technique can be done by using a query to filter the data based on the data destination.
For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-datafl...
Hope this helps. Please let me know if you have any further queries.
Hello, i want to ask some question. Is this method (from the link that you provided) is only works for append condition? If i have 100 rows from day one, and for the next day there are 2 rows that have been modified will that method also automatically update my 2 rows?
Hi @jashfabric
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.