Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
I have a dataflow that contains data for the past 8 years. Most of the changes to that data are additions that would include a current date in the date column, so an incremental refresh process would capture those additions. However, if an input is made that is to an existing row of data, it makes that change and maintaines the original date. Also, if a deletion is made in the system to an input, which can be done to data that is 5 days old or 2 years old, it completely removes that row from the data source. Over 90% of the data is going to be additions to the existing data table, but enough changes will be made to historical data that it impacts the reports utilizing that data. The data only contains the one date column and doesn't include a seperate modification date column that we could utilize the detect changes option in the incremental refresh. We are needing the data refreshed throughout the day for certain reports that could be done utilizing an incremental refresh, but would also need to have a full refresh done once or twice a day to capture those potential changes to historical information. Is there a way to structure this within a dataflow to do both without having to do a manual process of turning off the incremental refresh? Or is there possibly a way to use power automate flows to do one or the other (incremental or full refresh) depending on another variable? If not, is there anything in development that might account for this scenario?
What you need is differential refresh, not incremental refresh. You need to severely rethink your strategy, since incremental refresh is fundamentally based on the assumption that "cold storage" partitions are no longer modified.
I recommend you re-read this article. Using incremental refresh with dataflows | Microsoft Docs It's actually pretty good (albeit disappointing).
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
58 | |
20 | |
19 | |
18 | |
9 |