Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more.
Get startedGrow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.
I need a way to perform a "full" refresh of my 10 days window once a day with incremental refreshes in between.
Background
I have a very large datasource with billion of records in the future and past.
I only need a current sub-set of these records at a time. But even this sub-set is very large (approx. 500mio recrods each).
The reports need to be very up-to-date, but direct query is not an option for legal reasons.
Most of the data (98%) does not change. So a full refresh at this order of magnitute every 5min or so does not seem like the way to go.
Incremental refresh appears suitable.
Problem is my rolling window of only 10 current days.
Much of the data will not have changed since the last incremental refresh, thus not being pulled once the windows shifts one day.
I would need a way to perform a "full" refresh of my 10 days window once a day with incremental refreshes after that.
Anyone knows of a way to do this prrgamatically?
Many thanks for your input!
Jan
Wondering if you have found a solution to this issue
Hey @JSiebrecht ,
my 1st idea is to an incremental refresh by using the xmla endpoint, but this requires some kind of Premium capacity: Advanced incremental refresh with the XMLA endpoint in Power BI - Power BI | Microsoft Docs
Another solution is to create a view on top of your data source, that filters the required data in advance. Then you can publish your dataset each day (or whenever you want) and perform an initial load, after that you can use the incremental load.
Hopefully, this requires some ideas on how to tackle your challenge.
Regards,
Tom
Hi @TomMartens ,
thanks for your input!
My dataset is hosted in a P2 SKU Premium Capacity. If this gives options.
To publish the dataset each day for the next X years, I don't hink that's a viable option for this kind of business process.