Best Practices: Hybrid Data Refresh for Intraday & Historical Data (Power BI)

[manoj_0911]

Hi everyone,

I'm seeking insights from experienced Power BI users regarding data refresh strategies. I'm working on a scenario where we need to combine intraday refreshes for real-time data with daily refreshes for historical data to improve efficiency.

Here's the approach I'm considering:

* Schedule daily refreshes: Import the bulk of historical data once a day using a scheduled refresh. This is the larger dataset that likely updates less frequently.
* Set up intraday refreshes: Configure an additional refresh to update the most recent data (e.g., the last 24 hours) throughout the day. This ensures we have the latest information for real-time analysis.

Questions for the Community:

* Is this a recommended approach for handling mixed refresh requirements?
* Are there any potential challenges or considerations I should be aware of with this hybrid model?
* Have any of you implemented similar strategies? If so, what best practices can you share for managing data consistency and avoiding duplication issues?

I'm open to exploring alternative approaches or best practices that the community might suggest.

Thanks in advance for your valuable insights!

Best regards,

Manoj Prabhakar

[v-junyant-msft]

Hi @manoj_0911 ,

In my opinion, there may be a few things you need to be aware of:
1. Performance Impact: Frequent intraday refreshes, especially on large datasets, can impact performance. It's crucial to monitor and optimize the data model and queries to mitigate this.
2. Data Consistency: Ensuring consistency between the historical and real-time data segments can be challenging. Implementing a hybrid table approach, where a table operates in both import and DirectQuery modes, can help maintain consistency without significant performance penalties.
3. Licensing and Capacity: Depending on the volume of data and refresh frequency, you might need to consider Power BI Premium capacity for its enhanced refresh capabilities and performance benefits.

I suggest you to configure your fact tables as hybrid tables, with one or multiple Import partitions and one DirectQuery partition. This setup allows efficient querying from in-memory while including the latest data changes from the source.
Semantic model modes in the Power BI service - Power BI | Microsoft Learn
And please ensure related tables are in Dual storage mode to support both cached (import) and uncached (DirectQuery) operations efficiently. This is particularly important for maintaining performance and consistency in hybrid models.
Troubleshoot incremental refresh and real-time data in Power BI - Power BI | Microsoft Learn
What's more, for real-time dashboards, consider using automatic page refresh in Power BI to keep visuals up-to-date without manual intervention. This feature is particularly useful for reports based on DirectQuery data sources.
Automatic page refresh in Power BI Desktop - Power BI | Microsoft Learn

Finally, please note that balance the frequency of intraday refreshes to avoid overloading the system while ensuring data is up-to-date. Utilize Power BI's monitoring tools to track performance and adjust schedules as needed.

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.