Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
philthegaps
Helper I
Helper I

Streaming Dataset or Dataflow to store historical data ?

Hi there
 
I would like to store historical data on a daily basis to make trend KPI and I wonder which technology is the more relevant. Note that the old data are removed from the source (an SQL database with a custom OLEDB connector) and I don't really see if it can be stored into the dataflow or in the streaming dataset. My data model is not straighforward, and I don't think that simple json structures used in the streaming dataset could fit, so I guess dataflow should be the solution, however in all the examples I saw, the data are not deleted from the source, so I can't figure out what to do with Power BI service. 
1 ACCEPTED SOLUTION
GilbertQ
Super User
Super User

Hi there

If your source data is being deleted you could potentially store it in a dataflow. But if you ever needed to refresh the data you would loose it because the historical data is not there.

What I would suggest is to rather store the data from your original source into another database to keep the history.

And then load from the historical database?




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

View solution in original post

4 REPLIES 4
GilbertQ
Super User
Super User

Hi there

If your source data is being deleted you could potentially store it in a dataflow. But if you ever needed to refresh the data you would loose it because the historical data is not there.

What I would suggest is to rather store the data from your original source into another database to keep the history.

And then load from the historical database?




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Well if you are right, this is indeed the best solution.

Well as I can get the data from my sql source, I will create a scheduled task to extract them every week and store them in another database with a new "week number" column. It is the simpler solution.

Thanks for letting us know.




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors