Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The ultimate Microsoft Fabric, Power BI, Azure AI & SQL learning event! Join us in Las Vegas from March 26-28, 2024. Use code MSCUST for a $100 discount. Register Now

Reply
Cymbolz
Helper III
Helper III

Dataflow vs Dataset refresh

Haven't found any documentation on how data refresh works with respect to a dataflow and then a dataset sourced from that dataflow.

 

So looking for feedback, based on what I've discovered:

 

  • Both a dataflow and dataset need data to be refreshed
  • So I assume the dataflow is much like a data storage component on it's own that manages the updating from the data source, wherever that may be
  • And the dataset will refresh data from the dataflow 'storage'
  • Thus a logical refresh sequence (such as setting a scheduled refresh) would see the dataflow update first then the dataset aftewards (maybe 30 mins later as I suspect doing both at the same time may not yield the right results)

I've come to this conclusion after seeing the behaviour of having one or the other set for scheduled refresh.

 

I'm also seeing inconsistency in the workspace contents view where it shows last and next refresh times.  

 

For this dataflow, I've toggled off the scheduled refresh but it still shows a Next Refresh time (I would expect not to see any time stamp):

Capture 1.PNG

 

For these datasets, they have both had a refresh more recently than indicated here

 

 Capture 2.PNG

 

Here's the first one:

Capture 3.PNG

And the second:

Capture 4.PNG 

 

A bug?

 

2 ACCEPTED SOLUTIONS
otravers
Community Champion
Community Champion

>So I assume the dataflow is much like a data storage component on its own that manages the updating from the data source, wherever that may be

 

That's correct, technically PBI's Dataflow uses Azure Data Lake Gen2 for storage.

 

One use case I plan to use this dual refresh structure for, is to handle sources (e.g. static files) that don't need to be refreshed in Dataflows where they'll be imported but not under scheduled refresh. I've found PBI's scheduled refreshes to fail easily, so cutting down the service's scheduled refreshes to sources that actually need to be refreshed should lower incidents (e.g. web API timeouts, credential issues etc.).

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals

View solution in original post

Hi,

 

After internal checking, it seems that the next refresh time update (without browser refresh) was fixed and is should be available in the following updates.

 

I will keep monitor it.

 

Thanks,

Assaf

View solution in original post

28 REPLIES 28

The above link not going anywhere.

I think I've found the session here:

https://www.youtube.com/watch?v=m_oLq3uS238 

Hi,

NotifyOption in Power Automate custom connector for either dataflow or dataset refresh is not working for me.

Anyone recived mail on completion?

Assaf
Employee
Employee

Hi!

 

Regarding the inconsistency in the next refresh time of the dataflow, after a browser refresh, do you still see the next refresh time?

 

Thanks,

Assaf


@Assaf wrote:

 

Regarding the inconsistency in the next refresh time of the dataflow, after a browser refresh, do you still see the next refresh time?

 


I do see the refresh time after a browser refresh.  Also with the page remaining on screen, the refresh was scheduled to run and the date/time stamps updated without me having to refresh.

 

The issue I was experiencing included navigating away from that pgae, then returning to the page (so not refreshing the browser), so I'd have thought that would result in the updated time stamps too...I'll keep an eye on it.

Hi,

 

After internal checking, it seems that the next refresh time update (without browser refresh) was fixed and is should be available in the following updates.

 

I will keep monitor it.

 

Thanks,

Assaf

I've noticed the date & time stamps are now updating, even without any refresh or navigating away.

Good to hear. Thanks!

otravers
Community Champion
Community Champion

>So I assume the dataflow is much like a data storage component on its own that manages the updating from the data source, wherever that may be

 

That's correct, technically PBI's Dataflow uses Azure Data Lake Gen2 for storage.

 

One use case I plan to use this dual refresh structure for, is to handle sources (e.g. static files) that don't need to be refreshed in Dataflows where they'll be imported but not under scheduled refresh. I've found PBI's scheduled refreshes to fail easily, so cutting down the service's scheduled refreshes to sources that actually need to be refreshed should lower incidents (e.g. web API timeouts, credential issues etc.).

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals

Helpful resources

Announcements
March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Fabric Community Conference

Microsoft Fabric Community Conference

Join us at our first-ever Microsoft Fabric Community Conference, March 26-28, 2024 in Las Vegas with 100+ sessions by community experts and Microsoft engineering.

Fabric Partner Community

Microsoft Fabric Partner Community

Engage with the Fabric engineering team, hear of product updates, business opportunities, and resources in the Fabric Partner Community.

Top Solution Authors
Top Kudoed Authors