cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
Frequent Visitor

Re: Data Flows Schedule refresh not working consistently - Urgent help needed

Which level of CPU and RAM consumption for the whole capacity do you reach while dataflows refreshes are slow?

Maybe during your business hours users interactivity don't allow to perform backgroud operations like dataflows refresh because of to high resources request. Interactive operations are always prioritized over background operations.

 

Highlighted
Helper V
Helper V

Re: Data Flows Schedule refresh not working consistently - Urgent help needed

Hello Buddy,

 

Well, the resources arent hitting the maximum utilization when the data flows are running . Below are some of the details shown in screen shot. 

 

I understand interactive operations should be given priority over the back end data operations. However, given the seperate space for data flows to run fast , I still dont see what is claimed as advantages of data flows are really working here. Below attached link is a youtube video which is recently released . Not to blame anyone, but full advantages of Data flows are yet to come .. 

 

https://www.youtube.com/watch?v=jEuECrCRVdY

 

Thanks,

G Venkatesh Capture.PNG

Highlighted
Frequent Visitor

Re: Data Flows Schedule refresh not working consistently - Urgent help needed

I think that could be the point. Too much resources requested during business hours and since dataflows is a backend process and since it's dynamically assigned memory you could end up with a "no end process" for dataflows refresh.

 

I would consider to move this refresh out of business hours if it's suitable to your scenario.

Highlighted
Helper V
Helper V

Re: Data Flows Schedule refresh not working consistently - Urgent help needed

I would have definitely done that to manage resources well between datasets and dataflows refresh. However, as I said, looking at the benefits of data flows , I conveyed my team to move reports that hold millions of records to start using data flows as data source . 

 

If I move refreshes out of business hours , People looking at this reports will see old data(stale) in business hours  . In this way, we are not showing the required information . So , I am like stuck on what to do here. 

 

G Venkatesh 

Highlighted
Frequent Visitor

Re: Data Flows Schedule refresh not working consistently - Urgent help needed

Keep in mind that datasets <> dataflows and they have completely different goals.

 

I usually use dataflows just to centralize and to standardize Data Preparation phase when I have the same table and M scripts repeated in many reports and if it could be refreshed asynchronously from datasets.

 

Obviously it depends by your scenarios.

Highlighted
Helper V
Helper V

Re: Data Flows Schedule refresh not working consistently - Urgent help needed

Sorry I think i confused you here a bit. I completely understand the differences between datasets and dataflows. 

 

Let me explain my scenario again . i work on service management data (ITSM modules) like Incidents, Work orders ,changes etc. 

we use relational databases sql as data source ( Get data) in Power BI desktop and design reports > Publish them to service to view reports.   However , we need to capture at least one year to 18 months of data to do trend analysis of incidents etc in few reports board members look at . But to just load 4 months of Incidents data into report, it is taking 20 to 30 mins time (data retrieval) .. and some times it gets timed out .  We are not happy with this way . 

 

Then i have read some where about data flows and felt that this could help me in resolving this issue . I have seen that data flows are fetching 18 months in just few mins which is a positive sign. But now, i need to refresh the entities and then data flows to show accurate data in reports ( users always like to see real time or near time data) . so, i started using incremental refresh for entities and schedule refresh for data flows.  I am stuck at this point . 

 

Some where, i read about Flows/Streaming datasets as well . But do not know whether there are helpful for my scenario. 

 

Sorry for  long notes. But I am really stuck  . Please assist !!!

Highlighted
Helper V
Helper V

Re: Data Flows Schedule refresh not working consistently - Urgent help needed

Hello Community,

 

Any other leads on this ? Is some one else facing the same issue and sorted out with best practices ?

 

Any kind of suggestions are welcomed . Kindly assist !! 

 

G Venkatesh

Helpful resources

Announcements
Community Blog

Community Blog

Visit our Community Blog for articles, guides, and information created by fellow community members.

Using the Community

Using the Community

Need help with the Power BI Community? Our 'Using the Community' support articles are a great place to start.

Community Summit North America

Community Summit North America

Innovate, Collaborate, Grow. The top training and networking event across the globe for Microsoft Business Applications

Power Platform 2020 release wave 2 plan

Power Platform 2020 release wave 2 plan

Features releasing from October 2020 through March 2021

Top Solution Authors
Top Kudoed Authors