Which level of CPU and RAM consumption for the whole capacity do you reach while dataflows refreshes are slow?
Maybe during your business hours users interactivity don't allow to perform backgroud operations like dataflows refresh because of to high resources request. Interactive operations are always prioritized over background operations.
Well, the resources arent hitting the maximum utilization when the data flows are running . Below are some of the details shown in screen shot.
I understand interactive operations should be given priority over the back end data operations. However, given the seperate space for data flows to run fast , I still dont see what is claimed as advantages of data flows are really working here. Below attached link is a youtube video which is recently released . Not to blame anyone, but full advantages of Data flows are yet to come ..
I think that could be the point. Too much resources requested during business hours and since dataflows is a backend process and since it's dynamically assigned memory you could end up with a "no end process" for dataflows refresh.
I would consider to move this refresh out of business hours if it's suitable to your scenario.
I would have definitely done that to manage resources well between datasets and dataflows refresh. However, as I said, looking at the benefits of data flows , I conveyed my team to move reports that hold millions of records to start using data flows as data source .
If I move refreshes out of business hours , People looking at this reports will see old data(stale) in business hours . In this way, we are not showing the required information . So , I am like stuck on what to do here.
Keep in mind that datasets <> dataflows and they have completely different goals.
I usually use dataflows just to centralize and to standardize Data Preparation phase when I have the same table and M scripts repeated in many reports and if it could be refreshed asynchronously from datasets.
Sorry I think i confused you here a bit. I completely understand the differences between datasets and dataflows.
Let me explain my scenario again . i work on service management data (ITSM modules) like Incidents, Work orders ,changes etc.
we use relational databases sql as data source ( Get data) in Power BI desktop and design reports > Publish them to service to view reports. However , we need to capture at least one year to 18 months of data to do trend analysis of incidents etc in few reports board members look at . But to just load 4 months of Incidents data into report, it is taking 20 to 30 mins time (data retrieval) .. and some times it gets timed out . We are not happy with this way .
Then i have read some where about data flows and felt that this could help me in resolving this issue . I have seen that data flows are fetching 18 months in just few mins which is a positive sign. But now, i need to refresh the entities and then data flows to show accurate data in reports ( users always like to see real time or near time data) . so, i started using incremental refresh for entities and schedule refresh for data flows. I am stuck at this point .
Some where, i read about Flows/Streaming datasets as well . But do not know whether there are helpful for my scenario.
Sorry for long notes. But I am really stuck . Please assist !!!