Historic data (last 4 days data), dataflow vs dataset ?

[Anonymous]

I have premium capacity p1 workspace

I have an oracle pipeline query that gets data for the previous working day only (more than a million rows)

business need is to get data , at any given time, for the last 4 days...

what should be the best possible approach for this, is dataflow a better approach ?

i am right now experimenting with power bi dataset import mode, since 30 minutes screen shows loading from...so i think its loading into azure ?

how do I get historic data into my report if the business need is to get data for last 4 days ? mind you i cannot expand the pipeline query where clause to get 4 days data, each day is 1 million rows, so it has to be one day at a time

please suggest how to show historic data in the report ? and if dataflow is faster compared to dataset ?

[Anonymous]

Trying to apply incremental refresh to my DQ dataset in power bi desktop, I will close this topic once its done, if thats ok, also when I right click on the table (after transform data), I dont see see native query enabled, my understanding is it needs to be enabled for incremental refresh to work

and in general do you think incremental refresh makes sense for direct query ? cause its direct query and there's no refresh concept at all

[Anonymous]

Closing this topic, actually I could not even specify parameters to direct query dataset in desktop

[aj1973]

Hi @Anonymous 

Your questions are really clear;

"what should be the best possible approach for this, is dataflow a better approach ?"

Better approach than what?

"i am right now experimenting with power bi dataset import mode, since 30 minutes screen shows loading from...so i think its loading into azure ?"
Are you talking about after publishing to the service or in your desktop?

"how do I get historic data into my report if the business need is to get data for last 4 days ?"

Are you talking about how to refresh your report?

please suggest how to show historic data in the report ?

It depends where the source is, a Gateway is a solution

and if dataflow is faster compared to dataset ?

Can't campare as the Dataflow will become the source for your dataset.

[Anonymous]

This (below screenshot) is in power bi desktop import mode, does that mean the data is being loaded into azure ?

[aj1973]

No, It's been loaded into your desktop, into the Dataset File.

[Anonymous]

So my data is 3 million rows, takes forever to load in the power bi desktop. (import mode)

is it going to be the same refresh time when I upload it to service (import mode)

[aj1973]

@Anonymous 

Not sure about 3 million rows will take for ever! besides you are gonna have to do some ETL to your data, and build a dataset. You are gonna keep only the data you need for your reports. Direct Query mode is another way to bring your data.

Refreshing the data in the service: there are other techinques to make the refresh takes less time, like using paramaters or incremental refresh.

[Anonymous]

Thank you so much, one last question, is incremental refresh possible only with import mode ? I am having a timeout issue with dataflow eventhough i set command timeout to be 60 minutes but i opened another thread for it

[aj1973]

No both Import and DQ but maybe with some limitations, here is a link to follow

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview 

https://docs.microsoft.com/en-us/power-bi/guidance/power-query-folding

Where is the thread to your other issue?

[Anonymous]

The dataflow topic is here

https://community.powerbi.com/t5/Service/dataflow-operation-timed-out/m-p/1798401#M126631

[aj1973 · Accepted solution]

Can you just close this thread by accepting it, if it is solved

Thanks