cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
SamRock Regular Visitor
Regular Visitor

Dataflow +Dataset = NEED more Clarity!

Hi All,
I have successfully built an Enterprise Dataflow that loads all my tables in PBI service. Its incremental, scheduled and all is good! Now I use PowerBI Desktop to consume my Dataflow:
PS: I know I sound, irritated and angry, but this is after putting all hopes into this feature, waiting for a long time for it to be available!

SHOCKER #1: Why is PowerBI Desktop downloading EVERYTHING from Dataflow into my DATASET? Shouldn't it be a Thin-client like connection to the Dataflow? What was the purpose of having an on Cloud-ETL , if everything gets downloaded to my local desktop? I am talking above GBs of enterprise data here (which I could limit using top 100 using onpremise connection to build my Model)
Here is one of my Dataflow getting "downloaded" to my PBI Desktop. It contain 53M rows.... and its been 10+ mins since the download is running!

EDIT: 84 GB data getting downloaded from my Dataflow on my Desktop! This makes NO SENSE!
dataflow-download.PNG
Power BI -Dataflow.PNG

After about 30 mins, I get this error! What I am supposed to do now?
Power BI -Dataflow-error.PNG

Anyways, I try again patiently wait for the Dataflow to complete its download on my desktop, then I build my Datamodel, join my tables, create Lovely reports and publish it to my Premium Workspacem then... boom..


SHOCKER #2:
My new DATASET doesn't automatically refresh data from DATAFLOW either! I am told that I need to schedule the DATASET(which is supposed to be already linked to Dataflows) to be refreshed too! WHY? I again ask, whats then the purpose of preloading data on Dataflows? Why doesnt Dataset automcatially refreshes itself when Dataflow is refreshed?
Why are none of your articles talking about these two limitations? They all made it look like Datset will connect in a "Direct Query" or "Live connection" method to the Dataflows. Noboday says that you need to download the entire data and then refresh your datasets as well!


SHOCKER #3:

EDIT: MS has fixed this restriction on the recent PBI Services update! Thank you!
You need to be a Gateway Admin to access on Premises data sources? Why? We have close to 800 Power BI Pro developer. We cant make everyone an "Admin" to the Gateway. If a User has been granted access to a particular Gateway connection, he needs to be able to use it to build a Dataflow. This again defeats the purpose of Dataflow. Its very restricting! Now a Gateway admin needs to help developers build Dataflows becuase granting access to Enterprise Gateway (with 100s of connections) is too risky!
Other posts the confirm my fear

https://community.powerbi.com/t5/Service/Dataset-does-not-reflect-latest-data-from-dataflow/td-p/585...

https://community.powerbi.com/t5/Service/Dataflow-vs-Dataset-refresh/m-p/572076#M57973

20 REPLIES 20
SamRock Regular Visitor
Regular Visitor

Re: Dataflow +Dataset = MAJOR Confusion!

No response?? :'(

I have a Dataflow Entity that has about 54Million record.s
I am trying to create a report using this Dataflow and everytime, Power BI Desktop trys to download 10s of GB of data from the Dataflow. This does not make any sense!

If I multiple this with the other Entities that have similar volume, I will never be able to create a report on my Desktop!
Dataflow-refresh.PNG

SamRock Regular Visitor
Regular Visitor

Re: Dataflow +Dataset = MAJOR Confusion!

Still no response!

RBunting Regular Visitor
Regular Visitor

Re: Dataflow +Dataset = MAJOR Confusion!

@SamRockI saw your reply in our thread about dataflow timeouts.

 

I don't think you should need to load your entire dataflow.  When I open PBI Desktop and click 'Get Data,' then pick 'Power BI Dataflows,' and go through the wizard to eventually select some specific dataflow that is deployed to the service, the next thing that happens is the Power Query Editor opens and creates a new query pointing to the selected dataflow as it's source.

 

You should be able to apply whatever filters and limiters you want at this point before loading the data to your model.  You could curtail that 54 million rows by whatever degree you need. 

 

Highlighted
parishdapbi Frequent Visitor
Frequent Visitor

Re: Dataflow +Dataset = NEED more Clarity!

I can’t find Twitter link, but saw one today claiming that Admin restriction was changed for Dataflows
parishdapbi Frequent Visitor
Frequent Visitor

Re: Dataflow +Dataset = NEED more Clarity!

https://twitter.com/thomasfosterrrr/status/1159858398936096768?s=21 “You can now use data flows with enterprise gateways as a non admin user!”
SamRock Regular Visitor
Regular Visitor

Re: Dataflow +Dataset = MAJOR Confusion!

@RBunting Thanks for the response!

My requriemnt is to show historical data/trends and I need to have all the data from this Dataflow. Of course, there will be filters for Year/Month/Week and Region to ensure not everything is loaded at once.
This is the reson I decided to go with a Dataflow to pre-load my data into Cloud.

Even if I do apply a filter, the data is still downloaded from Dataflow , its still going to be GBs worth of data everytime I need to fetch new data for my development. Do you think I am doing something wrong here?

parishdapbi Frequent Visitor
Frequent Visitor

Re: Dataflow +Dataset = MAJOR Confusion!

Ideally, if you have Power BI Premium, you could use Incremental Refresh, both in loading data to the dataflow and to incrementally load new data into the dataset. Otherwise, you have to load all of the the data all the time. Only way around without Premium would be to create a table with history data that is not refreshed, then another table with currrent data. In a DAX measure you could combine the values, such as: Sales = SalesHistory[Units] + SalesCurrent[Units}.

SamRock Regular Visitor
Regular Visitor

Re: Dataflow +Dataset = MAJOR Confusion!

Thanks @parishdapbi 
We do have Premium capacity and I am using Incremental Refresh on the Dataflow.

I liked the idea of having Incremental refresh on the Dataset, however I always thought Dataset will have a live connection to Dataflows. This is really disappointing. 
And I hope MS does something to only enable a preview Data during development using Dataflow and not force use to download complete data onto desktop.

RBunting Regular Visitor
Regular Visitor

Re: Dataflow +Dataset = MAJOR Confusion!

Direct Query connections to dataflow is 'under review' in the ideas forum.  You should vote it up.

 

https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/35976663-dataflows-direct-query

 

Helpful resources

Announcements
New Topics Started Badges Coming

New Topics Started Badges Coming

We're releasing new versions of the badge that everyone's talking about. ;) Check your inbox for notifications.

MBAS 2020

Save the new date (and location)!

Our business applications community is growing—so we needed a different venue, resulting in a new date and location. See you there!

Difinity Conference

Difinity Conference

The largest Power BI, Power Platform, and Data conference in New Zealand

Power Platform 2019 release wave 2 plan

Power Platform 2019 release wave 2 plan

Features releasing from October 2019 through March 2020

Top Solution Authors
Top Kudoed Authors (Last 30 Days)