cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
SamRock
Advocate III
Advocate III

Dataflow - Validating Queries FOREVER + Gateway issues

I been a big evangalist for usage of Dataflow within my 60K something organization. Now all I hear is complains on how slow and unreliable it has become. And I personally experience it too. 


(PS: We own a P3 and two P2 SKUs)
To begin with preview data takes ages to show up, I am literally scared to add another step to an Entity! 
After all this I click "Save" and the dread "Validating Queries" runs forever. There is noway to tell if it wil suceed or not! I usually am greated with "Mash could not be retrieved" and there goes all my hours of efforts. Why do you need to do these multi validation? When the preview is generated, arent the validations already completed?
Dataflow-Validating.PNG

 Secondly, non-Gateway Admins (normal user with permission to a Gateway connection) are still not able to create an entity. PBI Service keeps telling them that they need to be a Gateway Administrator! WHY? I thought this issue was resloved months ago. Why are my users still getting this message?

PowerBI-Dataflow-Admin.PNG

7 REPLIES 7
Daryl-Lynch-Bzy
Frequent Visitor

@SamRock  I agree the performance of the Validating Output Schema when saving changes to a Dataflow is too slow.   There is no correlation between Dataflow refresh and Validation upon save and close.

 

I can wait up to an hour for the Validation upon saving to finish.  When done, the refresh takes less than 15 secounds.

trutz
Advocate II
Advocate II

Hi all,

I also see these issues. Loading a small dataset and added an AutoML calculation, the validation already took like 10 minutes. Now I added a table with 16 rows to be forecasted and applied the AutoML model and the validation takes over an hour and is still running 😞

Robertostaltari
Frequent Visitor

I went through the same path: advocated dataflows as the best solution but the reality is sad.

Validating queries takes ages even for querying a single file on a SharePoint. Why is that?

 

Annotation 2020-07-31 151607.png

 

 

There is a best Solution? Where? Working on Dataflow day by day is turning out to be a bad exprience.

I found that DirectQuery with Dataflow is 10 times slower than doing a DirectQuery from my 6 years old SQL Server, via a Enterprise Gateway. 

Can you imagine? Power BI Service to Power BI Service read takes more time than reading from an Power BI Service to OnPremise DB! 😄 

v-kelly-msft
Community Support
Community Support

Hi @SamRock ,

 

Question 1:

Which kind of data sources does the dataflow get data from? Is there any specific features made to the entity, eg: Merge/Append queries? 

 

Question 2:

There are a few known limitations to using Enterprise Gateways and dataflows:

  • Each dataflow may use only one gateway. As such, all queries should be configured using the same gateway.
  • Changing the gateway impact the entire dataflow.
  • If several gateways are needed, the best practice is to build several dataflows (one for each gateway) and use the compute or entity reference capabilities to unify the data.
  • Dataflows are only supported using enterprise gateways. Personal gateways will not be available for selection in the drop down lists and settings screens.

So it requires to assign an administator role of the gateway to others once you want them to add entities to the dataflow.

 

Here is the reference.

https://docs.microsoft.com/en-us/power-bi/service-dataflows-on-premises-gateways

 

Best Regards,
Kelly
Did I answer your question? Mark my post as a solution!

 

 

Hi @v-kelly-msft 

Thank you for the quick reponse! 

1: Its an ODBC connection to Hive Database. We use the HortonWorks ODBC Drivers. There are no additional steps after reading the source. Since about 1 week, I am seeing this error message a lot Dataflow - Mashup.PNG

2. We are using the Enterprise Gateway. There is only one gateway which has not been changed. The error message you see has to do with permissions. I as an Gateway Admin can create Entities, however my team mates who are just Gateway Connection Users are unable to create Entities. However they are able to use the same connection to schedule their Datasets.   
This issue was fixed by Microsoft in Septemebr 2019. But we are still experiencing this error message.
https://powerbi.microsoft.com/en-us/blog/power-bi-dataflows-september-2019-feature-summary/

same here - multiple types of source systems (ODBC to HIVE/Hadoop, MS SQL Server, etc.) - enterprise gateway, P1 capacity, have been using dataflows for a long time with great success. Now recently, these problems are preventing me from making several needed updates...

Helpful resources

Announcements
PBI_User Group Leader_768x460.jpg

Manage your user group events

Check out the News & Announcements to learn more.

Get Ready for Power BI Dev Camp

Microsoft named a Leader in The Forrester Wave

Microsoft received the highest score of any vendor in both the strategy and current offering categories.

Get Ready for Power BI Dev Camp

Power BI Dev Camp - September 30th, 2021

Mark your calendars and join us for our next Power BI Dev Camp!

PowerPlatform 768x460.png

Microsoft Learn

Check out our new Discover Your Career Path blog post series and get all the details.