cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Fjcgomes
Advocate I
Advocate I

Error in incremental refresh of a Dataflow over a Google Bigquery data source

Good morning guys,
I'm trying to refresh a dataflow that will have all the necessary data for DI&A teamDashboards, This dataflows is loading about 50 different entities and the biggest has 2.5 Million rows.


In this case I'm only loading one year of history and the incremental refresh will only load the last 3 days. However I'm not succeeding with this refresh
Ex: 

Fjcgomes_0-1618305546334.png

 

This is the returned Error Message:

Error: Data Source Error : DataSource.Error: ERROR [HY000] [Microsoft][BigQuery] (31780) Not found. Exceeded rate limits: too many api requests per user per method for this user_method. For more information  see https://cloud.google.com/bigquery/troubleshooting-errors .. RootActivityId = 59651e37-4ca3-4347-b904-8343a5b546e8.Param1 = DataSource.Error: ERROR [HY000] [Microsoft][BigQuery] (31780) Not found. Exceeded rate limits: too many api requests per user per method for this user_method. For more information  see https://cloud.google.com/bigquery/troubleshooting-errors  Request ID: 8864254e-a992-4c7c-bfb0-0bcc58ea7f5b.


I'm testing it on a Worsksapace PPE License, and I have exactly the same dataflow on a Workspace Pro License without incremental refresh and it works perfectly (despite the exaggerated refresh time).

Has anyone experienced something similar? Any suggestion?

Thanks a lot!

3 REPLIES 3
lbendlin
Super User III
Super User III

Copy the query to Power BI desktop and run Power Query diagnosis on your query - it will show you the full request traffic.

lbendlin
Super User III
Super User III

As you can see from the error message the incremental refresh process is too chatty for the Big Query server's taste.  Can you increase your API call rate limits?

hi @lbendlin, i already checked my quotas and in terms of requests I dont' have any limitation:

  • BigQuery API
    • Requests per day - Unlimited 
    • Extract bytes per day - Unlimited
    • Cloud SQL federated query cross region bytes per day - 1.1 TbI
    • IamPolicy requests per 100 seconds - 5000 (the maximum by default) 

What i saw is that with whis kind of Limit Quotas i shouldn't have any contstarins or issues. 
Could it be any issue with the Power BI default connector?
I'm runnig out without ideas to solve this.

Helpful resources

Announcements
MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

July 2021 Update 768x460.png

Check it out!

Click here to read more about the July 2021 Updates

Power Query PA Forum 768x460.png

Check it out!

Did you know that you can visit the Power Query Forum in Power BI and now Power Apps