PLEASE HELP - Incremental Refersh Refuses to work if you have 2 Web Datasources

[kmrastegar]

Hi Team,

Really need some help with this one as I have no idea why its happening.

Basically:

1. I am using a Web data source for my Power BI report. It is a REST API
2. It works fine offline, and online when i publish it using incremental refresh
3. I need a second source for the same report. I added another Web data source that is also a REST API
4. It works fine offline, and online when i publish it without incremental refresh on for THIS source
5. The Minute i turn on incremental refresh for both, incremental refresh fails. Here is the error: I have XXX out the API URL

[ValidateMarkupTags]{"error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","pbi.error":{"code":"DM_GWPipeline_Gateway_MashupDataAccessError","parameters":{},"details":[{"code":"DM_ErrorDetailNameCode_UnderlyingErrorCode","detail":{"type":1,"value":"[ccon]-2147467259"}},{"code":"DM_ErrorDetailNameCode_UnderlyingErrorMessage","detail":{"type":1,"value":"[ccon]Web.Contents failed to get contents from 'https://xxxxxxxxxxxxxxxxxxxxxxx' (504): Gateway Time-out[/ccon][/ccon]"}},{"code":"DM_ErrorDetailNameCode_UnderlyingHResult","detail":{"type":1,"value":"-2147467259"}},{"code":"Microsoft.Data.Mashup.ValueError.DataSourceKind","detail":{"type":1,"value":"Web"}},{"code":"Microsoft.Data.Mashup.ValueError.DataSourcePath","detail":{"type":1,"value":"[ccon]https://xxxxxxxxxxxxxxxxxxxxxxx[/ccon]"}},{"code":"Microsoft.Data.Mashup.ValueError.Reason","detail":{"type":1,"value":"DataSource.Error"}},{"code":"Microsoft.Data.Mashup.ValueError.Url","detail":{"type":1,"value":"[ccon]https://xxxxxxxxxxxxxxxxxxxxxxx[/ccon]"}}],"exceptionCulprit":1}}}

Now, there is not a problem with either of of my data sources OR their ability to incrementally refresh. I have troubleshooted this by:

1. Turning incremental refesh OFF on the first data source and ON on the second; upload, refreshes fine
2. Turning incremental refesh ON on the first data source and OFF on the second; upload, refreshes fine
3. Turning incremental refresh ON on the first data source and ON on the second; upload, refresh FAILS with the above error

What is interesting is the failure will rotate between API urls; its not always the same one. So its clearly not a problem with them individually, only when trying to refresh them together. This shouldnt be the case; pbi should be able to work with this.

Why is this? there should be no reason why this is failing but it is. Can someone help?

[v-yohua-msft]

Hi, @kmrastegar 

Have you solved the problem? If not, make sure that both web data sources have query folding. Query folding is essential for efficient incremental refresh. If query folding doesn't occur, the refresh operation may attempt to retrieve a large amount of data, resulting in a timeout. Hopefully, the documentation below will help you

Troubleshoot incremental refresh and real-time data in Power BI - Power BI | Microsoft Learn

Best Regards

Yongkang Hua

[kmrastegar]

The web source is an API; so it is returning a JSON. I dont think this suports Query Folding.

At the moment, I get around it by choosing a shorter "Incrementally refresh data starting" Time frame. if i only refresh 7 days, i can set archive at "1 year"

[v-yohua-msft]

Hi, @kmrastegar 

This issue may be due to certain limitations and resource conflicts when Power BI is handling incremental refreshes when two data sources are processed at the same time. While Power BI supports incremental refreshes for multiple data sources, refreshes can fail when data sources take a long time to process or when there is network latency.

The 504 Gateway Time-out error indicates that the requested data was not obtained from the gateway within the specified time, which may be due to insufficient gateway resources, slow API responses, or gateway configuration issues.

If you can, try setting different refresh times for both data sources to avoid resource contention caused by simultaneous refreshes.

How to Get Your Question Answered Quickly 

If it does not help, please provide more details with your desired output and pbix file without privacy information (or some sample data)

Best Regards

Yongkang Hua

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[kmrastegar]

I have an update on this that may provide further context;

my settings for Incremental Refresh are:

Archive data starting,  1 year

Incrementally refresh data starting, 1 month

I get the above issue with these settings.

If I change them to:

Archive data starting,  6 months

Incrementally refresh data starting, 1 month

It seems to work. This makes no sense as the refreshes are still loading 1 month of data so it cant be the API URLs timing out. Archive data timeframe is entirely on Microsofts side. 

Can anyone help with this so I can use longer archive periods?