cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Anonymous
Not applicable

Incremental Refresh didn't save time

I have a very large dataset of over 25 million rows and I set up a 90-day incremental refresh for it. But the refresh took nearly as long as the usual refresh. Does anybody know how to reduce the incremental refresh time? 

1 ACCEPTED SOLUTION

Hi there

It would then appear that both of those data sources do not support query folding.

Which means that it has to load all the data first, and then determine which data to load incrementally. In doing it in this manner it will not be any quicker using incremental refreshing.




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

View solution in original post

13 REPLIES 13
GilbertQ
Super User II
Super User II

Hi there

When you say you set up the incremental refresh what is your data source?

If it is not a data source where query folding can happen, or where it cannot read individual data it will have to read all the data first, before it can incrementally load the new data.




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Anonymous
Not applicable

Hi,

 

I tried to use Dataflow and Snowflake. But they were both slow. I saw the warning of query folding when I set it up. Is there any other way I can speed up the refresh without changing the data source? 

 

Thanks,

What is your data source?




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Anonymous
Not applicable

Snowflake and Dataflow 

Thanks for that.

As far as I am aware you currently cannot do incremental refreshes on a dataflow.

Have you enabled the Enhanced Dataflow ability in your Premium settings? That might help with the dataflows?

And I think the same happens for Snowflake in that it cannot create a query to just get the latest data, so it has to read all the data first to only load the incremental part.




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Anonymous
Not applicable

Thanks for the information. I applied incremental refresh for dataflow. But it shows the same warning as Snowflake: unable to confirm if the m query can be folded it is not recommended to use incremental refresh with non-foldable queries

 

Do you know what kind of data source support query folding? 

 

Thanks,

It would appear to be from the following documentation

https://docs.microsoft.com/en-us/power-bi/service-premium-incremental-refresh#limitations

If you read further up it says it should display the warning.

Are both the snowflake and the dataflow used in the same PBIX?




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Anonymous
Not applicable

No. I tried the same pbix file with two data sources and they both didn't work. 

Hi @Anonymous 

 

I am fairly certain that if you are combining the 2 data source that the incremental refreshing process has to load all the data before it can incrementally load the data.

 

And that is why you are not seeing it saving any more time.

 

What happens if you try them individually?





Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Anonymous
Not applicable

Hi @GilbertQ ,

 

Thank you for your advice. I didn't make it clearly. I use dataflow and snowflake individually for incremental refresh and they both didn't work. 

 

Thanks,

Hi there

It would then appear that both of those data sources do not support query folding.

Which means that it has to load all the data first, and then determine which data to load incrementally. In doing it in this manner it will not be any quicker using incremental refreshing.




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

View solution in original post

Anonymous
Not applicable

Hi @GilbertQ ,

 

Thanks for the info. 

No worries, glad to help

Incremental refreshing can be a bit of a challenge at times that is for sure. But when it works, it works really well!




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!







Power BI Blog

Helpful resources

Announcements
PBI User Groups

Welcome to the User Group Public Preview

Check out new user group experience and if you are a leader please create your group!

MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

July 2021 Update 768x460.png

Check it out!

Click here to read more about the July 2021 Updates

Power Query PA Forum 768x460.png

Check it out!

Did you know that you can visit the Power Query Forum in Power BI and now Power Apps

Top Solution Authors