dataflow incremental refresh slower then full refresh

[Braxx99]

I have a dataflow with 3 tables. 2 of them have the incremental refresh applied.  

- the dataflow takes about 1:57min to complete with incremental mode

- when incremental refresh is turn off it takes about 48min

That's a problem cos I expected incremental would be faster.

Let's take one of these incrementaly refreshed table to examin closely.

- it is configured to refreshed last 45 days, although 48 partitions are created

- the refresh starts at 13:08

- 32 out of 48 partitions are refreshed in parallel starting at 13:08

- rest of te partitions waiting in queue. The last two ones begin refresh at 14:09

here is a screenshot from dataflow refresh logs. (not all the columns and partition visible)

questions:

1) why not all of the partitions are refreshed in parallel? 

2) what can I do to be able to refresh all in parallel? I guess this is a problem why the incremental refresh takes so long. The partition which is refreshed at 14:09 need 37min to complete. 

[Braxx99]

@v-kongfanf-msft any updates? Thanks in advance

[v-kongfanf-msft]

Hi @Braxx99 ,

Detailed documentation on incremental refresh and real-time data troubleshooting is officially available. Refer to below document:

Troubleshoot incremental refresh and real-time data in Power BI - Power BI | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Braxx99]

Thanks for the link.

Unfortunatelly, I could not find answer to my questions there. 

What I am particulary interested in is what are the possible reasons some of the partitions are queued?

The source of the table mentioned earier is SQL Server on-premises. It has about 12mln rows and cover daily data from 2019-01-02