cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
fafhrd
Frequent Visitor

Auto generation of incremental filter step for dataflow results in inefficient folding

The way incremental refresh is set up in Power BI (PBI) is not the same as for Dataflow. In PBI desktop you manually create the reserved word variables RangeStart and RangeEnd and manually create the filter step using those variables. This means you can place the filter step wherever you like in the Power Query (PQ).

 

With Dataflow, you have to switch on incremental refresh and the system generates the variables automatically. Then the system also inserts a filter step automatically at the end of the PQ. The problem is the end of the PQ is not usually the best place for an incremental partition step to occur. Furthermore, it is impossible to manually override order of this automatic step.

 

The problem is that code generated by the folding engine is more efficient if the filtering occurs as close to the start of the PQ as possible. I was hoping that the behaviour of setting up incremental refresh in Dataflow could be made more similar to how it is in PBI Desktop.

 

I have seen exactly one post about this issue before in 2019. So I hope there is a workaround that you could share with me for this issue.

2 REPLIES 2
Greg_Deckler
Super User
Super User

@fafhrd Dumb question because I haven't tried this out, but you can't use Advanced Editor to edit your dataflow and change things around?


@ me in replies or I'll lose your thread!!!
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Mastering Power BI 2nd Edition

@Greg_Deckler Yeah you can. Can also click and drag to rearrange in the PQ editor. But if you save and go back into the PQ, you will see the system puts the filter step back at the end automatically.

Your edit isn't respected if you hit save and don't touch it again as well. Can confirm this by rearranging the order, then saving, then exporting the .JSON of the Dataflow. You'll see the filter step is still at the end.

Helpful resources

Announcements
August 2022 update 768x460.jpg

Check it Out!

Click here to learn more about the August 2022 updates!

August 1 episode 9_no_dates 768x460.jpg

The Power BI Community Show

Watch the playback when Priya Sathy and Charles Webb discuss Datamarts! Kelly also shares Power BI Community updates.

Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Top Solution Authors