cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
sedareports
Advocate II
Advocate II

Incremental Refresh with Non Native Query Transformations

Can I still utilize incremental refresh as long as a execute the incremental refresh (RangeStart/End parameters) before I break my query folding with a bunch of transformations? My dataset is pretty large, and incremental refresh is hard to test as the intial load regularly fails.

 

I was thinking that maybe it depends on the kind of transformations, so here are a few examples of what I am using:

 - merge queries to bring in key attributes from other tables

 - add a calculated column

 - remove duplicates

 - sort rows after removing records in order to add a clean index (i.e. one that can be used later in DAX to calc differences between records).

5 REPLIES 5
yingyinr
Community Support
Community Support

Hi @sedareports ,

The prerequisite for incremental refresh to work properly is that the data source used must support query folding. Most data sources that support SQL queries support query folding. Please check whether the following documentations can help you resolve your problem.

Dealing with Large Datasets

CHECKING QUERY FOLDING WITH VIEW NATIVE QUERY

Query Folding & Incremental Refresh

Not Folding; the Black Hole of Power Query Performance

Best Regards

Rena

Community Support Team _ Rena
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Thanks @yingyinr for the reference links. I should have clarified, my initial query can be folded. But subsequent transformations (after the folding and using RangeStart/RangeEnd) are performed. My question is what transformations after a query fold that would still cause the entire querying of the dataset (i.e. igore the query folding?)

 

 

query folding stops with the change type transformation:

let
    StartDate = "'"&DateTime.ToText(RangeStart)&"'",
    EndDate = "'"&DateTime.ToText(RangeEnd)&"'",
    SCADATags= "(" & Text.Combine(List.Transform(#"3SCADAFlowTags"[SCADAID], each "'"&_&"'"),",") & ")",
    Source = Odbc.Query("Driver={PostgreSQL ANSI(x64)};server=XXXX;port=5432;Database=XXXX", "SELECT * FROM XXXXX where scadaid in "&SCADATags&" and measurementdatetime >= "&StartDate&" and measurementdatetime < "&EndDate&" and measurementquality = '100'"),
  //query folding stops 
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"measurementvalue", type number}}),
    #"Merged Queries" = Table.NestedJoin(#"Changed Type", {"scadaid"}, #"3SCADAFlowTags", {"SCADAID"}, "SCADAFlowTags", JoinKind.FullOuter),
    #"Expanded SCADAFlowTags1" = Table.ExpandTableColumn(#"Merged Queries", "SCADAFlowTags", {"Description", "FromServiceArea", "MeterError", "ToServiceArea"}, {"Description", "FromServiceArea", "MeterError", "ToServiceArea"}),
    #"Added Custom" = Table.AddColumn(#"Expanded SCADAFlowTags1", "Date", each DateTime.Date([measurementdatetime])),
    #"Changed Type2" = Table.TransformColumnTypes(#"Added Custom",{{"MeterError", type number}}),
    #"Meter Error Calc" = Table.AddColumn(#"Changed Type2", "Multiplication", each -1*[measurementvalue]/(1+[MeterError]), type number),
    #"Renamed Columns" = Table.RenameColumns(#"Meter Error Calc",{{"Multiplication", "measurementvaluefinal"}}),
    #"Removed Duplicates1" = Table.Distinct(#"Renamed Columns", {"scadaid", "measurementvalue", "measurementdatetime","Description"}),
    #"Filtered Rows" = Table.SelectRows(#"Removed Duplicates1", each ([scadaid] <> null)),
    #"Sorted Rows" = Table.Sort(#"Filtered Rows",{{"scadaid", Order.Ascending}, {"measurementdatetime", Order.Ascending}}),
    #"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 0, 1),
    #"Removed Columns" = Table.RemoveColumns(#"Added Index",{"ToServiceArea","measurementtype","networkfacilityid","measurementquality","objectid"}),
    #"Renamed Columns1" = Table.RenameColumns(#"Removed Columns",{{"FromServiceArea", "ServiceArea"}}),
    #"Changed Type1" = Table.TransformColumnTypes(#"Renamed Columns1",{{"Date", type date}}),
    #"Removed Columns1" = Table.RemoveColumns(#"Changed Type1",{"MeterError"})
in
    #"Removed Columns1"

 

 

 

 

Hi @sedareports ,

I'm so sorry I'm not clear about your problem. Could you please provide more details on it?

Best Regards

Rena

Community Support Team _ Rena
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

My question/problem is, can a user perform transformations after a query folding and still maintain incremental refresh? I.e. will the subseqent tranformations only be applied to new incrementally loaded data, or will they possible trigger a full query/processing of the entire data set.

For example.. a sorting transformation or applying an index. 

Hi @sedareports ,

Hope the following documentations can help you.

Power Query query folding

Query Folding in Power Query to Improve Performance

Not Folding; the Black Hole of Power Query Performance

Best Regards

Rena

Community Support Team _ Rena
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
June 2022 update 768X460.jpg

Check it out!

Click here to read more about the June 2022 updates!

Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Power BI Dev Camp Session 23 768x460.jpg

Check it Out!

Mark your calendars and join us on Thursday, June 30 at 11a PDT for a great session with Ted Pattison!

June 20 episode 7 with aka link 768x460.jpg

The Power BI Community Show

Join us on June 20 at 11 am PDT when Kim Manis shares the latest on Azure Synapse analytics, the Microsoft Intelligent Data Platform, and notable Power BI Updates from Microsoft Build 2022.

Top Solution Authors
Top Kudoed Authors