Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
joshua1990
Post Prodigy
Post Prodigy

Will Date as Number break Query Folding / Incremental refresh?

Hello together!

I am accessing a transactional table (oracle / ODBC Connection type) that has several million rows.
I would like to update this table based on incremental refresh in Power BI Desktop.
This table has a column "Date Entry" which is a number in the format "CYYDDD".
For example, this means that the date is 01/01/2019 = 119001.
To put this number in a date format like "DD.MM.YYYY" I have already found a function.
But this function will stop the query folding. Anyway, after this step, I can't see a Native Query anymore.

 

Now I have the question if I can still somehow apply the incremental refresh?
What recommendations do you guys have?

 

3 REPLIES 3
artemus
Employee
Employee

Here is one way to do it:

Replace the Source line with your table. It expects a [DateEntry] column with the date string

 

 

let
    Source = ...,
    #"Filtered Rows" = Table.SelectRows(Source, each [AddressID] = 1),
    #"Added Custom" = Table.AddColumn(#"Filtered Rows", "DateEntry", each 107263, type number),
    Custom2 = Table.AddColumn(#"Added Custom", "Century",each Number.RoundDown([DateEntry] / 100000) + 19, Int32.Type),
    #"Added Custom1" = Table.AddColumn(Custom2, "Year", each Number.RoundDown(Number.Mod([DateEntry], 100000) / 1000), Int32.Type),
    #"Added Custom2" = Table.AddColumn(#"Added Custom1", "LeapYearOffset", each if Number.Mod([Year], 4) = 0 and (Number.Mod([Century], 4) = 0
or [Year] <> 0) then 1 else 0, Int32.Type),
    #"Added Custom3" = Table.AddColumn(#"Added Custom2", "MonthDayPart", each Number.Mod([DateEntry], 1000)),
    #"Added Custom4" = Table.AddColumn(#"Added Custom3", "MonthDay", let monthTable = #table(type table [MoNum = Int32.Type, Days = Int32.Type], List.Zip({{1 .. 12}, List.Accumulate({31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, {}, (current, next) => current & {List.Last(current, 0) + next})})) in Expression.Evaluate("each " & Text.Combine(Table.TransformRows(monthTable, each "if [MonthDayPart] <= " & Text.From([Days]) & (if [MoNum] = 1 then "" else " + [LeapYearOffset]") & " then " & Text.From([MoNum]) & " + ([MonthDayPart] - " & Text.From(monthTable{[MoNum = _[MoNum] - 1]}?[Days]? ?? 0) & ") / 100"), " else ") & " else 0")),
    #"Added Custom5" = Table.AddColumn(#"Added Custom4", "Month", each Number.RoundDown([MonthDay])),
    #"Added Custom6" = Table.AddColumn(#"Added Custom5", "Day", each Number.Mod(Number.RoundDown([MonthDay] *  100), 100)),
    #"Added Custom7" = Table.AddColumn(#"Added Custom6", "Date", each Text.From([Century]) & (if [Year] <  10 then "0" else "") & Text.From([Year]) & "/" & Text.From([Month]) & "/" & Text.From([Day]), type text)
in
    #"Added Custom7"

 

 

 

Native generated query:

 

 

select [_].[AddressID] as [AddressID],
    [_].[DateEntry] as [DateEntry],
    [_].[Century] as [Century],
    [_].[Year] as [Year],
    [_].[LeapYearOffset] as [LeapYearOffset],
    [_].[MonthDayPart] as [MonthDayPart],
    [_].[MonthDay] as [MonthDay],
    [_].[Month] as [Month],
    [_].[Day] as [Day],
    (((((convert(nvarchar(max), [_].[Century]) + (case
        when [_].[Year] < 10
        then '0'
        else ''
    end)) + convert(nvarchar(max), [_].[Year])) + '/') + convert(nvarchar(max), [_].[Month])) + '/') + convert(nvarchar(max), [_].[Day]) as [Date]
from 
(
    select [_].[AddressID] as [AddressID],
        [_].[DateEntry] as [DateEntry],
        [_].[Century] as [Century],
        [_].[Year] as [Year],
        [_].[MonthDayPart] as [MonthDayPart],
        [_].[LeapYearOffset] as [LeapYearOffset],
        [_].[MonthDay] as [MonthDay],
        floor([_].[MonthDay]) as [Month],
        floor([_].[MonthDay] * 100) % 100 as [Day]
    from 
    (
        select [_].[AddressID] as [AddressID],
            [_].[DateEntry] as [DateEntry],
            [_].[Century] as [Century],
            [_].[Year] as [Year],
            [_].[MonthDayPart] as [MonthDayPart],
            [_].[LeapYearOffset] as [LeapYearOffset],
            case
                when [_].[MonthDayPart] <= 31
                then 1 + convert(decimal(38,6), [_].[MonthDayPart] - 0) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 59 + [_].[LeapYearOffset]
                then 2 + convert(decimal(38,6), [_].[MonthDayPart] - 31) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 90 + [_].[LeapYearOffset]
                then 3 + convert(decimal(38,6), [_].[MonthDayPart] - 59) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 120 + [_].[LeapYearOffset]
                then 4 + convert(decimal(38,6), [_].[MonthDayPart] - 90) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 151 + [_].[LeapYearOffset]
                then 5 + convert(decimal(38,6), [_].[MonthDayPart] - 120) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 181 + [_].[LeapYearOffset]
                then 6 + convert(decimal(38,6), [_].[MonthDayPart] - 151) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 212 + [_].[LeapYearOffset]
                then 7 + convert(decimal(38,6), [_].[MonthDayPart] - 181) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 243 + [_].[LeapYearOffset]
                then 8 + convert(decimal(38,6), [_].[MonthDayPart] - 212) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 273 + [_].[LeapYearOffset]
                then 9 + convert(decimal(38,6), [_].[MonthDayPart] - 243) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 304 + [_].[LeapYearOffset]
                then 10 + convert(decimal(38,6), [_].[MonthDayPart] - 273) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 334 + [_].[LeapYearOffset]
                then 11 + convert(decimal(38,6), [_].[MonthDayPart] - 304) / convert(decimal(38,6), 100)
                when [_].[MonthDayPart] <= 365 + [_].[LeapYearOffset]
                then 12 + convert(decimal(38,6), [_].[MonthDayPart] - 334) / convert(decimal(38,6), 100)
                else 0
            end as [MonthDay]
        from 
        (
            select [_].[AddressID] as [AddressID],
                [_].[DateEntry] as [DateEntry],
                [_].[Century] as [Century],
                [_].[Year] as [Year],
                [_].[MonthDayPart] as [MonthDayPart],
                case
                    when ([_].[Year] % 4 = 0 and [_].[Year] % 4 is not null) and ([_].[Century] % 4 = 0 and [_].[Century] % 4 is not null or ([_].[Year] <> 0 or [_].[Year] is null))
                    then 1
                    else 0
                end as [LeapYearOffset]
            from 
            (
                select [_].[AddressID] as [AddressID],
                    [_].[DateEntry] as [DateEntry],
                    floor(convert(decimal(38,6), [_].[DateEntry]) / convert(decimal(38,6), 100000)) + 19 as [Century],
                    floor(convert(decimal(38,6), [_].[DateEntry] % 100000) / convert(decimal(38,6), 1000)) as [Year],
                    [_].[DateEntry] % 1000 as [MonthDayPart]
                from 
                   <<My Table Source>>
            ) as [_]
        ) as [_]
    ) as [_]
) as [_]

 

 

Thank you so much! It works almost perfectly fine. But somehow I get the result "32" as day for 

120091

 Can you help @artemus ?

aj1973
Community Champion
Community Champion

Hi @joshua1990 

Some Query folding are not supported by ODBC Driver connection. I recommend that you apply those steps in your model instead.

 

Regards
Amine Jerbi

If I answered your question, please mark this thread as accepted
and you can follow me on
My Website, LinkedIn and Facebook

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors