Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
Hello together!
I am accessing a transactional table (oracle / ODBC Connection type) that has several million rows.
I would like to update this table based on incremental refresh in Power BI Desktop.
This table has a column "Date Entry" which is a number in the format "CYYDDD".
For example, this means that the date is 01/01/2019 = 119001.
To put this number in a date format like "DD.MM.YYYY" I have already found a function.
But this function will stop the query folding. Anyway, after this step, I can't see a Native Query anymore.
Now I have the question if I can still somehow apply the incremental refresh?
What recommendations do you guys have?
Here is one way to do it:
Replace the Source line with your table. It expects a [DateEntry] column with the date string
let
Source = ...,
#"Filtered Rows" = Table.SelectRows(Source, each [AddressID] = 1),
#"Added Custom" = Table.AddColumn(#"Filtered Rows", "DateEntry", each 107263, type number),
Custom2 = Table.AddColumn(#"Added Custom", "Century",each Number.RoundDown([DateEntry] / 100000) + 19, Int32.Type),
#"Added Custom1" = Table.AddColumn(Custom2, "Year", each Number.RoundDown(Number.Mod([DateEntry], 100000) / 1000), Int32.Type),
#"Added Custom2" = Table.AddColumn(#"Added Custom1", "LeapYearOffset", each if Number.Mod([Year], 4) = 0 and (Number.Mod([Century], 4) = 0
or [Year] <> 0) then 1 else 0, Int32.Type),
#"Added Custom3" = Table.AddColumn(#"Added Custom2", "MonthDayPart", each Number.Mod([DateEntry], 1000)),
#"Added Custom4" = Table.AddColumn(#"Added Custom3", "MonthDay", let monthTable = #table(type table [MoNum = Int32.Type, Days = Int32.Type], List.Zip({{1 .. 12}, List.Accumulate({31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, {}, (current, next) => current & {List.Last(current, 0) + next})})) in Expression.Evaluate("each " & Text.Combine(Table.TransformRows(monthTable, each "if [MonthDayPart] <= " & Text.From([Days]) & (if [MoNum] = 1 then "" else " + [LeapYearOffset]") & " then " & Text.From([MoNum]) & " + ([MonthDayPart] - " & Text.From(monthTable{[MoNum = _[MoNum] - 1]}?[Days]? ?? 0) & ") / 100"), " else ") & " else 0")),
#"Added Custom5" = Table.AddColumn(#"Added Custom4", "Month", each Number.RoundDown([MonthDay])),
#"Added Custom6" = Table.AddColumn(#"Added Custom5", "Day", each Number.Mod(Number.RoundDown([MonthDay] * 100), 100)),
#"Added Custom7" = Table.AddColumn(#"Added Custom6", "Date", each Text.From([Century]) & (if [Year] < 10 then "0" else "") & Text.From([Year]) & "/" & Text.From([Month]) & "/" & Text.From([Day]), type text)
in
#"Added Custom7"
Native generated query:
select [_].[AddressID] as [AddressID],
[_].[DateEntry] as [DateEntry],
[_].[Century] as [Century],
[_].[Year] as [Year],
[_].[LeapYearOffset] as [LeapYearOffset],
[_].[MonthDayPart] as [MonthDayPart],
[_].[MonthDay] as [MonthDay],
[_].[Month] as [Month],
[_].[Day] as [Day],
(((((convert(nvarchar(max), [_].[Century]) + (case
when [_].[Year] < 10
then '0'
else ''
end)) + convert(nvarchar(max), [_].[Year])) + '/') + convert(nvarchar(max), [_].[Month])) + '/') + convert(nvarchar(max), [_].[Day]) as [Date]
from
(
select [_].[AddressID] as [AddressID],
[_].[DateEntry] as [DateEntry],
[_].[Century] as [Century],
[_].[Year] as [Year],
[_].[MonthDayPart] as [MonthDayPart],
[_].[LeapYearOffset] as [LeapYearOffset],
[_].[MonthDay] as [MonthDay],
floor([_].[MonthDay]) as [Month],
floor([_].[MonthDay] * 100) % 100 as [Day]
from
(
select [_].[AddressID] as [AddressID],
[_].[DateEntry] as [DateEntry],
[_].[Century] as [Century],
[_].[Year] as [Year],
[_].[MonthDayPart] as [MonthDayPart],
[_].[LeapYearOffset] as [LeapYearOffset],
case
when [_].[MonthDayPart] <= 31
then 1 + convert(decimal(38,6), [_].[MonthDayPart] - 0) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 59 + [_].[LeapYearOffset]
then 2 + convert(decimal(38,6), [_].[MonthDayPart] - 31) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 90 + [_].[LeapYearOffset]
then 3 + convert(decimal(38,6), [_].[MonthDayPart] - 59) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 120 + [_].[LeapYearOffset]
then 4 + convert(decimal(38,6), [_].[MonthDayPart] - 90) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 151 + [_].[LeapYearOffset]
then 5 + convert(decimal(38,6), [_].[MonthDayPart] - 120) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 181 + [_].[LeapYearOffset]
then 6 + convert(decimal(38,6), [_].[MonthDayPart] - 151) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 212 + [_].[LeapYearOffset]
then 7 + convert(decimal(38,6), [_].[MonthDayPart] - 181) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 243 + [_].[LeapYearOffset]
then 8 + convert(decimal(38,6), [_].[MonthDayPart] - 212) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 273 + [_].[LeapYearOffset]
then 9 + convert(decimal(38,6), [_].[MonthDayPart] - 243) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 304 + [_].[LeapYearOffset]
then 10 + convert(decimal(38,6), [_].[MonthDayPart] - 273) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 334 + [_].[LeapYearOffset]
then 11 + convert(decimal(38,6), [_].[MonthDayPart] - 304) / convert(decimal(38,6), 100)
when [_].[MonthDayPart] <= 365 + [_].[LeapYearOffset]
then 12 + convert(decimal(38,6), [_].[MonthDayPart] - 334) / convert(decimal(38,6), 100)
else 0
end as [MonthDay]
from
(
select [_].[AddressID] as [AddressID],
[_].[DateEntry] as [DateEntry],
[_].[Century] as [Century],
[_].[Year] as [Year],
[_].[MonthDayPart] as [MonthDayPart],
case
when ([_].[Year] % 4 = 0 and [_].[Year] % 4 is not null) and ([_].[Century] % 4 = 0 and [_].[Century] % 4 is not null or ([_].[Year] <> 0 or [_].[Year] is null))
then 1
else 0
end as [LeapYearOffset]
from
(
select [_].[AddressID] as [AddressID],
[_].[DateEntry] as [DateEntry],
floor(convert(decimal(38,6), [_].[DateEntry]) / convert(decimal(38,6), 100000)) + 19 as [Century],
floor(convert(decimal(38,6), [_].[DateEntry] % 100000) / convert(decimal(38,6), 1000)) as [Year],
[_].[DateEntry] % 1000 as [MonthDayPart]
from
<<My Table Source>>
) as [_]
) as [_]
) as [_]
) as [_]
Thank you so much! It works almost perfectly fine. But somehow I get the result "32" as day for
120091 |
Can you help @artemus ?
Hi @joshua1990
Some Query folding are not supported by ODBC Driver connection. I recommend that you apply those steps in your model instead.
Regards
Amine Jerbi
If I answered your question, please mark this thread as accepted
and you can follow me on
My Website, LinkedIn and Facebook
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
100 | |
54 | |
21 | |
12 | |
11 |