Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
arvest
New Member

speed up find closest earlier date in another table

I know this question has been asked in at least two other posts, but I believe my case is slightly different, also because it has to do with performance. I have two tables, one of sensor measurements [SensorData] the other of GPS data [Position]. They each have timestamps, and I am trying to find the GPS location that corresponds with a sensor measurement, based on the timestamp. This means I want to find the timestamp of a GPS-record that is closest (but earlier) in time to the timestamp of the sensor measurement. I would like to add this as a new column to the [SensorData] table, so I can add a relation to the two tables based on this column.

 

Table [Position]:

[Id]  [Timestamp] [Lon] [Lat]

 

Table [SensorData]:

[ID] [SensorID] [Timestamp] [Data]

 

So far I have added a VLOOKUP custom function from the example in this blog post, but it is very slow (i'm handling several hunderds of thousands of records). I guess it is because for each record in [SensorData] the function gets called, which sorts [Position], and compares all elements in [Position] and eventually only takes the first element. I know this can be sped up considerably, but I lack the Power Query skills to actually do it.

 

My query for the [SensorData] table is as follows:

 

 

let
    Source = Sql.Databases("some_link.net"),
    #"SQL-01" = Source{[Name="SQL-01"]}[Data],
    dbo_mydb = #"SQL-01"{[Schema="dbo",Item="mydb"]}[Data],
    #"Filtered Rows" = Table.SelectRows(dbo_mydb, each ([SensorId] = 3)),
    #"Filtered Rows1" = Table.SelectRows(#"Filtered Rows", each not Text.Contains([Data], ",")),
    #"Changed Type with Locale" = Table.TransformColumnTypes(#"Filtered Rows1", {{"Data", type number}}, "en-US"),
    #"Filtered Rows2" = Table.SelectRows(#"Changed Type with Locale", each [Timestamp] > #datetime(2017, 7, 1, 0, 0, 0)),
    #"Invoked Custom Function" = Table.AddColumn(#"Filtered Rows2", "pqVLOOKUP", each pqVLOOKUP([Timestamp], Position, 2, null)),
in
    #"Invoked Custom Function"

 

Any suggestions / improvements?

 

 

1 REPLY 1
Greg_Deckler
Super User
Super User

Does this have to be done in Power Query? I am just wondering if it might be faster in DAX as a calculated column. I would think that some combination of calculating the MAX of Timestamp with the right filters would get you there.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
The Definitive Guide to Power Query (M)

DAX is easy, CALCULATE makes DAX hard...

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.