Combining new data into existing data daily

[EmilyM2019]

Hi all,

I hope someone is able to offer me some advice with a problem i've been trying to figure out for a while now.

So I have 2 data sets with the same columns, one is static (data set 1) and the other refreshes daily (data set 2). Sometimes data set 2 will contain duplicates of what is in data set 1, but sometimes there is new data that I then need to be added to data set 1 (the duplicate data can be discarded).

So then 2 things:

1 - How do I set it up so that data set 2 combines with data set 1 on a daily basis removing the duplicates and adding the new data?

2 - How do I date stamp the new data? I want to include within data set 1 the date that the new data from data set 2 is added to that report.

Any help will be appreciated.

Regards,

Emily.

[v-shex-msft · Accepted solution]

HI @EmilyM2019,

#1, You can add a custom step in the dynamic query table with Table.Combine and Table.Distinct functions to combine and remove duplicate records:

Custom= Table.Distinct(Table.Combine{Query1,#"Previous Step"}))

#2, Nope, query tables are the processing review of current query steps. It wasn't able to save/log the specific operation timestamps in the current query table fields. I'd like to suggest you remark the operation timestamp on your Datasource side.
Regards,
Xiaoxin Sheng