Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!
Hi all,
I hope someone is able to offer me some advice with a problem i've been trying to figure out for a while now.
So I have 2 data sets with the same columns, one is static (data set 1) and the other refreshes daily (data set 2). Sometimes data set 2 will contain duplicates of what is in data set 1, but sometimes there is new data that I then need to be added to data set 1 (the duplicate data can be discarded).
So then 2 things:
1 - How do I set it up so that data set 2 combines with data set 1 on a daily basis removing the duplicates and adding the new data?
2 - How do I date stamp the new data? I want to include within data set 1 the date that the new data from data set 2 is added to that report.
Any help will be appreciated.
Regards,
Emily.
Solved! Go to Solution.
HI @EmilyM2019,
#1, You can add a custom step in the dynamic query table with Table.Combine and Table.Distinct functions to combine and remove duplicate records:
Custom= Table.Distinct(Table.Combine{Query1,#"Previous Step"}))
#2, Nope, query tables are the processing review of current query steps. It wasn't able to save/log the specific operation timestamps in the current query table fields. I'd like to suggest you remark the operation timestamp on your Datasource side.
Regards,
Xiaoxin Sheng
HI @EmilyM2019,
#1, You can add a custom step in the dynamic query table with Table.Combine and Table.Distinct functions to combine and remove duplicate records:
Custom= Table.Distinct(Table.Combine{Query1,#"Previous Step"}))
#2, Nope, query tables are the processing review of current query steps. It wasn't able to save/log the specific operation timestamps in the current query table fields. I'd like to suggest you remark the operation timestamp on your Datasource side.
Regards,
Xiaoxin Sheng
User | Count |
---|---|
15 | |
2 | |
1 | |
1 | |
1 |
User | Count |
---|---|
18 | |
11 | |
5 | |
4 | |
3 |