cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
Helper I
Helper I

Combining new data into existing data daily

Hi all,

 

I hope someone is able to offer me some advice with a problem i've been trying to figure out for a while now.

 

So I have 2 data sets with the same columns, one is static (data set 1) and the other refreshes daily (data set 2). Sometimes data set 2 will contain duplicates of what is in data set 1, but sometimes there is new data that I then need to be added to data set 1 (the duplicate data can be discarded).

 

So then 2 things:

 

1 - How do I set it up so that data set 2 combines with data set 1 on a daily basis removing the duplicates and adding the new data?

 

2 - How do I date stamp the new data? I want to include within data set 1 the date that the new data from data set 2 is added to that report.

 

Any help will be appreciated.

 

Regards,

 

Emily.

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Community Support
Community Support

Re: Combining new data into existing data daily

HI @EmilyM2019,

#1, You can add a custom step in the dynamic query table with Table.Combine and Table.Distinct functions to combine and remove duplicate records:

Custom= Table.Distinct(Table.Combine{Query1,#"Previous Step"}))

#2, Nope, query tables are the processing review of current query steps. It wasn't able to save/log the specific operation timestamps in the current query table fields. I'd like to suggest you remark the operation timestamp on your Datasource side.
Regards,
Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

1 REPLY 1
Highlighted
Community Support
Community Support

Re: Combining new data into existing data daily

HI @EmilyM2019,

#1, You can add a custom step in the dynamic query table with Table.Combine and Table.Distinct functions to combine and remove duplicate records:

Custom= Table.Distinct(Table.Combine{Query1,#"Previous Step"}))

#2, Nope, query tables are the processing review of current query steps. It wasn't able to save/log the specific operation timestamps in the current query table fields. I'd like to suggest you remark the operation timestamp on your Datasource side.
Regards,
Xiaoxin Sheng

Community Support Team _ Xiaoxin
If this post helps, please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

Helpful resources

Announcements
Community Blog

Community Blog

Visit our Community Blog for articles, guides, and information created by fellow community members.

Using the Community

Using the Community

Need help with the Power BI Community? Our 'Using the Community' support articles are a great place to start.

Community Summit North America

Community Summit North America

Innovate, Collaborate, Grow. The top training and networking event across the globe for Microsoft Business Applications

Power Platform 2020 release wave 2 plan

Power Platform 2020 release wave 2 plan

Features releasing from October 2020 through March 2021

Top Solution Authors