Showing results for 
Search instead for 
Did you mean: 
Colinu Regular Visitor
Regular Visitor

Auto Append and Delete Duplicate Data

HI All If you can assist with this be very helpful.


I have data in a few tables with the common link in a column named 'Submmision ID'. In this column there are duplicates and this caused issue on creating the needed relationships. to get around this I used the method of Appending the the column into a new table and then removing the duplicates. this left me with a table with Submission ID that had no duplicates that is could link all my other column through. 


Seen in the clip:


The one issue i am left with is that when is on Refreshing the data to my SQL data source to collect the new data, new duplicates are added. i then need to manually go though the process in the clip all over again. 


Is it possible to automate the process above so that any new data that is duplicated is appended and then duplicates remove? 

rajendran Super Contributor
Super Contributor

Re: Auto Append and Delete Duplicate Data

Hi @Colinu


The applied steps in M , will be performed in each refresh and as per that new duplicates should be removed and only unique duplicates should be added to the new table and this should happen automatically . Request you to check your design step by step, may be with example data to undersatnd where its going wrong.


some more points :


* If any one of your table has master list which contains all values, then you dont need to do the append. directly take the distinct vales from that column adn use that as briding table.


* You can do this ( the above step) in more dynamic way in DAX. Create new table in Modeeling-> New table using DAX formula DISTINCT of that master duplicated column and use that in relationship.




Community Support Team
Community Support Team

Re: Auto Append and Delete Duplicate Data

Hi Colinu,


In your senario, I would recommend you to use increment refresh:



Jimmy Tao

Helpful resources

Can You Solve These Challenge

Challenge: Can You Solve These?

Find out how to participate in the first Power BI 'Can You Solve These?' challenge.

Community News & Announcements

Community News & Announcements

Get your latest community news and announcements.

Virtual Launch Event

Microsoft Business Applications October Virtual Launch Event

Join us for an in-depth look at the new innovations across Dynamics 365 and the Microsoft Power Platform.

Community Kudopalooza

Win Power BI Swag with Community Kudopalooza!

Each week, complete activities and be qualified in the drawing for cool Power BI Swag.

Users Online
Currently online: 201 members 2,149 guests
Please welcome our newest community members: