Identify Duplicates based on multiple conditions/columns

[v_mark]

I have an entire dataset which has multiple entries that has duplicates. 

So far this is how it looks like. The 2nd row has a duplicate 8 Characters on the top
I need to identify in any event these behaviors occurs as considered duplicates.  TIA

[v-alq-msft · Accepted solution]

Hi, @v_mark 

Based on your description, I created data to reproduce your scenario. The pbix file is attached in the end.

Table:

You may create a custom column with the following m codes.

let 
device=[Device],
status=[Status],
state=[State],
sn=[SN],
tab = Table.SelectRows(#"Renamed Columns",each 
[Device]=device and 
[Status]=status and 
[State]=state and (
Text.EndsWith([SN],sn) or Text.EndsWith(sn,[SN])) ),
re = Table.RowCount(tab)
in 
if re>=2 
then 
"duplicate"
else null

Result:

Best Regards

Allan

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Jimmy801]

Hello @v_mark 

For this kinda request Table.Distinct isn't working because it doesn't accept a custom comparer. However there is a workaround using Table.Group where you can use the 5th parameter to create your own custom comparer but you have to implement it for every column in your table (as well as in the second parameter). Here an example

code that does the comparing of your SN. It checks wheter value a is an ending of value b or the other way round (dont't forget you have to implement all your columns that you need the distincted value as well)

Text.EndsWith(o[C], n[C]) or Text.EndsWith(n[C], o[C])

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUXICYkMj48SkZKVYHYQQmB8LAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [A = _t, B = _t, C = _t]),
    #"Grouped Rows" = Table.Group(Source, {"A", "B", "C"}, {{"AllRows", each _, type table [A=text, B=text, C=text]}}, GroupKind.Global, (o,n)=> if o[A]=n[A] and o[B]=n[B] and (Text.EndsWith(o[C], n[C]) or Text.EndsWith(n[C], o[C])) then 0 else -1)
in
    #"Grouped Rows"

Outcome

Copy paste this code to the advanced editor in a new blank query to see how the solution works.

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

[edhans]

Is it alwaysthe last 8 chars, or is it random lengths, random positions? Also, we cannot work with images. Images are great for expected outcome, but not for supplying data.

How to get good help fast. Help us help you.
How to Get Your Question Answered Quickly - Give us a good and concise explanation
How to provide sample data in the Power BI Forum - Provide data in a table format per the link. Provide expected output using a screenshot of Excel or other image. Do not provide a screenshot of the source data. I cannot paste an image into Power BI tables.