Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
AliceW
Impactful Individual
Impactful Individual

Dataset based on dataflows failed to refresh - claims duplicates, there are none

Hi team,

I've recently started using dataflows and love the concept! However, I have a problem and I could use your help.

I have two dataflows: Account and Opportunity. Each has one table.

In the dataset, I have linked them via a one-to-many relationship.

I have scheduled the refresh of both dataflows and dataset.

It worked fine for about 6 refreshes, when I got this error:

Column 'Account ID' in Table 'Account' contains a duplicate value '<pii>215584</pii>' and this is not allowed for columns on the one side of a many-to-one relationship or for columns that are used as the primary key of a table. Table: Account.

Which I know it does not, as I included a step, in Power Query, in the Account dataflow, to remove duplicates for this specific column.

What do I do, please?

Thank you in advance,

Alice

5 REPLIES 5
v-diye-msft
Community Support
Community Support

Hi @AliceW 

 

Kindly refer to the same solved thread for some hints:

https://community.powerbi.com/t5/Desktop/Refresh-error-quot-contains-duplicate-value-and-this-is-not... 

 

If not fit your issue, please kindly raise it to the support team for better advice:

https://powerbi.microsoft.com/en-us/support/ 

Support Ticket.gif

Community Support Team _ Dina Ye
If this post helps, then please consider Accept it as the solution to help the other members find it more
quickly.
AliceW
Impactful Individual
Impactful Individual

unfortunately, the suggestion marked in there as solution doesn't work for me. The relationship between tables is 'many-to-one', and the issue is with the 'one' side.

AliceW
Impactful Individual
Impactful Individual

Update: I've also removed duplicates in the dataset itself and that did the trick. However, I shouldn't have to, should I? I'm confused.

Hi,

 

If you removed duplicates in PQ inside the pbix file then that explains the original error you got and why you now say its working.

 

You are correct in saying you shoudln't have to do that...if the PQ step in the dataflow was working correctly.

 

You need to investigate that dataflow PQ code....

 

Any chance it was just a timing issue, ie the dataflow hadn't finished refreshing when you did your first test ?

AliceW
Impactful Individual
Impactful Individual

That dataflow code was so basic - just connect to the table in the database, remove some columns, change some names, and... remove duplicates.

And the dataset refresh is scheduled half an hour after the dataflow one, and a refresh lasts about 7 minutes.

Super weird...

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors