Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
I have a database with dates that is refreshed every X hours. I want once I hit the refresh button to overwrite the rows that have the same Date with the new data.
Currently, when I hit the the refresh button I get duplicates of the same date. I don't want that.
How could I fix this?
Thank you
Solved! Go to Solution.
Hi,
I feel that in this scenario removing duplicates is a better option to go with as compared to merge table, considering the fact the there is a single souce table.
Rgds,
Vivek
PS: It is good that you are asking questions. That is how we all learn 🙂
Hi,
You may use tips from the article having similar issue
Rgds,
Vivek
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Thank you for the help @vivran22. This is a nice article.
However, I want this to be done automatically, once I hit the "refresh" button.
If I have the 2 queries the article says (the original data and the override data), how can I perform the merge and append functions automatically?
Thank you.
Hi,
Ususally when you hit refresh, Power Query will perform all those tasks (which includes merge and append) which has been defined in the Edit Query mode. So, your entire process should be automated will work with the refresh button.
Otherwise, you may share the sample files and I would like to recreate it for the demonstration purpose.
Rgds,
Vivek
Sorry for the late reply. I experimented a little and I realised that what I ask for can be done with the "Remove Duplicates" option. Seems more simple solution than creating 2 separate tables.
However, does that option deletes the rows or just hides them? What do you think?
Thank you
If your question is whether Remove Duplicate will make changes in the source data then the answer is it will hide the rows. "Remove Duplicates" will keep the first row of the duplicates row and removes/hides the rest before loading to Power BI desktop.
Hope it answers your query.
Rgds,
Vivek
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
As per my understanding, Power Query will load the data first and the perform then desired actions. You may improve the efficiency of the entire process by restricting what all tables/queries be refreshed or to be loaded to Power BI data model.
This article may guide you to some extent
Rgds,
Vivek
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Because I don't understand how I could apply the advice given on this article I will stick with the 2 solutions we discussed so far. So the 2 solutions for my problem are:
1) This article that explains how to override data
2) Hiding duplicate rows
The 2nd option is not memory effective.
If I choose to go with the 1st option, how could I create the override table every time?
For example, let's say original data table has data for dates 13th-16th of June.
I hit refresh button and new data comes in, for the dates 14th-17th of June. Now I need to replace the already existing 14th-16th of June on the original data table with the new ones, and then append the data for 17th on the same table.
To do this, as far as the article says, I will have to create an "override" table, with the new data (14th-17th of June).
But the next time I hit the "refresh" button another bunch of new data will come in, for the dates 15th-18th. What happens now with the override data table? I will need to replace all of the dates on the override table (14th-17th) with the new ones (15th-18th). Am I correct? If yes, how could I do that?
Thank you and sorry for the lot of questions but I am new to power BI
Hi,
I feel that in this scenario removing duplicates is a better option to go with as compared to merge table, considering the fact the there is a single souce table.
Rgds,
Vivek
PS: It is good that you are asking questions. That is how we all learn 🙂
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
112 | |
100 | |
76 | |
74 | |
49 |
User | Count |
---|---|
146 | |
108 | |
106 | |
90 | |
62 |