Ususally when you hit refresh, Power Query will perform all those tasks (which includes merge and append) which has been defined in the Edit Query mode. So, your entire process should be automated will work with the refresh button.
Otherwise, you may share the sample files and I would like to recreate it for the demonstration purpose.
If your question is whether Remove Duplicate will make changes in the source data then the answer is it will hide the rows. "Remove Duplicates" will keep the first row of the duplicates row and removes/hides the rest before loading to Power BI desktop.
Hope it answers your query.
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
As per my understanding, Power Query will load the data first and the perform then desired actions. You may improve the efficiency of the entire process by restricting what all tables/queries be refreshed or to be loaded to Power BI data model.
Because I don't understand how I could apply the advice given on this article I will stick with the 2 solutions we discussed so far. So the 2 solutions for my problem are:
1) This article that explains how to override data
2) Hiding duplicate rows
The 2nd option is not memory effective.
If I choose to go with the 1st option, how could I create the override table every time?
For example, let's say original data table has data for dates 13th-16th of June.
I hit refresh button and new data comes in, for the dates 14th-17th of June. Now I need to replace the already existing 14th-16th of June on the original data table with the new ones, and then append the data for 17th on the same table.
To do this, as far as the article says, I will have to create an "override" table, with the new data (14th-17th of June).
But the next time I hit the "refresh" button another bunch of new data will come in, for the dates 15th-18th. What happens now with the override data table? I will need to replace all of the dates on the override table (14th-17th) with the new ones (15th-18th). Am I correct? If yes, how could I do that?
Thank you and sorry for the lot of questions but I am new to power BI