Append data from dynamic daily refreshed data

[Jpmercier00]

Good afternoon.

I am trying to append data from a csv file that is refreshed daily. What I am trying to have is a accumulalive table that will keep all data and only add (append) the new data.

Example source data

PRODUCT_TYPE / ID / PRICE

TV / 7 / 1226

.

.

As you can see, there is no date, so I am adding a column with the current date to have:

PRODUCT_TYPE / ID / PRICE / DATE

TV / 7 / 1226 / 10/16/2020

.

.

What I am trying to acheive is something like this:

PRODUCT_TYPE / ID / PRICE / DATE

WASHER / 8 / 560 / 10/15/2020

GAME CONSOLE / 7 / 10/15/2020

TV / 9 / 840 / 10/14/2020

And append today´s records without loosing the previous data.

TV / 7 / 1226 / 10/16/2020

.

.

Did I explain myself?

Any guidance would be appreciate!

Regards

Jean Phillip Mercier

[v-yingjl]

Hi @Jpmercier00 ,

Based on your description, wants to know couple of problems:

1. Since there is no date column in the source data, how could you add it? In the csv file or in powe query?
2. Are there some other previous fields in the file or in another file? How could their dates field create?

In addtion, the .csv file will refresh when click the refresh button in power query, not certain what does append a .csv file represent.

Best Regards,
Yingjie Li

[Jpmercier00]

1. Since there is no date column in the source data, how could you add it? In the csv file or in powe query?

Data is refrehed every day in a we site at 12:00, so the date is the current date, which I add as a new column.

2. Are there some other previous fields in the file or in another file? How could their dates field create?

No, there are no other files. Date can be created as a new column with the Today´s date.

Each time you refresh you will dowload the complete .csv that is on the web at that moment that corresponds to the current day data.

Thank you!

[mahoneypat]

Power BI doesn't store data like that.  In your case, I would set up a Flow to name and store your daily csv files (choose a name that incorporates the date so you can use it downstream), and have Power BI combine all the CSVs.  If the csv is emailed to you, you can trigger the flow based on that.  If manual, you could rename it (with date info) and store in a SharePoint or OneDrive location for easy refresh from Power BI.

Regards,

Pat

[Jpmercier00]

That is how I do It but it is no reliable a practical a the data is about 32,000 records daily and if you miss to do the store and renaming process I loose the entire day. Data is refreshed daily at 12:00.

It looks I will help for the SQL guys to to develop a query.

Thank you so much!