Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
Jpmercier00
Frequent Visitor

Append data from dynamic daily refreshed data

Good afternoon.

 

I am trying to append data from a csv file that is refreshed daily. What I am trying to have is a accumulalive table that will keep all data and only add (append) the new data.

 

Example source data

PRODUCT_TYPE / ID / PRICE

TV / 7 / 1226

.

.

 

As you can see, there is no date, so I am adding a column with the current date to have:

PRODUCT_TYPE / ID / PRICE / DATE

TV / 7 / 1226 / 10/16/2020

.

.

 

What I am trying to acheive is something like this:

PRODUCT_TYPE / ID / PRICE / DATE

WASHER / 8 / 560 / 10/15/2020

GAME CONSOLE / 7 / 10/15/2020

TV / 9 / 840 / 10/14/2020

 

And append today´s records without loosing the previous data.

 

TV / 7 / 1226 / 10/16/2020

.

.

 

Did I explain myself?

Any guidance would be appreciate!

 

Regards

Jean Phillip Mercier

4 REPLIES 4
v-yingjl
Community Support
Community Support

Hi @Jpmercier00 ,

Based on your description, wants to know couple of problems:

  1. Since there is no date column in the source data, how could you add it? In the csv file or in powe query?
  2. Are there some other previous fields in the file or in another file? How could their dates field create?

In addtion, the .csv file will refresh when click the refresh button in power query, not certain what does append a .csv file represent.

 

Best Regards,
Yingjie Li

 

  1. Since there is no date column in the source data, how could you add it? In the csv file or in powe query?

Data is refrehed every day in a we site at 12:00, so the date is the current date, which I add as a new column.

 

2. Are there some other previous fields in the file or in another file? How could their dates field create?

No, there are no other files. Date can be created as a new column with the Today´s date.

 

Each time you refresh you will dowload the complete .csv that is on the web at that moment that corresponds to the current day data.

 

Thank you!

 

mahoneypat
Employee
Employee

Power BI doesn't store data like that.  In your case, I would set up a Flow to name and store your daily csv files (choose a name that incorporates the date so you can use it downstream), and have Power BI combine all the CSVs.  If the csv is emailed to you, you can trigger the flow based on that.  If manual, you could rename it (with date info) and store in a SharePoint or OneDrive location for easy refresh from Power BI.

 

Regards,

Pat

 





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


That is how I do It but it is no reliable a practical a the data is about 32,000 records daily and if you miss to do the store and renaming process I loose the entire day. Data is refreshed daily at 12:00.

 

It looks I will help for the SQL guys to to develop a query.

 

Thank you so much!

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors