cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
jankura
Helper I
Helper I

How to model queries in Power BI for daily append of new data instead of overwriting

Hi,

I'm trying to build a simple report based upon data published on a website. Here is what I want to achieve

  1. This website publishes data for COVID cases in the country.
  2. The number are just the current numbers, without any time-series.
  3. I want to fetch these numbers from this website daily and build a report on top of it (with time series kind of analysis).
  4. So I fetch these numbers (Get Data > Web > URL) and get this into a query
  5. I then add a custom column with a timestmap (M's DateTime.LocalNow() fucntion) and get this data with the required timestamp.
  6. Now I want to refresh this query daily, so that I get daily results in this query.
  7. As expected, PBI simply overwritse the existing rows with new data, with the latest timestamp (my custom column).

I tried few things like:-

  1. Creating a new query and appending data to it, it doesn't seem to work, existing data gets over-written (maybe the way I have created the new query).
  2.  Explored incremental refresh, it doesn't seem to fit my use case.
  3. Tried looking at other similar posts, none seem to help me resolve this.

Questions:-

  1. Is there a simple workaround to circumvent this (point#7) and have PBI append new data instead of overwriting existing data.
  2. Am i correct on point#2 above (incremental refresh)? 

 

Appreciate any pointers. Thanks in advance!

3 REPLIES 3
FrankAT
Community Champion
Community Champion

Hi @jankura ,

which URL do you use to get the data?

 

Regards FrankAT

Greg_Deckler
Super User
Super User

Couple options. 

 

One, create a new query. Disable refresh on the original query. Create an Append query as New that appends the new query to the original query.

 

Or, download the files to a folder every day. Use a Folder query.


@ me in replies or I'll lose your thread!!!
Check out my External Tool for Power BI Desktop!
Microsoft Hates Greg's Quick Measures
Check out my latest book!

Option#1: Based upon what I could try, I could see my append query's data was also got overwritten during refresh, which I didn't want. Is it really an option? May be I didn't implement it the way it shoud have.

 

Option#2: I did some further lookup and found that this is probably the right way to do this, have my own little staging and transformation layer (add time series attribute) and then have PBI connect to this layer.

 

Thanks. 

Helpful resources

Announcements
UG GA Amplification 768x460.png

Launching new user group features

Learn how to create your own user groups today!

Power BI October Update 2021.jpg

Power BI Release

Click here to read more about the October 2021 Release!

Community Connections 768x460.jpg

Community & How To Videos

Check out the new Power Platform Community Connections gallery!

Teds Dev Camp Oct. 2021 768x460.jpg

Power BI Dev Camp - October 28th, 2021

Mark your calendars and join us for our next Power BI Dev Camp!