cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
jankura
Helper I
Helper I

How to model queries in Power BI for daily append of new data instead of overwriting

Hi,

I'm trying to build a simple report based upon data published on a website. Here is what I want to achieve

  1. This website publishes data for COVID cases in the country.
  2. The number are just the current numbers, without any time-series.
  3. I want to fetch these numbers from this website daily and build a report on top of it (with time series kind of analysis).
  4. So I fetch these numbers (Get Data > Web > URL) and get this into a query
  5. I then add a custom column with a timestmap (M's DateTime.LocalNow() fucntion) and get this data with the required timestamp.
  6. Now I want to refresh this query daily, so that I get daily results in this query.
  7. As expected, PBI simply overwritse the existing rows with new data, with the latest timestamp (my custom column).

I tried few things like:-

  1. Creating a new query and appending data to it, it doesn't seem to work, existing data gets over-written (maybe the way I have created the new query).
  2.  Explored incremental refresh, it doesn't seem to fit my use case.
  3. Tried looking at other similar posts, none seem to help me resolve this.

Questions:-

  1. Is there a simple workaround to circumvent this (point#7) and have PBI append new data instead of overwriting existing data.
  2. Am i correct on point#2 above (incremental refresh)? 

 

Appreciate any pointers. Thanks in advance!

3 REPLIES 3
FrankAT
Super User II
Super User II

Hi @jankura ,

which URL do you use to get the data?

 

Regards FrankAT

Greg_Deckler
Super User IV
Super User IV

Couple options. 

 

One, create a new query. Disable refresh on the original query. Create an Append query as New that appends the new query to the original query.

 

Or, download the files to a folder every day. Use a Folder query.


---------------------------------------

@ me in replies or I'll lose your thread!!!

I have a NEW book! 
DAX Cookbook from Packt
Over 120 DAX Recipes!




Did I answer your question? Mark my post as a solution!

Proud to be a Super User!




Option#1: Based upon what I could try, I could see my append query's data was also got overwritten during refresh, which I didn't want. Is it really an option? May be I didn't implement it the way it shoud have.

 

Option#2: I did some further lookup and found that this is probably the right way to do this, have my own little staging and transformation layer (add time series attribute) and then have PBI connect to this layer.

 

Thanks. 

Helpful resources

Announcements
PBI_User Group Leader_768x460.jpg

Manage your user group events

Check out the News & Announcements to learn more.

MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

Get Ready for Power BI Dev Camp

Microsoft named a Leader in The Forrester Wave

Microsoft received the highest score of any vendor in both the strategy and current offering categories.

Top Solution Authors