Creating an updating database

[DrewCarousel]

Dear community,

I have a data source which overwrites itself each day with latest data. What i want to do is use PowerBI to extract the data, save it into a database, then the next day, collect the "new" overwritten data but only pick up any rows of data which have changed (there is a last modified field i can pull from). I want to then append this onto the bottom of my ever-growing database.

Previously i worked with Qlikview where i would have saved the first file as a database and then loaded from this first, to see if anything had changed but i cant see how to do this in powerBI - help please!

Best regards,

Drew

[DrewCarousel]

Hi @BA_Pete 

Appreciate your response. I think maybe i need to be more clear on my situation.

I have a blob azure database updating with CSV files each day from our transactional system - each day, the CSV will contain duplicates of the day before's information, but some of the rows of data will have updated info. I have a date modified field so i can use this to tell whether info has updated from the day before, but im not sure how i go about only updating the new rows (and therefore removing the old rows of data) using incremental refresh - all i can see i can do is to load all the information and then filter out the older data. This feels like heavy lifting each day if not much has changed.

Best regards,

Drew

[BA_Pete]

Hi @DrewCarousel ,

The only way I think you can do this with Power BI is by using incremental refresh in the Power BI Service.

You need a Premium Capacity or Premium-per-User licence.

More info:

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview 

Other than this, it needs to be done within your data warehouse.

Pete