cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
DrewCarousel
Frequent Visitor

Creating an updating database

Dear community,

 

I have a data source which overwrites itself each day with latest data. What i want to do is use PowerBI to extract the data, save it into a database, then the next day, collect the "new" overwritten data but only pick up any rows of data which have changed (there is a last modified field i can pull from). I want to then append this onto the bottom of my ever-growing database.

 

Previously i worked with Qlikview where i would have saved the first file as a database and then loaded from this first, to see if anything had changed but i cant see how to do this in powerBI - help please!

 

Best regards,

Drew

2 REPLIES 2
DrewCarousel
Frequent Visitor

Hi @BA_Pete 

 

Appreciate your response. I think maybe i need to be more clear on my situation.

 

I have a blob azure database updating with CSV files each day from our transactional system - each day, the CSV will contain duplicates of the day before's information, but some of the rows of data will have updated info. I have a date modified field so i can use this to tell whether info has updated from the day before, but im not sure how i go about only updating the new rows (and therefore removing the old rows of data) using incremental refresh - all i can see i can do is to load all the information and then filter out the older data. This feels like heavy lifting each day if not much has changed.

 

Best regards,

Drew

BA_Pete
Super User
Super User

Hi @DrewCarousel ,

 

The only way I think you can do this with Power BI is by using incremental refresh in the Power BI Service.

You need a Premium Capacity or Premium-per-User licence.

 

More info:

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview 

 

Other than this, it needs to be done within your data warehouse.

 

Pete

Helpful resources

Announcements
Microsoft Build 768x460.png

Microsoft Build is May 24-26. Have you registered yet?

Come together to explore latest innovations in code and application development—and gain insights from experts from around the world.

charticulator_carousel_with_text (1).png

Charticulator Design Challenge

Put your data visualization and design skills to the test! This exciting challenge is happening now through May 31st!

May UG Leader Call Carousel 768x460.png

What difference can a User Group make for you?

At the monthly call, connect with other leaders and find out how community makes your experience even better.

Power BI Release May 2022 768x460.png

Check it out!

Click here to read more about the May 2022 updates!