Web scraping and storing historical scraped data

[FrankH-]

Hi,

I have made a web scraper in Power BI using the same principal as in this video tutorial . I am tracking the distribution of products sold in a certain shop so I have invoked the custom web scraper function in a product number table so that the function is returning the distribution for all products in the table. So the table now has two columns: Product ID (number) and Distribution (number).

However, to keep track of the distribution development, I would like this data to be refreshed weekly and that the historical values would be stored under each refresh date rather than just replacing the data with the latest as it's doing now. Does anybody have any idea on how to do this? 

Any help is really appreciated! 

Regards

Frank

[v-yulgu-msft]

Hi @FrankH-,

Currently, incremental data load that keeps historical data and just added new data records per refresh is not supported in Power Bi desktop. Here is an idea and it mentions that this feature has been released in Premium and it would be added to Power BI Pro in the future.

Besides, please refer to this blog to try the workaround to achieve incremental load. 

Best regards,

Yuliana Gu