Showing results for 
Search instead for 
Did you mean: 
FrankH- Frequent Visitor
Frequent Visitor

Web scraping and storing historical scraped data


I have made a web scraper in Power BI using the same principal as in this video tutorial . I am tracking the distribution of products sold in a certain shop so I have invoked the custom web scraper function in a product number table so that the function is returning the distribution for all products in the table. So the table now has two columns: Product ID (number) and Distribution (number).


However, to keep track of the distribution development, I would like this data to be refreshed weekly and that the historical values would be stored under each refresh date rather than just replacing the data with the latest as it's doing now. Does anybody have any idea on how to do this? 


Any help is really appreciated! 






v-yulgu-msft Super Contributor
Super Contributor

Re: Web scraping and storing historical scraped data

Hi @FrankH-,


Currently, incremental data load that keeps historical data and just added new data records per refresh is not supported in Power Bi desktop. Here is an idea and it mentions that this feature has been released in Premium and it would be added to Power BI Pro in the future.


Besides, please refer to this blog to try the workaround to achieve incremental load. 


Best regards,

Yuliana Gu

Community Support Team _ Yuliana Gu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Back to School Contest

Back to School Contest

Engage and empower students with Power BI!

MBAS Gallery

Watch Sessions On Demand!

Continue your learning in our online communities.

Summit Australia 2019

Summit Australia 2019

Travel to Melbourne and network with thousands of peers!

Top Ideas
Users Online
Currently online: 119 members 1,663 guests
Please welcome our newest community members: