Hello Power Bi community,
I was trying to extract information using python code from a website that only holds the last 30 days of the records that I need and append on a dataset in PowerBI. My goal is to be able to refresh the process in Power BI, so the new content posted on the website will be appended to my PowerBI dataset.
So, for example, my source file (csv) contains records from 8/15-9/14, which I extracted on 9/14.
And now in PowerBI, I ran a python script that 1. scraped all the information on the website today (8/29-9/28) and 2. appended the scraped records to my source file, so now i have a dataset in PowerBI that contains records from 8/15-9/28. I call it "Appended Dataset".
My question is, on a future date like 10/25, the website will only have the records from 9/26-10/25. If I clicked "refresh all" on Query Editor, will the python code be applied on the source csv so I will get a dataset that contains (8/15-9/14, 9/26-10/25) records, or the python code will be applied on the "Appended Dataset" so I will get a dataset that contains (8/15-10/25) records. If it runs on the original dataset that misses the records from 9/15-9/25, what's the best thing I can do to prevent that from happening?
I know it sounds confusing so please let me know if there's anything I can do to clarify my question. Thank you!
