Hello Everyone!
I'm new to Power BI seeking your advise on a query which I have.
For context Below is the flow which I'm currently following.
1. GET the data from an API which comes in an JSON format.
2. Currently i'm PreProcessing the data via a python script outside of Power BI which creates one main files and 4 different files from the json output which I get from the API and data is saved as csv files on my local drive.
3. Built the Power BI report based on the processed data saved on my local drive.
4. I've published the report from my desktop to the Power BI Service.
Now the query which I have is I need to refresh the dataset everyday which will have new entries and also exsisting enries might have a modification. That being said I cannot refresh the complete dataset I need to append the new entries and update the exsisting entries if there are any modification.
My idea was to write a python script and run it on power bi which has a schedule refresh option which can run everyday for step 1 and 2. But I'm kind of confused as to how do I append the data to the current dataset instead of updating the entire dataset which is huge and time consuming and not ideal. If it was just one "Sheet/Dataset" that wouldnt be a problem because I think i can first create the base dataset and once the base dataset is ready I can write another python script which keeps appending the data to the dataset. But since the data is in JSON format everytime I need to process it to create five different files and update/append on each files seperately from just one main dataset(hope that makes sense).
How would you tackle this? How would I automate the process?
So my question is, is there a better way of doing this without the help of any other apps by just using powerbi service?
Not sure what kind of options does Premium offer but is it possible in Power BI premium? Or is there any other apps which can help me acheive this from the Microsoft catalogue of apps? if yes please advise how would the flow look like?
Thanks in advance.
