Real Time Pulling Data From Public Website (ie weather.gov)

[captainlaw]

Hello Power BI Community,

I've a simple scenario, I'm pulling data from weather.gov (or any public website) into Power BI Desktop, but I would like to make it real-time meaning refresh every 10 minutes.  What's your recommended easy route to go about refreshing public website dataset?

Your help is much appreciated.

-John

[Eric_Zhang]

---

@captainlaw wrote:

Hello Power BI Community,

I've a simple scenario, I'm pulling data from weather.gov (or any public website) into Power BI Desktop, but I would like to make it real-time meaning refresh every 10 minutes.  What's your recommended easy route to go about refreshing public website dataset?

Your help is much appreciated.

-John

---

@captainlaw

For that real-time purpose, the normal "get data"->"other"->"web" doesn't work, as it still requires "schedule refresh" and the refresh is limited to 8 times daily.

One way promising is to use Power BI REST API to push data, see Real-time streaming in Power BI. Alternatively you can save the html data in a database and then connect to the database in DirectQuery mode. For both ways, it requires some coding skill, you need to scrape html page, parse the html and save data in a database for DQ mode or send it via REST API call.

[captainlaw]

Thanks.  If there is sample example of public website being scraped, it would help understand the architecture with Power BI API.

Does the API require some JSON file provided by the website?  If so, do you happen to know a list of websites that provide JSON scripts?  Websites that can provide weather, sports... etc.

Cheers.

[Eric_Zhang]

---

@captainlaw wrote:

Thanks.  If there is sample example of public website being scraped, it would help understand the architecture with Power BI API.

Does the API require some JSON file provided by the website?  If so, do you happen to know a list of websites that provide JSON scripts?  Websites that can provide weather, sports... etc.

Cheers.

---

@captainlaw

You can send data to Power BI following this tutorial Push data into a Power BI dataset. As to how to scrape html to get the sent data, I'd say it is actually not on Power BI aspect. There're various demos on internet, see a simple demo Parsing HTML Table in C#.