cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
nynni Frequent Visitor
Frequent Visitor

Website scraping advice

I'm trying to scrape a website with a pretty simple HTML table, but it uses Javascript for pagination and I can only get the first 25 results when using the web connector. I've tried using 

 [WaitFor = [Timeout = #duration(0,0,0,0)]])

to see if Power BI could pick up the table before the javascript loads -- I'm not sure if that's how it works but it hasn't given me any results yet.

 

Is there anything I can do? This is the website and data in question: 

http://www.onequestionshootout.xyz/episodes/series_all.htm

1 ACCEPTED SOLUTION

Accepted Solutions
Community Support Team
Community Support Team

Re: Website scraping advice

@nynni ,

 

I would suggest you to use python script in power bi to scrapy the website. About how to configure python environment and implement python script in power bi desktop, I would suggest you to refer to doc below:

https://docs.microsoft.com/en-us/power-bi/desktop-python-scripts

 

Community Support Team _ Jimmy Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

6 REPLIES 6
Highlighted
kcantor Super Contributor
Super Contributor

Re: Website scraping advice

@nynni 

Perhaps this resource will help.

https://datachant.com/2017/03/30/web-scraping-power-bi-excel-power-query/



Did I answer your question? Mark my post as a solution!

Proud to be a Datanaut!




nynni Frequent Visitor
Frequent Visitor

Re: Website scraping advice

It started as promising, but unfortunately I can't get any parameters from the url as it doesn't produce any when you navigate through the pages... Tricky!
Community Support Team
Community Support Team

Re: Website scraping advice

@nynni ,

 

I would suggest you to use python script in power bi to scrapy the website. About how to configure python environment and implement python script in power bi desktop, I would suggest you to refer to doc below:

https://docs.microsoft.com/en-us/power-bi/desktop-python-scripts

 

Community Support Team _ Jimmy Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

nynni Frequent Visitor
Frequent Visitor

Re: Website scraping advice

I'm afraid my skills at this point won't allow for python scripting, so in the meanwhile I've downloaded the page as .html and I used the Text/CSV data connector to get the table in plain HTML. The downside of course is I cannot get the latest updates to my report over the internet.

Community Support Team
Community Support Team

Re: Website scraping advice

@nynni ,

 

Power query only support simple web scrapying. If the website needs dynamic scrapying, I'm afraid power query won't work.

 

Community Support Team _ Jimmy Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

PaulDBrown Senior Member
Senior Member

Re: Website scraping advice

@nynni 

Wow! Thank you for sharing the idea of dowloading the page as an HTML! I was having the same problem as you and was completely stuck. With your solution, I have at least succeeded in extracting a "snapshot" of the data as it stands currently, which is better than no data at all...

I would have never thought of downloading the actual page!

 

Thanks!!

Helpful resources

Announcements
October 2019 Community Highlights

October 2019 Community Highlights

October was a busy month in the community. Read the recap article to learn about some of the events and content.

New Badges

Incoming: New and Improved Badges

Exciting news: We've given our badges an overhaul and added brand news ones.

Ask Amir Anything

Exclusive LIVE Community Event No. 2 – Ask Amir Anything

Next in our Triple A series: Ask Amir Netz questions about the latest updates, features and future.

Analytics in Azure virtual event

Analytics in Azure virtual event

Experience a limitless analytics service built to ingest, prep, manage, and serve data for immediate use in Power BI.

Users Online
Currently online: 196 members 2,233 guests
Please welcome our newest community members: