Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
I'm trying to scrape a website with a pretty simple HTML table, but it uses Javascript for pagination and I can only get the first 25 results when using the web connector. I've tried using
[WaitFor = [Timeout = #duration(0,0,0,0)]])
to see if Power BI could pick up the table before the javascript loads -- I'm not sure if that's how it works but it hasn't given me any results yet.
Is there anything I can do? This is the website and data in question:
http://www.onequestionshootout.xyz/episodes/series_all.htm
Solved! Go to Solution.
@Anonymous ,
I would suggest you to use python script in power bi to scrapy the website. About how to configure python environment and implement python script in power bi desktop, I would suggest you to refer to doc below:
https://docs.microsoft.com/en-us/power-bi/desktop-python-scripts
Community Support Team _ Jimmy Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
I'm afraid my skills at this point won't allow for python scripting, so in the meanwhile I've downloaded the page as .html and I used the Text/CSV data connector to get the table in plain HTML. The downside of course is I cannot get the latest updates to my report over the internet.
@Anonymous
Wow! Thank you for sharing the idea of dowloading the page as an HTML! I was having the same problem as you and was completely stuck. With your solution, I have at least succeeded in extracting a "snapshot" of the data as it stands currently, which is better than no data at all...
I would have never thought of downloading the actual page!
Thanks!!
Proud to be a Super User!
Paul on Linkedin.
@Anonymous ,
Power query only support simple web scrapying. If the website needs dynamic scrapying, I'm afraid power query won't work.
Community Support Team _ Jimmy Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
@Anonymous
Perhaps this resource will help.
https://datachant.com/2017/03/30/web-scraping-power-bi-excel-power-query/
Proud to be a Super User!
@Anonymous ,
I would suggest you to use python script in power bi to scrapy the website. About how to configure python environment and implement python script in power bi desktop, I would suggest you to refer to doc below:
https://docs.microsoft.com/en-us/power-bi/desktop-python-scripts
Community Support Team _ Jimmy Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
107 | |
98 | |
77 | |
66 | |
53 |
User | Count |
---|---|
144 | |
104 | |
100 | |
86 | |
64 |