Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
yk1984
Regular Visitor

Web Scraping always bringing back first 10 line items only!

Hello all,

 

I am trying to use the web scraping feature in Power BI to extract all open job positions from the following site: Estee Lauder Job Search - Jobs - Page 1 (elcompanies.com)

 

It shows the top 100 open vacancies but when ever i try to extract, it always only extracts the first 10 openings. I have also changed the page to show only 10 vacanices per page and then used a custom invoke logic to bring back the 10 openings per every page. When i do this it always shows me the same 10 job openings. Even if i enter the URL for page 2 of the vacancies, it still only brings back the first 10 vacancies from page one.

 

Web Scraping Example.JPG

 

Has anyone faced anything like this before or know of a solution to get around this?

 

Your help would be appreciated, many thanks in advance

1 ACCEPTED SOLUTION

Hello. I am using the version from the microsoft store. 

 

I have found a work around by bringing the data through data flow on the service which seems to have no issues in extracting all the data 😊

View solution in original post

9 REPLIES 9
DanBT
Regular Visitor

Hey anychance you could explain how you solved this through a dataflow? I'm having the exact same problem with a different website. 

What i did is use the web scraping as you would in desktop, copy the query and then paste into a dataflow blank query. This worked for me and showed all records, on desktop still shows on 10 records for some reason

yk1984
Regular Visitor

hello, yes i am on March 2020 version. Very strange that you get all 100 and i only keep getting 10 rows extracted. Are you just copying the web link using the Web data source?

 

Are you using the link https://jobs.elcompanies.com/estee-lauder-companies/jobs?page=1&limit=100

Yes, same link. See the screenshot below to see more than 10 records

1.png

ah thats so not fair its working for you but not me 😕

Hi @yk1984 ,

It also works fine in my side with the same version of Power BI Desktop.

vyingjl_0-1649903939855.png

So it may relate to the client site of Power BI Desktop not the web source site.

 

Have you used the Microsoft store app version or the web download version of Power BI Desktop?

In addition, you can try to use the previous version of Power BI Desktop to check it again.

 

Best Regards,
Community Support Team _ Yingjie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Hello. I am using the version from the microsoft store. 

 

I have found a work around by bringing the data through data flow on the service which seems to have no issues in extracting all the data 😊

I will suggest that you make a post to sister forum as well - Power BI Desktop Forum

Vijay_A_Verma
Super User
Super User

Not able to reproduce as I get all 100 through the above link. Are you on latest version of Power BI Desktop (Mar 2022)? See it in Help - About

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors