Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
Anonymous
Not applicable

How to bypass cookie popup when web scraping?

Hi there

 

Many websites have popups asking what cookies to allow, how do I get past these popups and load the content directly?

Best Regardes
Søren

8 REPLIES 8
v-stephen-msft
Community Support
Community Support

Hi @Anonymous ,

 

Based on my test, the data scraped by the webpage does not seem to be affected by the cookie.

let
    Source = Web.BrowserContents("https://www.betriheim.fo/solulisti/"),
    #"Extracted Table From Html" = Html.Table(Source, {{"Column1", ".section-inner BR + A"}, {"Column2", ".suggestedPrice"}, {"Column3", ".latestBid"}, {"Column4", ".icon-buildingSize + *"}, {"Column5", ".icon-buildingBedrooms + *"}, {"Column6", ".icon-plotSize + *"}, {"Column7", ".icon-buildingFloors + *"}, {"Column8", ".icon-buildingYearBuilt + *"}, {"Column9", ".suggestedPrice *"}}, [RowSelector=".section-inner BR + A"]),
    #"Changed Type" = Table.TransformColumnTypes(#"Extracted Table From Html",{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}, {"Column4", type text}, {"Column5", type text}, {"Column6", type text}, {"Column7", type text}, {"Column8", type text}, {"Column9", type text}})
in
    #"Changed Type"

 

I used to get data from the web, and the credentials were obtained anonymously.

1.png2.png

 

 

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Hi Stephen
Thanks for your effort.

Your proposed script does not provide usable data, I get a table with the email adress heim@betri.fo in column1 as the only data.

sosje_0-1629442407825.png

Am I doing something wrong?

Best regards
Søren

Anonymous
Not applicable

Hi Stephen
Thanks for your effort.

Your proposed script does not provide usable data, I get a table with the email adress heim@betri.fo in column1 as the only data.

sosje_0-1629442407825.png

Am I doing something wrong?

Best regards
Søren

Hi @Anonymous ,

 

My fault, sorry. Because copying the code directly to the advanced editor seems to make an error. Please try to follow the steps in my gif to connect to the web.

10.gif

 

 

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Anonymous
Not applicable

Hi again Stephen

 

Hi again Stephen

I tried on another computer and got what you show in your GIF. But data only appears as examples, when I load I get again only one row with an email address just like when I copy the query directly in the advanced editor.
Strange?
It looks like something is happening from preview to loading.

Best Regards
Søren

Anonymous
Not applicable

Hi Stephen

 

I dont get the same screens as you?

I get:

sosje_0-1629451984570.png

 

sosje_1-1629451984704.png

 

 

sosje_2-1629451984575.png

 

which is the content of the cookie. 

 

Best Regards

Søren

v-stephen-msft
Community Support
Community Support

Hi @Anonymous ,

 

Please kindly refer to the similar post:

Passing Cookie using Web.Contents in PowerBI Desktop

 

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

 

Anonymous
Not applicable

Hi Stephen

Thanks for the replay, it unfortuntely does not solv my problem. This is the website I'm trying to scrape: https://www.betriheim.fo/solulisti/

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors