Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Bug: Web.Page function in Power Query ignores HTTP Headers

When trying to fetch content from a web page, you sometimes want to fetch it in a specific language. This is possible by setting the Accept-Language in the HTTP Header.

 

However, this doesn't work as you expect. The following code:

let
 Source = Web.Page(Web.Contents("http://www.imdb.com/chart/top", [Headers=[#"Accept-Language"="en-US"]])),
 Data0 = Source{0}[Data],
 #"Changed Type" = Table.TransformColumnTypes(Data0,{{"", type text}, {"Rank & Title", type text}, {"IMDb Rating", type number}, {"Your Rating", type text}, {"2", type text}})
in
 #"Changed Type"

 

Produces these headers (collected with Fiddler):

GET /chart/top HTTP/1.1
Accept: */*
Accept-Language: fi,en-US;q=0.7,en;q=0.3
UA-CPU: AMD64
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; Win64; x64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)
Host: www.imdb.com
Connection: Keep-Alive

As you can see, it first language is fi (since my laptop is configured with a Finnish setup). I would expect the header to be only en-US as defined in the Power Query.

 

The bug is in the Web.Page function, because if I first fetch the content and the parse it then it works. I.e. the code below works fine:

let
 SourceAsText = Text.FromBinary(Web.Contents("http://www.imdb.com/chart/top", [Headers=[Accept="html", #"Accept-Language"="en-US"]])),
 Source = Web.Page(SourceAsText),
 Data0 = Source{0}[Data],
 #"Changed Type" = Table.TransformColumnTypes(Data0,{{"", type text}, {"Rank & Title", type text}, {"IMDb Rating", type number}, {"Your Rating", type text}, {"2", type text}})
in
 #"Changed Type"

 

Here is the header, from the http query which now is correct:

GET /chart/top HTTP/1.1
Accept: html
Accept-Language: en-US
User-Agent: Microsoft.Data.Mashup (https://go.microsoft.com/fwlink/?LinkID=304225)
Accept-Encoding: gzip, deflate
Host: www.imdb.com

Status: Accepted
Comments
v-jiascu-msft
Employee

Hi @craiha,

 

I have reported this issue to the Product Team: CRI 58776671. I will update the latest news here.

 

Best Regards,

Dale

Vicky_Song
Impactful Individual
Status changed to: Accepted
 
v-jiascu-msft
Employee

Hi @craiha,

 

I have gotten responses from the Product Team:

 

This is a result of the way M is evaluated, and how Web.Page is built on old IE technology. The workaround the customer has already discovered (where you force an evaluation of Web.Contents (for instance by using Text.FromBinary) before passing the results to Web.Page) is what should be recommended in these cases in the future.

 

Best Regards,

Dale