Hi all,
The main purpose is to find a way to scrap all source code from a web page. So far, "From web" method doesn't work reliably for me when imported as csv, text or html because some of the content is not imported.
I was trying to scrap page ids from facebook pages. In the source code, one occurence always comes after the string "/?id=", as such:
I first tried through the get "From web" option of powerquery, but the result is rather disappointing; it either times out or imports (very) incomplete source code, without most of the <head> part:
I the imported it as text and wrote the following function which worked:
let page_id = (local_FacebookPage) =>
let
Source = Table.FromColumns({Lines.FromBinary(Web.Contents(local_FacebookPage))}),
#"Kept First Rows" = Table.FirstN(Source,4),
#"Split Column by Delimiter" = Table.SplitColumn(#"Kept First Rows", "Column1", Splitter.SplitTextByEachDelimiter({"/?id="}, QuoteStyle.None, false), {"Column1.1", "Column1.2"}),
#"Split Column by Delimiter1" = Table.SplitColumn(#"Split Column by Delimiter", "Column1.2", Splitter.SplitTextByEachDelimiter({""""}, QuoteStyle.None, false), {"Column1.2.1", "Column1.2.2"}),
#"Removed Columns" = Table.RemoveColumns(#"Split Column by Delimiter1",{"Column1.1", "Column1.2.2"}),
extract_page_id = Text.Combine(Table.Column(#"Removed Columns", "Column1.2.1"))
in
extract_page_id
in
page_id
A day ago it stopped working on my test page: https://www.facebook.com/groups/TalkPowerBI/.
The explanation is that even imported as text, it shows only part of the content now, and not the part where my function is looking for the id.
Somehow though, it still works on other pages like: https://www.facebook.com/PowerBIGuru/.
- Any thought as to why ?
- Anyone has a reliable method to scrap through powerquery ?
Antoine
I think the reason is that the Guru page is public while the Talk page is not. When I open a Chrome window in incognito mode, the Talk page redirects me to login. I think that's the same thing happening for Power BI desktop. It actually scarps the login page so that's why no "/?id" found.
I don't find that "page/?id" even when I view the source code from Chrome for the TalkPowerBI page.
For the page PowerBIGuru I can find the "page/?id" and your PowerQuery captured the expected id.
So I think Power BI is just working fine. The issue is the facebook pages seems not always containing the page id.
By the way, in my opinion, Power BI is a good tool for web scarping.
Hi @Eric_Zhang,
Thank you for your reply.
The string to look for is "/?id=" and not "page/?id" (facebook has different page types, including "page/?id=", "group/?id=" and such).
So in your example, for talkPowerBI, it looks like this :
And the id is "661419710731947".
And this one does not work in my function, this part is not imported.
Antoine
I think the reason is that the Guru page is public while the Talk page is not. When I open a Chrome window in incognito mode, the Talk page redirects me to login. I think that's the same thing happening for Power BI desktop. It actually scarps the login page so that's why no "/?id" found.
Helpful resources
Microsoft Fabric Learn Together
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Power BI Monthly Update - April 2024
Check out the April 2024 Power BI update to learn about new features.
Fabric Community Update - April 2024
Find out what's new and trending in the Fabric Community.
