Not able to load all the rows from web to Query Editor

[Anonymous]

Hi,

I am trying to load historical stock data from Yahoo to power BI using web URL https://au.finance.yahoo.com/quote/%5EAXJO/history?period1=722476800&period2=1585353600&interval=1d&...

When I load the data to query editor, it loads only first 100 rows even though it has more than 5000 records. 

Can anyone help me with the hack to load all the records from that URL?

My final goal is to get dynamic stock data into Power BI. 

Thank you

[Greg_Deckler]

I think @ImkeF has tricks for getting data off of web pages via query string. Also @edhans might be able to help.

[edhans]

I'm kinda stumped. As you scroll that page, the browser is making additional calls to the site, which is returning more data and returning more advertising. It seems to be limiting it intentionally.

One thing you could do is download the CSV file and use that as one table, then download these 100 records as the most recent, then combine into one table and eliminate those in the CSV via an anti-merge, but you'd have to redownload the CSV file every 100 business days or so to keep it going. Not at all ideal....

[Anonymous]

Yes, I could do that. But as I add more stocks, I need to download CSV file for each stock every time I want to update. 

I want to make it completely automatic. 

[edhans]

Yes, it isn't a good solution long term. I'll be curious to see if some responds with an answer that can get PQ to keep pulling all 7K+ records in.

[Anonymous]

Even I tried to import the data in Google Sheet using ImportHTML function. 

Even the Spreadsheet is pulling 100 rows only. The records stop at when the first page load takes on the yahoo page. 

[v-lionel-msft]

Hi @Anonymous ,

The data on this website is dynamically loaded through JS scripts, there is no way to import all the data through the Web Connector.
If possible, I suggest you use other data source.

Best regards,
Lionel Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[amitchandak]

@Anonymous ,

Refer if these can help,

https://docs.microsoft.com/en-us/power-bi/service-real-time-streaming

https://community.powerbi.com/t5/Desktop/Live-Stream-Web-URL-data-in-PBI/td-p/412651