Pull all files from all URL's that are visible on a webpage

[Anonymous]

Hi,

An agency I work with has given me a download of the data I need. It's 2,500 (!) CSV files. They've given me a webpage with the links to the files. So imagine that you need to go to www.download.com/thisisthepage.html and then you only see the following:

https://download01-01-2019.csv

https://download02-01-2019.csv

https://download03-01-2019.csv

... and this 2500 times.

I can easily pull a list of all the download link URL's in powerbi by using get data from web, and then drilling a bit deeper into the HTML. But the next step would be to load all those URL's and turn them into one query how can I pull all the data of these links in one query so I can start manipulating it?

Regards

Bas

[Greg_Deckler]

Most likely need a custom M function. @ImkeF or @edhans can probably assist.

[ImkeF · Accepted solution]

Hi @Anonymous 

once you have all csv-URLs in a column of a table, add another column with the formula: Web.Contents([YourColumnName])

This will retrieve the binary content from the web and create some arrows in the table name the should trigger the "Combine binaries"-procedure. This will create a query that will automatically be connected to a function ... that will automatically be applied to all your files. You can then perform all your transformations in the query and everything you do in there will automagically be applied to the other files as well (through the connected function).

[MmmVvv_23]

Wow! I just bumped into this post, and I tried it - it is functioning perfectly! Thanks for that!

[Anonymous]

That works perfect @ImkeF , you're a star! Thank you very much.