Power Query very slow in extracting data from website
So I am trying to extract data from a retail website , to get urls first for the products , then from the URL extracting the data.
So I am tryingn to get 29 pages of data , each page 96 products , 2784 products data .
Problem is everthing works fine except data loading into the model from power query , its very slow , it take 5-6 hours to get the data and ultimately it fails , though i can see the data in power query editor screen.
How can I increase the Power Query Speed to extract data from internet , and also how to load it faster ,without it again extracting data while loading in to model. And also not crashing finally.
Guys this is really annoying, extremely slow, I have tried against several webpages and to no avail. Currently it seems it doesn't run in parallel. Let's say you have 1000 pages then it should at least assign the workload simultaneously, with R or Python you can run a loop in parallel, without that feature, is a deal breaker.
I was hoping to use a M query in a package in SSIS, but the performance is simply not enterprise or business ready!