Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
So I am trying to extract data from a retail website , to get urls first for the products , then from the URL extracting the data.
So I am tryingn to get 29 pages of data , each page 96 products , 2784 products data .
Problem is everthing works fine except data loading into the model from power query , its very slow , it take 5-6 hours to get the data and ultimately it fails , though i can see the data in power query editor screen.
How can I increase the Power Query Speed to extract data from internet , and also how to load it faster ,without it again extracting data while loading in to model. And also not crashing finally.
below is the code for same.
et Source = {1..29}, #"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error), #"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Column1", Int64.Type}}), #"Renamed Columns" = Table.RenameColumns(#"Changed Type",{{"Column1", "Page"}}), #"Added Custom" = Table.AddColumn(#"Renamed Columns", "Custom", each women([Page])), #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom", "Custom", {"Link", "Merged", "desc", "Type", "URL"}, {"Link", "Merged", "desc", "Type", "URL"}), #"Removed Errors" = Table.RemoveRowsWithErrors(#"Expanded Custom"), #"Removed Columns" = Table.RemoveColumns(#"Removed Errors",{"Page", "Link", "Merged", "desc", "Type"}), #"Invoked Custom Function" = Table.AddColumn(#"Removed Columns", "skudata", each skudata([URL])), #"Expanded skudata" = Table.ExpandTableColumn(#"Invoked Custom Function", "skudata", {"SKU", "cl", "cl 1.", "Suppliet", "Supplier No.", "Column1", "Column2", "Column3", "Supplier Style No."}, {"SKU", "cl", "cl 1.", "Suppliet", "Supplier No.", "Column1", "Column2", "Column3", "Supplier Style No."}) in #"Expanded skudata"
Try to disable File > Options and settings > Options > CURRENT FILE > Data Load > Allow data preview to download in the background.
Did you find any solution? I have same problem. I'm also trying to get some data from website and its terribly slow. It's taking 6-7 hours.
I have tried the preview diasble also , still its very slow and unable to extract all data for 29 pages .
The URL link extraction function works fine even for 400 pages , the first function (women) created to extract links from the website..
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
112 | |
97 | |
84 | |
67 | |
59 |
User | Count |
---|---|
150 | |
120 | |
99 | |
87 | |
68 |