Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
I'm getting a dataflow timeout with some files I'm trying to pull from a Sharepoint folder. To my knowledge, there is no way to specify a timeout duration for the Sharepoint.Files connector - is there some way I can pull the data without timing out? I am unable to proceed with the query transformation due to this. Files being pulled are approximately 35mb each (total 200+mb).
Code below:
let
Source = SharePoint.Files("https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/", [ApiVersion = 15]),
#"Filter Path" = Table.SelectRows(Source, each Text.StartsWith([Folder Path], "https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/Data Repository/VData/")),
#"Get Year" = Table.AddColumn(#"Filter Path", "Year", each Number.FromText(Text.BetweenDelimiters([Name], "(", ")")), type number),
#"Filter Files" = Table.SelectRows(#"Get Year", each [Year] >= 2014 and [Year] <= 2021 and not Text.Contains([Name], "_q")),
#"Get Data" = Table.AddColumn(#"Filter Files", "Data", each Excel.Workbook([Content])),
#"Select Data Column" = Table.SelectColumns(#"Get Data", "Data")
in
#"Select Data Column"
We utilize a rudimentary retry approach, like so:
...
#"Added custom" = Table.AddColumn(#"Filtered rows", "Data", each Table.SelectColumns(Table.PromoteHeaders(
try Csv.Document([Content], [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv])
otherwise Csv.Document(Binary.Buffer([Content]), [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv])
, [PromoteAllScalars = true]), Columns)),
...
That allows us to ingest 160 CSV files (up to 300 MB each) from a sharepoint folder.
Thanks, I think this has something to do with our dataflow code overall since when I get this error, the dialog box shows the all of the queries that are inside the dataflow which gives me the impression that we will have to silo some of our data on a different dataflow/query altogether.
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.