cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
olimilo
Helper V
Helper V

Dataflow timeout with Sharepoint.Files connector

I'm getting a dataflow timeout with some files I'm trying to pull from a Sharepoint folder. To my knowledge, there is no way to specify a timeout duration for the Sharepoint.Files connector - is there some way I can pull the data without timing out? I am unable to proceed with the query transformation due to this. Files being pulled are approximately 35mb each (total 200+mb).

 

Code below:

let
  Source = SharePoint.Files("https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/", [ApiVersion = 15]),
  #"Filter Path" = Table.SelectRows(Source, each Text.StartsWith([Folder Path], "https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/Data Repository/VData/")),
  #"Get Year" = Table.AddColumn(#"Filter Path", "Year", each Number.FromText(Text.BetweenDelimiters([Name], "(", ")")), type number),
  #"Filter Files" = Table.SelectRows(#"Get Year", each [Year] >= 2014 and [Year] <= 2021 and not Text.Contains([Name], "_q")),
  #"Get Data" = Table.AddColumn(#"Filter Files", "Data", each Excel.Workbook([Content])),
  #"Select Data Column" = Table.SelectColumns(#"Get Data", "Data")
in
  #"Select Data Column"

 

2 REPLIES 2
lbendlin
Super User
Super User

We utilize a rudimentary retry approach, like so:

 

 

...
#"Added custom" = Table.AddColumn(#"Filtered rows", "Data", each Table.SelectColumns(Table.PromoteHeaders(
    try Csv.Document([Content], [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv])
    otherwise Csv.Document(Binary.Buffer([Content]), [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv])
    , [PromoteAllScalars = true]), Columns)),
...

 

That allows us to ingest 160 CSV files (up to 300 MB each) from a sharepoint folder.

 

Thanks, I think this has something to do with our dataflow code overall since when I get this error, the dialog box shows the all of the queries that are inside the dataflow which gives me the impression that we will have to silo some of our data on a different dataflow/query altogether.

Helpful resources

Announcements
UG GA Amplification 768x460.png

Launching new user group features

Learn how to create your own user groups today!

November Power BI Update 768x460.png

Check it Out!

Click here to read more about the November 2021 Updates!

M365 768x460.jpg

Microsoft 365 Collaboration Conference | December 7–9, 2021

Join us, in-person, December 7–9 in Las Vegas, for the largest gathering of the Microsoft community in the world.

Top Kudoed Authors