Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
olimilo
Responsive Resident
Responsive Resident

Dataflow timeout with Sharepoint.Files connector

I'm getting a dataflow timeout with some files I'm trying to pull from a Sharepoint folder. To my knowledge, there is no way to specify a timeout duration for the Sharepoint.Files connector - is there some way I can pull the data without timing out? I am unable to proceed with the query transformation due to this. Files being pulled are approximately 35mb each (total 200+mb).

 

Code below:

let
  Source = SharePoint.Files("https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/", [ApiVersion = 15]),
  #"Filter Path" = Table.SelectRows(Source, each Text.StartsWith([Folder Path], "https://xxxxx.sharepoint.com/sites/xxxxx/xxxxx/Data Repository/VData/")),
  #"Get Year" = Table.AddColumn(#"Filter Path", "Year", each Number.FromText(Text.BetweenDelimiters([Name], "(", ")")), type number),
  #"Filter Files" = Table.SelectRows(#"Get Year", each [Year] >= 2014 and [Year] <= 2021 and not Text.Contains([Name], "_q")),
  #"Get Data" = Table.AddColumn(#"Filter Files", "Data", each Excel.Workbook([Content])),
  #"Select Data Column" = Table.SelectColumns(#"Get Data", "Data")
in
  #"Select Data Column"

 

2 REPLIES 2
lbendlin
Super User
Super User

We utilize a rudimentary retry approach, like so:

 

 

...
#"Added custom" = Table.AddColumn(#"Filtered rows", "Data", each Table.SelectColumns(Table.PromoteHeaders(
    try Csv.Document([Content], [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv])
    otherwise Csv.Document(Binary.Buffer([Content]), [Delimiter = ",", Encoding = 1252, QuoteStyle = QuoteStyle.Csv])
    , [PromoteAllScalars = true]), Columns)),
...

 

That allows us to ingest 160 CSV files (up to 300 MB each) from a sharepoint folder.

 

olimilo
Responsive Resident
Responsive Resident

Thanks, I think this has something to do with our dataflow code overall since when I get this error, the dialog box shows the all of the queries that are inside the dataflow which gives me the impression that we will have to silo some of our data on a different dataflow/query altogether.

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors