cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
mumbaicharaja
Regular Visitor

How to process files from onedrive subfolders recursively

I want to know how to recursively process files in subfolders of onedrive. This is needed for consolidating the data from all the excel files in given folder and all subfolders underneath.. I already have PowerQuery written to go to a specific folder and process all the files in that. However, it needs to happen recursively for all the sub-folders and sub-sub-folder in that. Any guidance please?

5 REPLIES 5
Rocco_sprmnt21
Community Champion
Community Champion

it is just a guess, but try if this is a good starting point:

 

 

let

   dirall=(path)=>
   let 
    Source = Folder.Contents(path), 
    dir=List.Accumulate(Table.ToRows(Source[[Content],[Name]]),{},(s,c)=>if Value.Is(c{0}, type table) then s&@dirall(path &"\"& c{1})  else s&{c{1}} )
    in dir

in
    dirall

 

 

this function recursively goes through all the folders and subfolders of a directory and supplies only the name of the files.
but only you know what else to do with these files

here a version that add some action to the files.

 

let

   dirall=(path)=>
   let 
    Source = Folder.Contents(path), 
    dir=List.Accumulate(Table.ToRows(Source[[Content],[Name]]),{},(s,c)=>if Value.Is(c{0}, type table) then s&@dirall(path &"\"& c{1})  else s&{[name=c{1},len=Text.Length(Text.BeforeDelimiter(c{1},".")) ]} )
    in dir

in
    dirall

 

 

image.png

for each files name the last number is the number file in the subfolder.

 

f212 for example is the second file in the filder f21 which is the first subfolder of folder  f2 wich is the second subfolder of folder "test"

 

 

image.png

 

 

image.png

 

 

 

image.png

may be this could be more usefull to your goal

 

 

 

 

let

   dirall=(path)=>
   let 
    Source = Folder.Contents(path), 
    dir=List.Accumulate(Table.ToRows(Source[[Content],[Name]]),{},(s,c)=>if Value.Is(c{0}, type table) then s&@dirall(path &"\"& c{1})  else s&{path & "\"& c{1}} )
    in  
dir
in
    dirall

 

 

 

 

image.png

having all the files with complete path you can apply your function (I suppose)

 

#####

in the following form it is probably easier to understand

 

 

 

 

let
  dirall = (path) => List.Accumulate(
      Table.ToRecords(Folder.Contents(path)), 
      {}, 
      (s, c) => 
        if Value.Is(c[Content], type table) then 
          s & @dirall(path & "\" & c[Name])
        else 
          s & {path & "\" & c[Name]}
    )
in
  dirall

 

 

edhans
Super User
Super User

Do this using the SharePoint connector @mumbaicharaja 

  1. Go to OneDrive in your browser. You will get a URL that looks like this:
    1. https://tenantname-my.sharepoint.com/personal/username_tenantname_com/_layouts/15/onedrive.aspx
  2. You want to use this part - https://tenantname-my.sharepoint.com/personal/username_tenantname_com
  3. Sign in
  4. Click Transform Data
  5. Go to the folder path column
  6. Filter for the folder you want. You want to use "Contains" so it will get that folder and all folders with that info below it.
  7. Filter by file type is needed. A good idea just in case other files creep in there.
  8. Press the Combine button and follow the wizard 
     
     
     

    2020-08-14 11_22_56-Query1 - Power Query Editor.png



Did I answer your question? Mark my post as a solution!
Did my answers help arrive at a solution? Give it a kudos by clicking the Thumbs Up!

DAX is for Analysis. Power Query is for Data Modeling


Proud to be a Super User!

MCSA: BI Reporting
amitchandak
Super User
Super User

@mumbaicharaja , refer if these can help

https://powerbi.microsoft.com/en-us/blog/combining-excel-files-hosted-on-a-sharepoint-folder/

https://medium.com/@wkrzywiec/getting-data-from-the-latest-file-in-a-folder-using-power-query-51dfa4...



New Power BI Features
Datamarts: https://youtu.be/8tskWsJTEpg
Field Parameters : https://youtu.be/lqF3Wa1FllE?t=70
Did I answer your question? Mark my post as a solution! Appreciate your Kudos!! !!
Dashboard of My Blogs !! Connect on Linkedin !! Proud to be a Super User!
Want To Learn Power BI
Learn Power BI Beginners !! Advance Power BI Concepts !! Power BI For Tableau User !! Learn Power BI in Hindi !!
!! Subscribe to my youtube Channel !!

Helpful resources

Announcements
Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Power BI Dev Camp Session 23 768x460.jpg

Check it Out!

Mark your calendars and join us on Thursday, June 30 at 11a PDT for a great session with Ted Pattison!

Top Solution Authors