Get and use metadata in a pipeline

[Bendelyon]

Hello 

I'm on a MC fabric project and i use notebook to concatenate severals csv. I would have an other way to do this.

I want ton use pipeline and the get metadata activity.

The folder architecture is on 3 folder level. 

Example 

Parent folder / child folder / csv 

How should i use a pipeline and a for each activity to get the metadata and could copy the csv ?

Thanks for help

[Expiscornovus]

Hi @Bendelyon,

With in the for each activity you can use an array. With in the for each you can reference to items of that array via the Dynamic content. A reference would be:

@item()

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity#create-a-foreach...

Below is an example to demonstrate this with the Get metadata activity

1. Example Folder structure

2. Array variable 

3. Usage of variable in For Each Items property

@variables('CSVArray')

4. Reference to item of array in GetMetadata activity

@item()

5. Test run pipeline

[Bendelyon]

Hello

Thanks for your help and for five me the "logical" to think.

However i have many parent folder.

Example 

Parent folder a / child folder a / csv

Parent folder b / child folder a / csv 

Etc

I need to iterate on thé parent folder to enter on the child folder ans get the csv whitch is on.

Thanks for your help

[Expiscornovus · Accepted solution]

Hi @Bendelyon,

In that case you could also use a Get metadata activity before the for each to retrieve your structure first and use that as items in your for each loop instead of the variable I proposed.

Here is a YouTube video which shares such an approach: https://www.youtube.com/watch?v=EU6Ex9EhWyA

[v-gchenna-msft]

Hi @Bendelyon ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
Otherwise, will respond back with the more details and we will try to help .

[v-gchenna-msft]

Hi @Bendelyon ,

Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.

[NandanHegde]

Do you mean to concatenate all the CSV files into a single CSV file ?
Then the below blog acts as a sample reference :

https://www.c-sharpcorner.com/article/merge-multiple-json-files-via-synapse-data-factory-pipelines/

wherein you can use the Copy activity recursive and merge into single property

[Bendelyon]

Hello

My principal problem is to get metadata to navigate onto the folder structure.

Thanks