Load distinct excel Sheets from Share point folder

[Anonymous]

Dear Experts,

I would like to load distinct excel spreadsheets from the sharepoint folder. 

In detail: My share point has a folder called 'Customer List'--> inside 'Customer List' folder  -->there are Monthly folders example 'Jan2019', 'Feb2019', 'Mar2019' folders available each folder has approximatly 10 to 15 excel spreadsheets.

when i load the data into power BI, it consumes all the spreasheets from Customer List, but i got many duplicates which i want to avoid duplicates please suggest me the solution

the data is like below

Folder-Oct2019

1. 231Tom.xlsx

2. 232Racheal.xlsx

3. 768Kiran.xlsx

Folder-Nov2019

1. 988John.xlsx

2. 232Racheal.xlsx

I am expecting the end result like below:

1. 231Tom.xlsx

2. 232Racheal.xlsx

3. 768Kiran.xlsx

4. 988John.xlsx

please suggest me the solution in powerBI

[JarroVGIT]

I've recreated an example with a local folder, it would work the same. If you load the folder and have a list of all files, it should look like this:

Note that data1 and data3 are duplicates, they are in different subfolders in my case. 

Right click on the column "Name" and select Remove duplicates

This deletes all subsequent occurences. If you want to keep the last occurence rather then the first, perform some filter and sorting magic before removing duplicates 🙂

From here, you can merge and load all binaries like normal.

Kind regards

Djerro123

-------------------------------

If this answered your question, please mark it as the Solution. This also helps others to find what they are looking for.

Kudo's are welcome 🙂

[Anonymous]

Hi Djerro, 

Many thanks for the kind reply.  nearly reached.. before mark it as solution, I have couple of files with the below format which cannot remove

example:

Data1Nov.csv

Data1Dec.csv

I want to pickup recent distict file which is just Data1Dec.csv.  

please shed some light.

thanks,

[JarroVGIT · Accepted solution]

Hmm that would require some date magic I guess, based on folder name and file name. For a full logic solution we would need to know what the exact structure is. 

First step is to create columns based on the filename and/or filepaths, containing just 'data1' rather then 'data1Nov.csv'. Then you can remove duplicates but without the full list of folders/files (or a full list of possible combinations), it would be guessing 🙂

Kind regards

Djerro123

-------------------------------

If this answered your question, please mark it as the Solution. This also helps others to find what they are looking for.

Kudo's are welcome 🙂