How to migrate data

[Anonymous]

Hello all

I first want to say that I am a complete beginner at this. Any and all advice and/or external reading material you can give me is greatly appreciated.

So I have been given data that represents end-of-day inventory records for a supply plant. The data is separated by plant into tables (5 total). In each of the tables (they're sharepoint lists), there is a date column and then two columns for each "bin" in the plant, namely product and quantity. It ends up as like 40ish columns in each table. I understand that this is impractical for working with Power BI, so I would like to transpose my data.

I would like to end up with a single table in which I have the columns: date, plant, bin, product, and quantity. Unfortunately I am quite inexperienced in power BI and I don't know how write a formula to calculate this table from the 5 separate sharepoint lists.

Any ideas?

Thank you in advance.

[mahoneypat]

Not sure how big your SharePoint lists are but here is a video that shows you how to pull the data fast if your current refresh times are slow.

Get SharePoint List Data with Power BI ... Fast - YouTube

Regards,

Pat

[Jimmy801 · Accepted solution]

Hello @Anonymous 

first of all you need to normalize your sharepoint lists. It means that the information of your bin is not in a column, but in a column name, or am I wrong? This probably can be achieved by using a Table.Unpivot. After that you have also to add a new column that identifies your list-name (plant information). Maybe some filtering is also needed, depending if your list contains also oder data and you need to find only the last version. After that you can combine your 5 transformed table into 1 table. Here a short example of a possible normalization, but without any knowledge of your data, it's not possible to make any serious proposal. 

This code was generated only by using the GUI, so it was not even needed to write m-code.

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjTSMzDUMzJQ0lFyBGJTKG2hFKsDlDRGkTREYoClEXqdwKIwloWBAbp+kLCpAZJSoIpYAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Date = _t, #"BINA - Product" = _t, #"BINA - Quantity" = _t, #"BINB - Product" = _t, #"BINB - Quantity" = _t]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"Date",type date}, {"BINA - Product", type text}, {"BINA - Quantity", Int64.Type}, {"BINB - Product", type text}, {"BINB - Quantity", Int64.Type}}, "de-DE"),
    FilterForLastDayIfNeeded = Table.SelectRows(#"Changed Type", let latest = List.Max(#"Changed Type"[Date]) in each [Date] = latest),
    #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(FilterForLastDayIfNeeded, {"Date"}, "Attribute", "Value"),
    #"Split Column by Delimiter" = Table.SplitColumn(#"Unpivoted Other Columns", "Attribute", Splitter.SplitTextByEachDelimiter({" - "}, QuoteStyle.Csv, false), {"Attribute.1", "Attribute.2"}),
    #"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Attribute.1", type text}, {"Attribute.2", type text}}),
    #"Added Custom" = Table.AddColumn(#"Changed Type1", "Product", each if [Attribute.2]="Product" then [Value] else null),
    #"Filled Down" = Table.FillDown(#"Added Custom",{"Product"}),
    #"Filtered Rows" = Table.SelectRows(#"Filled Down", each ([Attribute.2] = "Quantity")),
    #"Renamed Columns1" = Table.RenameColumns(#"Filtered Rows",{{"Value", "Quantity"}, {"Attribute.1", "BIN"}}),
    #"Removed Columns" = Table.RemoveColumns(#"Renamed Columns1",{"Attribute.2"})
in
    #"Removed Columns"

Copy paste this code to the advanced editor in a new blank query to see how the solution works. 

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

[lbendlin]

Power BI has two distinct components - the data preparation part (sometimes called ETL -Extract, Transform, Load) and the data presentation part (called the UI - user interface).  The problem you describe falls into the former part, meaning that you can prepare your data in Power Query (that's the ETL component of Power BI) rather than in DAX.

Power Query has lots of options for unpivoting, and for combining lists into a single table. Give it a try, and let us know if you get stuck.  

The number of columns is not really something you should be worried about - You should first learn why they created these columns in the first place, what their usage and cardinality is, if there are hierarchies etc.  Then you can decide if the table/list structures can be improved. That's all part of ETL.