Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
Hi.
I have been looking through the Search for a way to accomplish my desired format manipulation but I'm not sure that I'm using the appropriate terminology. Is it possible to perform the following data conversion when data is imported?
I would like to take data that is exported in this format;
Event Id | Res Type | Action | Response Time |
468745 | MA | OnScene | 4/1/2020 5:29 |
468745 | MA | Departed | 4/1/2020 5:31 |
468746 | TW | OnScene | 4/1/2020 5:58 |
468746 | TW | Departed | 4/1/2020 6:53 |
468746 | P | EnRoute | 4/1/2020 5:34 |
468746 | P | OnScene | 4/1/2020 5:37 |
468746 | P | Departed | 4/1/2020 6:53 |
468746 | F | EnRoute | 4/1/2020 5:34 |
468746 | F | OnScene | 4/1/2020 5:36 |
468746 | F | Departed | 4/1/2020 6:23 |
468746 | E | EnRoute | 4/1/2020 5:34 |
468746 | E | OnScene | 4/1/2020 5:37 |
468746 | E | Departed | 4/1/2020 6:23 |
468746 | MA | Notified | 4/1/2020 5:34 |
468746 | MA | EnRoute | 4/1/2020 5:34 |
468746 | MA | OnScene | 4/1/2020 5:40 |
468746 | MA | Departed | 4/1/2020 5:41 |
468746 | MA | Notified | 4/1/2020 5:34 |
468746 | MA | EnRoute | 4/1/2020 5:34 |
468746 | MA | OnScene | 4/1/2020 5:41 |
468746 | MA | Departed | 4/1/2020 6:53 |
468746 | TW | OnScene | 4/1/2020 6:31 |
468746 | TW | Departed | 4/1/2020 6:53 |
468747 | MA | OnScene | 4/1/2020 5:56 |
468747 | MA | Departed | 4/1/2020 5:56 |
468748 | MA | OnScene | 4/1/2020 6:00 |
468748 | MA | Departed | 4/1/2020 6:02 |
So that it reads in this format;
EventID | Resp1Type | Resp1Notified | Resp1EnRoute | Resp1OnScene | Resp1Departed | Resp2Type | Resp2Notified | Resp2EnRoute | Resp2OnScene | Resp2Departed | Resp3Type | Resp3Notified | Resp3EnRoute | Resp3OnScene | Resp3Departed | Resp4Type | Resp4Notified | Resp4EnRoute | Resp4OnScene | Resp4Departed | Resp5Type | Resp5Notified | Resp5EnRoute | Resp5OnScene | Resp5Departed | Resp6Type | Resp6Notified | Resp6EnRoute | Resp6OnScene | Resp6Departed | Resp7Type | Resp7Notified | Resp7EnRoute | Resp7OnScene | Resp7Departed |
468745 | MA | 4/1/2020 5:29 | 4/1/2020 5:31 | ||||||||||||||||||||||||||||||||
468746 | TW | 4/1/2020 5:58 | 4/1/2020 6:53 | P | 4/1/2020 5:34 | 4/1/2020 5:37 | 4/1/2020 6:53 | F | 4/1/2020 5:34 | 4/1/2020 5:36 | 4/1/2020 6:23 | E | 4/1/2020 5:34 | 4/1/2020 5:37 | 4/1/2020 6:23 | MA | 4/1/2020 5:34 | 4/1/2020 5:34 | 4/1/2020 5:40 | 4/1/2020 5:41 | MA | 4/1/2020 5:34 | 4/1/2020 5:34 | 4/1/2020 5:41 | 4/1/2020 6:53 | TW | 4/1/2020 6:31 | 4/1/2020 6:53 | |||||||
468747 | MA | 4/1/2020 5:56 | 4/1/2020 5:56 | ||||||||||||||||||||||||||||||||
468748 | MA | 4/1/2020 6:00 | 4/1/2020 6:02 |
Solved! Go to Solution.
@Anonymous ,
Based on the Excel example, I've removed the Responder Id column.
Try this new m code:
let
Source = Csv.Document(File.Contents("D:\Downloads\Book2.csv"),[Delimiter=",", Columns=6, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Filtered Rows" = Table.SelectRows(#"Promoted Headers", each ([Event Id] <> "")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Responder Id"}),
#"Changed Type" = Table.TransformColumnTypes(#"Removed Columns",{{"Event Id", Int64.Type}, {"Responder Type", type text}, {"Name", type text}, {"Action", type text}, {"Response Time", type datetime}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Event Id", "Action"}, {{"Rows", each Table.AddIndexColumn(_, "Index", 1,1), type table}}),
#"Removed Other Columns" = Table.SelectColumns(#"Grouped Rows",{"Rows"}),
#"Expanded Rows" = Table.ExpandTableColumn(#"Removed Other Columns", "Rows", {"Event Id", "Responder Type", "Name", "Action", "Response Time", "Index", "Res Type"}, {"Event Id", "Responder Type", "Name", "Action", "Response Time", "Index", "Res Type"}),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Expanded Rows", {"Event Id", "Name", "Action", "Response Time", "Index"}, "Item", "Value"),
#"Unpivoted Columns1" = Table.UnpivotOtherColumns(#"Unpivoted Columns", {"Event Id", "Action", "Response Time", "Index", "Item", "Value"}, "Attribute", "Value.1"),
#"Duplicated Column" = Table.DuplicateColumn(#"Unpivoted Columns1", "Index", "Index - Copy"),
#"Duplicated Column1" = Table.DuplicateColumn(#"Duplicated Column", "Index", "Index - Copy.1"),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Duplicated Column1", {{"Index - Copy", type text}}, "pt-BR"),{"Item", "Index - Copy"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Item"),
#"Merged Columns1" = Table.CombineColumns(Table.TransformColumnTypes(#"Merged Columns", {{"Index", type text}}, "pt-BR"),{"Action", "Index"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Action.1"),
#"Merged Columns2" = Table.CombineColumns(Table.TransformColumnTypes(#"Merged Columns1", {{"Index - Copy.1", type text}}, "pt-BR"),{"Attribute", "Index - Copy.1"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Name"),
Union =
Table.Combine({
Table.RenameColumns(Table.SelectColumns(#"Merged Columns2", {"Event Id", "Response Time", "Action.1"}), {{"Response Time", "Value"}, {"Action.1", "Item"}}),
Table.Distinct(Table.SelectColumns(#"Merged Columns2", {"Event Id", "Item", "Value"}), {"Event Id", "Item"}),
Table.Distinct(Table.RenameColumns(Table.SelectColumns(#"Merged Columns2", {"Event Id", "Name", "Value.1"}), {{"Name", "Item"}, {"Value.1", "Value"}}), {"Event Id", "Item"})
}),
#"Pivoted Column" = Table.Pivot(Union, List.Distinct(Union[Item]), "Item", "Value")
in
#"Pivoted Column"
@Anonymous ,
Based on the Excel example, I've removed the Responder Id column.
Try this new m code:
let
Source = Csv.Document(File.Contents("D:\Downloads\Book2.csv"),[Delimiter=",", Columns=6, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Filtered Rows" = Table.SelectRows(#"Promoted Headers", each ([Event Id] <> "")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Responder Id"}),
#"Changed Type" = Table.TransformColumnTypes(#"Removed Columns",{{"Event Id", Int64.Type}, {"Responder Type", type text}, {"Name", type text}, {"Action", type text}, {"Response Time", type datetime}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Event Id", "Action"}, {{"Rows", each Table.AddIndexColumn(_, "Index", 1,1), type table}}),
#"Removed Other Columns" = Table.SelectColumns(#"Grouped Rows",{"Rows"}),
#"Expanded Rows" = Table.ExpandTableColumn(#"Removed Other Columns", "Rows", {"Event Id", "Responder Type", "Name", "Action", "Response Time", "Index", "Res Type"}, {"Event Id", "Responder Type", "Name", "Action", "Response Time", "Index", "Res Type"}),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Expanded Rows", {"Event Id", "Name", "Action", "Response Time", "Index"}, "Item", "Value"),
#"Unpivoted Columns1" = Table.UnpivotOtherColumns(#"Unpivoted Columns", {"Event Id", "Action", "Response Time", "Index", "Item", "Value"}, "Attribute", "Value.1"),
#"Duplicated Column" = Table.DuplicateColumn(#"Unpivoted Columns1", "Index", "Index - Copy"),
#"Duplicated Column1" = Table.DuplicateColumn(#"Duplicated Column", "Index", "Index - Copy.1"),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Duplicated Column1", {{"Index - Copy", type text}}, "pt-BR"),{"Item", "Index - Copy"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Item"),
#"Merged Columns1" = Table.CombineColumns(Table.TransformColumnTypes(#"Merged Columns", {{"Index", type text}}, "pt-BR"),{"Action", "Index"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Action.1"),
#"Merged Columns2" = Table.CombineColumns(Table.TransformColumnTypes(#"Merged Columns1", {{"Index - Copy.1", type text}}, "pt-BR"),{"Attribute", "Index - Copy.1"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Name"),
Union =
Table.Combine({
Table.RenameColumns(Table.SelectColumns(#"Merged Columns2", {"Event Id", "Response Time", "Action.1"}), {{"Response Time", "Value"}, {"Action.1", "Item"}}),
Table.Distinct(Table.SelectColumns(#"Merged Columns2", {"Event Id", "Item", "Value"}), {"Event Id", "Item"}),
Table.Distinct(Table.RenameColumns(Table.SelectColumns(#"Merged Columns2", {"Event Id", "Name", "Value.1"}), {{"Name", "Item"}, {"Value.1", "Value"}}), {"Event Id", "Item"})
}),
#"Pivoted Column" = Table.Pivot(Union, List.Distinct(Union[Item]), "Item", "Value")
in
#"Pivoted Column"
Wow that was quick. This is awesome! Exactly what I was looking for and works great on other raw files. I can't thank you enough for being patient and taking the time to work with me.
Yeah I see what you're saying. I was hoping to get those on one row. You've done so much to this point I hate to ask too much more. I really appreciate all your help.
I am able to get it to work using the Raw Format file but when I attempt to change the source to the following file, Book2.csv, it doesn't produce the same results. I'm getting multiple row entries for Event Id 468738.
Here are the raw files and resulting formated file.
https://www.dropbox.com/s/4sgvdica6731m5e/Book2%20Final%20Format.zip?dl=0
@Anonymous ,
I'm not sure if it's the result you want:
let
Source = Csv.Document(File.Contents("D:\Downloads\responders2.csv"),[Delimiter=",", Columns=6, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Event Id", Int64.Type}, {"Responder Id", Int64.Type}, {"Responder Type", type text}, {"Name", type text}, {"Action", type text}, {"Response Time", type datetime}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Event Id", "Action"}, {{"Rows", each Table.AddIndexColumn(_, "Index", 1,1), type table}}),
#"Removed Other Columns" = Table.SelectColumns(#"Grouped Rows",{"Rows"}),
#"Expanded Rows" = Table.ExpandTableColumn(#"Removed Other Columns", "Rows", {"Event Id", "Responder Id", "Responder Type", "Name", "Action", "Response Time", "Index", "Res Type"}, {"Event Id", "Responder Id", "Responder Type", "Name", "Action", "Response Time", "Index", "Res Type"}),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Expanded Rows", {"Event Id", "Responder Id", "Name", "Action", "Response Time", "Index"}, "Item", "Value"),
#"Duplicated Column" = Table.DuplicateColumn(#"Unpivoted Columns", "Index", "Index - Copy"),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Duplicated Column", {{"Index - Copy", type text}}, "pt-BR"),{"Item", "Index - Copy"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Item"),
#"Merged Columns1" = Table.CombineColumns(Table.TransformColumnTypes(#"Merged Columns", {{"Index", type text}}, "pt-BR"),{"Action", "Index"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Action.1"),
Union =
Table.Combine({
Table.RenameColumns(Table.SelectColumns(#"Merged Columns1", {"Event Id", "Responder Id", "Name", "Response Time", "Action.1"}), {{"Response Time", "Value"}, {"Action.1", "Item"}}),
Table.Distinct(Table.SelectColumns(#"Merged Columns1", {"Event Id", "Responder Id", "Name", "Item", "Value"}), {"Event Id", "Item"})
}),
#"Pivoted Column" = Table.Pivot(Union, List.Distinct(Union[Item]), "Item", "Value")
in
#"Pivoted Column"
Just change the path in the beginning:
Source = Csv.Document(File.Contents("D:\Downloads\responders2.csv"),[Delimiter=",", Columns=6, Encoding=1252, QuoteStyle=QuoteStyle.None]),
The previous version was closer. Here is a link to the raw file and the final format desired.
https://www.dropbox.com/s/qp4g4xsim5yfh8r/Example.zip?dl=0
@Anonymous ,
I don't have this option. However paste this code on Edit Queries -> New Source -> Blank Query:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("xZNPC4IwGIe/iuwsOOc2zVvQhA79wYQO4iFsgYem5Ar69q06lPpqu3WYjO3x97y6d3mOxE0q7SyPyEWpbJ3s3kgznZe6qtV7ralVK52sOktUuDmiPAopM1uruXls1K6U6vkK9XyPYIIdFpMZQC5kc7hoeeyigf9BudnK9mOhLAJIKJTHLOiiWzOESuur7mUGdAiC8iAcgnbuxNadjLr5EITdpOcWtm5h+93C2v0683Wtq1M1OHMKoFZ1jrccxQAJtxz1/1sppLfrpbHLweFr9Ds0nKyUcYCE/+k3Gk2E8hhjgIQrxQQVxQM=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"(blank)" = _t, #"(blank).1" = _t, #"(blank).2" = _t, #"(blank).3" = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"(blank)", type text}, {"(blank).1", type text}, {"(blank).2", type text}, {"(blank).3", type text}}),
#"Promoted Headers" = Table.PromoteHeaders(#"Changed Type", [PromoteAllScalars=true]),
#"Changed Type1" = Table.TransformColumnTypes(#"Promoted Headers",{{"Event Id", Int64.Type}, {"Res Type", type text}, {"Action", type text}, {"Response Time", type datetime}}),
#"Grouped Rows" = Table.Group(#"Changed Type1", {"Event Id", "Action"}, {{"Rows", each Table.AddIndexColumn(_, "Index", 1,1), type table}}),
#"Removed Other Columns" = Table.SelectColumns(#"Grouped Rows",{"Rows"}),
#"Expanded Rows" = Table.ExpandTableColumn(#"Removed Other Columns", "Rows", {"Event Id", "Res Type", "Action", "Response Time", "Index"}, {"Event Id", "Res Type", "Action", "Response Time", "Index"}),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Expanded Rows", {"Event Id", "Action", "Response Time", "Index"}, "Attribute", "Value"),
#"Duplicated Column" = Table.DuplicateColumn(#"Unpivoted Columns", "Index", "Index - Copy"),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Duplicated Column", {{"Index - Copy", type text}}, "pt-BR"),{"Attribute", "Index - Copy"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"RespType"),
#"Merged Columns1" = Table.CombineColumns(Table.TransformColumnTypes(#"Merged Columns", {{"Index", type text}}, "pt-BR"),{"Action", "Index"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Action.1"),
Union =
Table.Combine({
Table.RenameColumns(Table.SelectColumns(#"Merged Columns1", {"Event Id", "Response Time", "Action.1"}), {{"Response Time", "Value"}, {"Action.1", "Item"}}),
Table.Distinct(Table.RenameColumns(Table.SelectColumns(#"Merged Columns1", {"Event Id", "RespType", "Value"}), {{"RespType", "Item"}}), {"Event Id", "Item"})
}),
#"Pivoted Column" = Table.Pivot(Union, List.Distinct(Union[Item]), "Item", "Value")
in
#"Pivoted Column"
That's awesome...you are a Super User! That worked exactly how I envisioned.
Only problem, and it's not something that you provided. I want to apply it to my actual raw data file but it contains more columns than I provided. I thought I would be able to expand it but the code is more complicated than I imagined. What would the code be for this format and where in the Steps would I copy/paste it? This file has been reduced to what will end up being a single row with the corresponding entries. Here's a dropbox link to the file.
https://www.dropbox.com/s/3shkz9p481vx57r/responders2.csv?dl=0
If this is asking too much feel free to say so...I would completely understand.
Would it be possible to save this version down to the 2.78 (Feb 2020) version. I have to go through my Admin to update to the newest version and it's likely to take a while to accomplish?
I'm not entirely sure. I was waiting to see if I could format my database in the manner I've mentioned. Bar chart showing the total MA entries to start and then see what other options I can develop.
Thank you for the suggestion, unfortunately, this isn't exactly what I was wanting to do. I want to take the raw data as multiple row entries and convert it to a single row entry.
Any other suggestions?
Absolutely. Hope this helps.
My raw data is exported in a vertical fashion where 1 Event ID will be entered into a row followed by the Res Type, Action, and Response time. So using Event ID 468745 has the 1st entry where the Action is OnScene and Response Time then a 2nd entry when the Action is Departed and Respone Time. Instead of having mulitple rows, I'd like to automatically change the format to the 2nd table where I have 1 row entry with the corresponding Res Type, Action, Respnose Time as Colunns.
Event Id | Res Type | Action | Response Time |
468745 | MA | OnScene | 4/1/2020 5:29 |
468745 | MA | Departed | 4/1/2020 5:31 |
468746 | TW | OnScene | 4/1/2020 5:58 |
468746 | TW | Departed | 4/1/2020 6:53 |
468746 | P | EnRoute | 4/1/2020 5:34 |
468746 | P | OnScene | 4/1/2020 5:37 |
468746 | P | Departed | 4/1/2020 6:53 |
468746 | F | EnRoute | 4/1/2020 5:34 |
468746 | F | OnScene | 4/1/2020 5:36 |
468746 | F | Departed | 4/1/2020 6:23 |
468746 | E | EnRoute | 4/1/2020 5:34 |
468746 | E | OnScene | 4/1/2020 5:37 |
468746 | E | Departed | 4/1/2020 6:23 |
468746 | MA | Notified | 4/1/2020 5:34 |
468746 | MA | EnRoute | 4/1/2020 5:34 |
468746 | MA | OnScene | 4/1/2020 5:40 |
468746 | MA | Departed | 4/1/2020 5:41 |
468746 | MA | Notified | 4/1/2020 5:34 |
468746 | MA | EnRoute | 4/1/2020 5:34 |
468746 | MA | OnScene | 4/1/2020 5:41 |
468746 | MA | Departed | 4/1/2020 6:53 |
468746 | TW | OnScene | 4/1/2020 6:31 |
468746 | TW | Departed | 4/1/2020 6:53 |
468747 | MA | OnScene | 4/1/2020 5:56 |
468747 | MA | Departed | 4/1/2020 5:56 |
468748 | MA | OnScene | 4/1/2020 6:00 |
468748 | MA | Departed | 4/1/2020 6:02 |
In this table I have the multiple row entries converted to 1 row entry with column headings representing the row entries. Using Event ID 468746 as an example, I have multiple Response Types, multiple entries of OnScene and Departed, etc. that I'm hoping to convert to this table format.
EventID | Resp1Type | Resp1Notified | Resp1EnRoute | Resp1OnScene | Resp1Departed | Resp2Type | Resp2Notified | Resp2EnRoute | Resp2OnScene | Resp2Departed | Resp3Type | Resp3Notified | Resp3EnRoute | Resp3OnScene | Resp3Departed | Resp4Type | Resp4Notified | Resp4EnRoute | Resp4OnScene | Resp4Departed | Resp5Type | Resp5Notified | Resp5EnRoute | Resp5OnScene | Resp5Departed | Resp6Type | Resp6Notified | Resp6EnRoute | Resp6OnScene | Resp6Departed | Resp7Type | Resp7Notified | Resp7EnRoute | Resp7OnScene | Resp7Departed |
468745 | MA | 4/1/2020 5:29 | 4/1/2020 5:31 | ||||||||||||||||||||||||||||||||
468746 | TW | 4/1/2020 5:58 | 4/1/2020 6:53 | P | 4/1/2020 5:34 | 4/1/2020 5:37 | 4/1/2020 6:53 | F | 4/1/2020 5:34 | 4/1/2020 5:36 | 4/1/2020 6:23 | E | 4/1/2020 5:34 | 4/1/2020 5:37 | 4/1/2020 6:23 | MA | 4/1/2020 5:34 | 4/1/2020 5:34 | 4/1/2020 5:40 | 4/1/2020 5:41 | MA | 4/1/2020 5:34 | 4/1/2020 5:34 | 4/1/2020 5:41 | 4/1/2020 6:53 | TW | 4/1/2020 6:31 | 4/1/2020 6:53 | |||||||
468747 | MA | 4/1/2020 5:56 | 4/1/2020 5:56 | ||||||||||||||||||||||||||||||||
468748 | MA | 4/1/2020 6:00 | 4/1/2020 6:02 |
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the April 2024 Power BI update to learn about new features.
User | Count |
---|---|
106 | |
93 | |
75 | |
62 | |
50 |
User | Count |
---|---|
147 | |
107 | |
105 | |
87 | |
61 |