Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
johnlhaase
Helper I
Helper I

Grouping Date sets within a month

Hello

 

I work for an airline and I trying to group On days together to get the Min and Max Dates within each date set. The problem is there are more than one On/Off days within each month. So, just going for the Max/Min does not work since there can be several of these sets in a month. The trick of even doing Max-1 does not work due to multiple sets. I included part of the data set below and I have almost 200,000 rows of data. I am not an expert writer of M Code or Dax just a beiginner but I get the concepts. I am pretty good with use the GUI and working with If statements etc. plus I have been working with Power Quey Power BI for 3 years.

 

 

StartofMonth  EmpID    Fleet    On/Off   RotationDate

1/1/2020375B744 1/1/2020
1/1/2020375B744 1/2/2020
1/1/2020375B744 1/3/2020
1/1/2020375B744 1/4/2020
1/1/2020375B744 1/5/2020
1/1/2020375B744 1/6/2020
1/1/2020375B744 1/7/2020
1/1/2020375B744 1/8/2020
1/1/2020375B744 1/9/2020
1/1/2020375B744 1/10/2020
1/1/2020375B744 1/11/2020
1/1/2020375B744On1/12/2020
1/1/2020375B744On1/13/2020
1/1/2020375B744On1/14/2020
1/1/2020375B744On1/15/2020
1/1/2020375B744On1/16/2020
1/1/2020375B744On1/17/2020
1/1/2020375B744On1/18/2020
1/1/2020375B744On1/19/2020
1/1/2020375B744On1/20/2020
1/1/2020375B744On1/21/2020
1/1/2020375B744On1/22/2020
1/1/2020375B744On1/23/2020
1/1/2020375B744On1/24/2020
1/1/2020375B744On1/25/2020
1/1/2020375B744On1/26/2020
1/1/2020375B744On1/27/2020
1/1/2020375B744 1/28/2020
1/1/2020375B744 1/29/2020
1/1/2020375B744 1/30/2020
1/1/2020375B744On1/31/2020
2/1/2020375B744On2/1/2020
2/1/2020375B744On2/2/2020
2/1/2020375B744On2/3/2020
2/1/2020375B744On2/4/2020
2 ACCEPTED SOLUTIONS
Jimmy801
Community Champion
Community Champion

Hello @johnlhaase 

 

check out this solution. I hope I got you right. The group-function with GroupKind.Local does the trick here

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jdOxDYMwGEThXVwj2b7f4KTNAhkAsQL7l1iCQ1TJkyy78Nc93bqmmmtWUUlTij6P+9NbG88499c2/XGCLqBr0M3QLdB16F7QvaGrhUJQ5LufFEQxBV1MQRpTUMcUBDIFjUxBJlNQ6qICrUx5LfFa4rXEa4nXEq8luijRSYluKnineHTSL6rMpbAMLJ1zOwA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Start of Month" = _t, EmpID = _t, Fleet = _t, OnOff = _t, #"Rotation Date" = _t]),
    TransformDate = Table.TransformColumns(Source,{{"Rotation Date", each Date.From(_,"en-US"), type date},{"Start of Month", each Date.From(_,"en-US"), type date}}),
    Group = Table.Group(TransformDate, {"EmpID", "OnOff"}, {{"AllRows", each _, type table [Start of Month=date, EmpID=text, Fleet=text, OnOff=text, Rotation Date=date]}}, GroupKind.Local),
    AddMin = Table.AddColumn(Group, "Min", each List.Min([AllRows][Rotation Date]), type date),
    AddMax = Table.AddColumn(AddMin, "Max", each List.Max([AllRows][Rotation Date]), type date),
    ExpandGroupedTable = Table.ExpandTableColumn(AddMax, "AllRows", {"Start of Month", "Fleet", "Rotation Date"}, {"Start of Month", "Fleet", "Rotation Date"})

in
    ExpandGroupedTable

 

Copy paste this code to the advanced editor in a new blank query to see how the solution works.

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

View solution in original post

Hello

 

You are correct. I did ask about the same month. This is a large data set and when I was going through the restlts I realized some of the On/Off days transition into the following month. 

 

The first solution is fantasitic and is very useful. I wish I realized the data set had the transition issue priro to my first post. The law of unintended consequences. Anyways I am trying to get the scripts to ignore the start of month all together and just get the min/max for each On/Off sets.

 

Thanks for your time

 

John

View solution in original post

11 REPLIES 11
Jimmy801
Community Champion
Community Champion

Hello @johnlhaase 

 

check out this solution. I hope I got you right. The group-function with GroupKind.Local does the trick here

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jdOxDYMwGEThXVwj2b7f4KTNAhkAsQL7l1iCQ1TJkyy78Nc93bqmmmtWUUlTij6P+9NbG88499c2/XGCLqBr0M3QLdB16F7QvaGrhUJQ5LufFEQxBV1MQRpTUMcUBDIFjUxBJlNQ6qICrUx5LfFa4rXEa4nXEq8luijRSYluKnineHTSL6rMpbAMLJ1zOwA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Start of Month" = _t, EmpID = _t, Fleet = _t, OnOff = _t, #"Rotation Date" = _t]),
    TransformDate = Table.TransformColumns(Source,{{"Rotation Date", each Date.From(_,"en-US"), type date},{"Start of Month", each Date.From(_,"en-US"), type date}}),
    Group = Table.Group(TransformDate, {"EmpID", "OnOff"}, {{"AllRows", each _, type table [Start of Month=date, EmpID=text, Fleet=text, OnOff=text, Rotation Date=date]}}, GroupKind.Local),
    AddMin = Table.AddColumn(Group, "Min", each List.Min([AllRows][Rotation Date]), type date),
    AddMax = Table.AddColumn(AddMin, "Max", each List.Max([AllRows][Rotation Date]), type date),
    ExpandGroupedTable = Table.ExpandTableColumn(AddMax, "AllRows", {"Start of Month", "Fleet", "Rotation Date"}, {"Start of Month", "Fleet", "Rotation Date"})

in
    ExpandGroupedTable

 

Copy paste this code to the advanced editor in a new blank query to see how the solution works.

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

Hello

 

The solution works. I cut and pasted my full data set into the solution. Now we have start and end dates to fill in other activity to our model. Thanks a bunck.

 

John Haase

Hello @johnlhaase 

 

as you asked to take in consideration the change of month, i just added this column when grouping. I hope I got your right. If yes, the solution would look like this

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jdOxDYMwGEThXVwj2b7f4KTNAhkAsQL7l1iCQ1TJkyy78Nc93bqmmmtWUUlTij6P+9NbG88499c2/XGCLqBr0M3QLdB16F7QvaGrhUJQ5LufFEQxBV1MQRpTUMcUBDIFjUxBJlNQ6qICrUx5LfFa4rXEa4nXEq8luijRSYluKnineHTSL6rMpbAMLJ1zOwA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Start of Month" = _t, EmpID = _t, Fleet = _t, OnOff = _t, #"Rotation Date" = _t]),
    TransformDate = Table.TransformColumns(Source,{{"Rotation Date", each Date.From(_,"en-US"), type date},{"Start of Month", each Date.From(_,"en-US"), type date}}),
    Group = Table.Group(TransformDate, {"Start of Month","EmpID", "OnOff"}, {{"AllRows", each _, type table [Start of Month=date, EmpID=text, Fleet=text, OnOff=text, Rotation Date=date]}}, GroupKind.Local),
    AddMin = Table.AddColumn(Group, "Min", each List.Min([AllRows][Rotation Date]), type date),
    AddMax = Table.AddColumn(AddMin, "Max", each List.Max([AllRows][Rotation Date]), type date),
    ExpandGroupedTable = Table.ExpandTableColumn(AddMax, "AllRows", { "Fleet", "Rotation Date"}, { "Fleet", "Rotation Date"})

in
    ExpandGroupedTable

 

Copy paste this code to the advanced editor in a new blank query to see how the solution works.

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

Hello

 

Thanks again for the help! Below are the results of the last run which starts the min max over when march starts. I treid removing the start of the month hoping that will keep the Min/Max from restarting in a new month. So with the example below the Min would show 2/28/2020 and the max would show 3/18.

 

Dates are a funny data type even though they are really just a number based on 1/1/1900. However, the rules seem to be a little different.

 

 

 

Start of MonthEmpIDOnOffFleetRotation DateMinMax

2/1/20201073 B7442/27/20202/25/20202/27/2020
2/1/20201073OnB7442/28/20202/28/20202/29/2020
2/1/20201073OnB7442/29/20202/28/20202/29/2020
3/1/20201073OnB7443/1/20203/1/20203/18/2020
3/1/20201073OnB7443/2/20203/1/20203/18/2020
3/1/20201073OnB7443/3/20203/1/20203/18/2020
3/1/20201073OnB7443/4/20203/1/20203/18/2020
3/1/20201073OnB7443/5/20203/1/20203/18/2020
3/1/20201073OnB7443/6/20203/1/20203/18/2020
3/1/20201073OnB7443/7/20203/1/20203/18/2020
3/1/20201073OnB7443/8/20203/1/20203/18/2020
3/1/20201073OnB7443/9/20203/1/20203/18/2020
3/1/20201073OnB7443/10/20203/1/20203/18/2020
3/1/20201073OnB7443/11/20203/1/20203/18/2020
3/1/20201073OnB7443/12/20203/1/20203/18/2020
3/1/20201073OnB7443/13/20203/1/20203/18/2020
3/1/20201073OnB7443/14/20203/1/20203/18/2020
3/1/20201073OnB7443/15/20203/1/20203/18/2020
3/1/20201073OnB7443/16/20203/1/20203/18/2020
3/1/20201073OnB7443/17/20203/1/20203/18/2020
3/1/20201073OnB7443/18/20203/1/20203/18/2020

Hello @johnlhaase 

 

help me.... didn't you ask me to separate months with min/max? Maybe I got you wrong. But this result you should achieve with my first code

Jimmy801_0-1603085444746.png

and it's the 28.02, because this is the date when the on starts.

 

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

 

Hello

 

You are correct. I did ask about the same month. This is a large data set and when I was going through the restlts I realized some of the On/Off days transition into the following month. 

 

The first solution is fantasitic and is very useful. I wish I realized the data set had the transition issue priro to my first post. The law of unintended consequences. Anyways I am trying to get the scripts to ignore the start of month all together and just get the min/max for each On/Off sets.

 

Thanks for your time

 

John

I keep forgetting about the GroupKind.Local parameter @Jimmy801. 👍
Handy in some scenarios like this.



Did I answer your question? Mark my post as a solution!
Did my answers help arrive at a solution? Give it a kudos by clicking the Thumbs Up!

DAX is for Analysis. Power Query is for Data Modeling


Proud to be a Super User!

MCSA: BI Reporting
edhans
Super User
Super User

Hi @johnlhaase - See if this works:

 

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jdOxDYMwGEThXVwj2b7f4KTNAhkAsQL7l1iCQ1TJkyy78Nc93bqmmmtWUUlTij6P+9NbG88499c2/XGCLqBr0M3QLdB16F7QvaGrhUJQ5LufFEQxBV1MQRpTUMcUBDIFjUxBJlNQ6qICrUx5LfFa4rXEa4nXEq8luijRSYluKnineHTSL6rMpbAMLJ1zOwA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Start of Month" = _t, EmpID = _t, Fleet = _t, OnOff = _t, #"Rotation Date" = _t]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"Start of Month", type date}, {"Rotation Date", type date}}),
    FakeNullRow =
        {null} &
        List.RemoveLastN(
            Table.Column(
                Source, "OnOff"
                ),
                1
            ),
    CombinedWithNullColumn = Table.ToColumns(#"Changed Type") & {FakeNullRow},
    BackToTable = Table.FromColumns(CombinedWithNullColumn, Table.ColumnNames(#"Changed Type") & {"Previous OnOff Status"}),
    #"Added Changed Status" = Table.AddColumn(BackToTable, "Changed Status", each if [Previous OnOff Status] = null then null else if [OnOff] = [Previous OnOff Status] then null else "Changed"),
    #"Added Index" = Table.AddIndexColumn(#"Added Changed Status", "Index", 0, 1, Int64.Type),
    #"Added Grouping" = Table.AddColumn(#"Added Index", "Grouping", each if [Changed Status] = "Changed" then [Index] else null),
    #"Filled Down" = Table.FillDown(#"Added Grouping",{"Grouping"}),
    #"Grouped Rows" = Table.Group(#"Filled Down", {"Grouping"}, {{"AllRows", each _, type table [Start of Month=nullable date, EmpID=nullable text, Fleet=nullable text, OnOff=nullable text, Rotation Date=nullable date, Previous OnOff Status=nullable text, Changed Status=nullable text, Index=number, Grouping=nullable number]}}),
    #"Added Min Date" = Table.AddColumn(#"Grouped Rows", "Min Date", each Table.Min([AllRows], "Rotation Date")[Rotation Date]),
    #"Added Max Date" = Table.AddColumn(#"Added Min Date", "Max Date", each Table.Max([AllRows], "Rotation Date")[Rotation Date]),
    #"Expanded AllRows" = Table.ExpandTableColumn(#"Added Max Date", "AllRows", {"Start of Month", "EmpID", "Fleet", "OnOff", "Rotation Date"}, {"Start of Month", "EmpID", "Fleet", "OnOff", "Rotation Date"}),
    #"Removed Other Columns" = Table.SelectColumns(#"Expanded AllRows",{"Start of Month", "EmpID", "Fleet", "OnOff", "Rotation Date", "Min Date", "Max Date"})
in
    #"Removed Other Columns"

 

It turns this:

edhans_0-1602539581027.png

into this:

edhans_1-1602539611756.png

I did it by using some cool techniques from @ImkeF at this blog article, but instead of using a custom function, I just did it in the code.

  1. It takes your OnOff column and adds it to the end of the current table as a new column, but shifted by one row. 
  2. Adds a "Changed" text field to see if there is a change from on to blank (off I presume.)
  3. Adds an index. I just needed a sequence of numbers.
  4. If it is "Changed" give me the number in the Index column, otherwise, give me a null
  5. I then filled this column down. Now at every change, I have a different number.
  6. Grouped by that number, then used Table.Min() and Table.Max() to extract the min/max dates, then re-expanded your original columns.

Everything is UI driven (or in the Add Custom Column box) except for three steps:

  • FakeNullRow
  • CombinedWithNullColumn
  • BackToTable

You can see what each step is doing though by looking at it as you walk through it.

 

How to use M code provided in a blank query:
1) In Power Query, select New Source, then Blank Query
2) On the Home ribbon, select "Advanced Editor" button
3) Remove everything you see, then paste the M code I've given you in that box.
4) Press Done
5) See this article if you need help using this M code in your model.

 

 



Did I answer your question? Mark my post as a solution!
Did my answers help arrive at a solution? Give it a kudos by clicking the Thumbs Up!

DAX is for Analysis. Power Query is for Data Modeling


Proud to be a Super User!

MCSA: BI Reporting

Hello

 

Looks good. The only thing I cannot follow is the firat part with Compress and Deflate with all the letters in between. What does that do for the script?

 

Thanks again

 

John

Hello @johnlhaase 

 

this is to reproduce your dataset. You have to replace it with your datasource, or with the query already in place

 

If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too

Have fun

Jimmy

Thank you I will check it out!

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors