cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
Helper V
Helper V

Perform Query Steps on Excel file dynamically based on Dynamic Parameter

So i'm importing a bunch of Excel files from SharePoint. For some reason, the underlying data structure is a bit strange. Each Excel file has column headers starting on row 12. I have confirmed this by going into each file. However, when imported into Power BI some files have column headers starting on row 11 and some on row 12. These files are generated from an automated job. Some files have the sheet name as the date the file was created and some files just have Sheet1 as the date. This makes me think that some underlying data is in csv format if the sheet name has the date created instead of Sheet1?

 

That's the root of the problem.

 

My solution was to somehow develop a way in Power BI to look at each incoming Excel file and see if the column headers start on row 11 then perform query steps A and if they come in on row 12 then perform Query Steps B.

 

1.) I duplicated the transform sample file step and converted it to a list to identify which row the column headers are coming in at and assigned it as a parameter:

 

let
    Source = Excel.Workbook(Parameter2, null, true),
    FirstSheet = Table.SelectRows(Source, each [Kind] = "Sheet"){0}[Data],
    #"Removed Top Rows" = Table.Skip(FirstSheet,10),
    #"Duplicated Column" = Table.DuplicateColumn(#"Removed Top Rows", "Column1", "Column1 - Copy"),
    #"Replaced Value" = Table.ReplaceValue(#"Duplicated Column",null,"",Replacer.ReplaceValue,{"Column1 - Copy"}),
    #"Added Custom" = Table.AddColumn(#"Replaced Value", "Custom", each if Text.Contains([#"Column1 - Copy"], "Shift") then [#"Column1 - Copy"] else "Adjust"),
    #"Removed Columns1" = Table.RemoveColumns(#"Added Custom",{"Column1 - Copy"}),
    #"Filled Down" = Table.FillDown(#"Removed Columns1",{"Custom"}),
    #"Removed Other Columns" = Table.SelectColumns(#"Filled Down",{"Custom"}),
    #"Kept First Rows" = Table.FirstN(#"Removed Other Columns",1),
    Custom = #"Kept First Rows"[Custom]



in
    Custom

 Parameter:

"Shift" meta [IsParameterQuery=true, ExpressionIdentifier=#"Transform Sample File (3)", Type="Text", IsParameterQueryRequired=true]

 

 

2.) In the original transform Sample file, do conditional branching to do Query Steps A or B based on the parameter setup previously:

 

let
    Source = Excel.Workbook(Parameter2, null, true),
    FirstSheet = Table.SelectRows(Source, each [Kind] = "Sheet"){0}[Data],
    #"Removed Top Rows" = Table.Skip(FirstSheet,10),
    #"Duplicated Column" = Table.DuplicateColumn(#"Removed Top Rows", "Column1", "Column1 - Copy"),
    #"Replaced Value" = Table.ReplaceValue(#"Duplicated Column",null,"",Replacer.ReplaceValue,{"Column1 - Copy"}),
    #"Added Custom" = Table.AddColumn(#"Replaced Value", "Custom", each if Text.Contains([#"Column1 - Copy"], "Shift") then [#"Column1 - Copy"] else "Adjust"),
    #"Removed Columns1" = Table.RemoveColumns(#"Added Custom",{"Column1 - Copy"}),
    #"Filled Down" = Table.FillDown(#"Removed Columns1",{"Custom"}),
    //Adjust
    #"Removed Top Rows1000" = Table.Skip(#"Filled Down",1),
    #"Promoted Headers100" = Table.PromoteHeaders(#"Removed Top Rows1000", [PromoteAllScalars=true]),
    #"Changed Type100" = Table.TransformColumnTypes(#"Promoted Headers100",{{"Shift", Int64.Type}, {"Column2", type any}, {"EmpNbr", Int64.Type}, {"Name", type text}, {"Column5", type any}, {"Column6", type any}, {"Extn", type text}, {"Area", type text}, {"Location", type text}, {"Duration", Int64.Type}, {"Arvl", Int64.Type}, {"Lunch", Int64.Type}, {"Dptr", Int64.Type}, {"Total", Int64.Type}, {"Column15", type text}}),
    #"Removed Top Rows100" = Table.Skip(#"Changed Type100",2),
    #"Removed Columns100" = Table.RemoveColumns(#"Removed Top Rows100",{"Shift", "Column2", "Column5", "Column6", "Column15"}),
    #"Removed Blank Rows100" = Table.SelectRows(#"Removed Columns100", each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))),
    #"Adjust Result" = #"Removed Blank Rows100",

    //Shift No adjust
    #"Promoted Headers" = Table.PromoteHeaders(#"Filled Down", [PromoteAllScalars=true]),
    #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Shift", Int64.Type}, {"Column2", type any}, {"EmpNbr", Int64.Type}, {"Name", type text}, {"Column5", type any}, {"Column6", type any}, {"Extn", type text}, {"Area", type text}, {"Location", type text}, {"Duration", Int64.Type}, {"Arvl", Int64.Type}, {"Lunch", Int64.Type}, {"Dptr", Int64.Type}, {"Total", Int64.Type}, {"Column15", type text}}),
    #"Removed Top Rows1" = Table.Skip(#"Changed Type",2),
    #"Removed Columns" = Table.RemoveColumns(#"Removed Top Rows1",{"Shift", "Column2", "Column5", "Column6", "Column15"}),
    #"Removed Blank Rows" = Table.SelectRows(#"Removed Columns", each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))),
    #"Shift Result" = #"Removed Blank Rows",

    result =
        if ManufacturingTeam= "Adjust"
        then #"Adjust Result"
        else #"Shift Result",
    #"Removed Columns2" = Table.RemoveColumns(result,{"Shift_1"})



in
    #"Removed Columns2"

 

Added this also to the transform file function:

 

let
    Source = (Parameter2 as binary, ManufacturingTeam as text) => let
        Source = Excel.Workbook(Parameter2, null, true),
        FirstSheet = Table.SelectRows(Source, each [Kind] = "Sheet"){0}[Data],
        #"Removed Top Rows" = Table.Skip(FirstSheet,10),
        #"Duplicated Column" = Table.DuplicateColumn(#"Removed Top Rows", "Column1", "Column1 - Copy"),
        #"Replaced Value" = Table.ReplaceValue(#"Duplicated Column",null,"",Replacer.ReplaceValue,{"Column1 - Copy"}),
        #"Added Custom" = Table.AddColumn(#"Replaced Value", "Custom", each if Text.Contains([#"Column1 - Copy"], "Shift") then [#"Column1 - Copy"] else "Adjust"),
        #"Removed Columns1" = Table.RemoveColumns(#"Added Custom",{"Column1 - Copy"}),
        #"Filled Down" = Table.FillDown(#"Removed Columns1",{"Custom"}),
        //Adjust
        #"Removed Top Rows1000" = Table.Skip(#"Filled Down",1),
        #"Promoted Headers100" = Table.PromoteHeaders(#"Removed Top Rows1000", [PromoteAllScalars=true]),
        #"Changed Type100" = Table.TransformColumnTypes(#"Promoted Headers100",{{"Shift", Int64.Type}, {"Column2", type any}, {"EmpNbr", Int64.Type}, {"Name", type text}, {"Column5", type any}, {"Column6", type any}, {"Extn", type text}, {"Area", type text}, {"Location", type text}, {"Duration", Int64.Type}, {"Arvl", Int64.Type}, {"Lunch", Int64.Type}, {"Dptr", Int64.Type}, {"Total", Int64.Type}, {"Column15", type text}}),
        #"Removed Top Rows100" = Table.Skip(#"Changed Type100",2),
        #"Removed Columns100" = Table.RemoveColumns(#"Removed Top Rows100",{"Shift", "Column2", "Column5", "Column6", "Column15"}),
        #"Removed Blank Rows100" = Table.SelectRows(#"Removed Columns100", each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))),
        #"Adjust Result" = #"Removed Blank Rows100",
    
        //Shift No adjust
        #"Promoted Headers" = Table.PromoteHeaders(#"Filled Down", [PromoteAllScalars=true]),
        #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"Shift", Int64.Type}, {"Column2", type any}, {"EmpNbr", Int64.Type}, {"Name", type text}, {"Column5", type any}, {"Column6", type any}, {"Extn", type text}, {"Area", type text}, {"Location", type text}, {"Duration", Int64.Type}, {"Arvl", Int64.Type}, {"Lunch", Int64.Type}, {"Dptr", Int64.Type}, {"Total", Int64.Type}, {"Column15", type text}}),
        #"Removed Top Rows1" = Table.Skip(#"Changed Type",2),
        #"Removed Columns" = Table.RemoveColumns(#"Removed Top Rows1",{"Shift", "Column2", "Column5", "Column6", "Column15"}),
        #"Removed Blank Rows" = Table.SelectRows(#"Removed Columns", each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))),
        #"Shift Result" = #"Removed Blank Rows",
    
        result =
            if ManufacturingTeam= "Adjust"
            then #"Adjust Result"
            else #"Shift Result",
        #"Removed Columns2" = Table.RemoveColumns(result,{"Shift_1"})
    
    
    
    in
        #"Removed Columns2"
in
    Source

 

Now when I let this run in my original query, it breaks at the invoking of the function saying an error that 1 arguments were passed to a function which expects 2.

 

I think I have the groundwork set for what I need, but I think I'm not doing the steps in the correct order which is causing the misfiring.

 

Anyone have any ideas?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted
Anonymous
Not applicable

Re: Perform Query Steps on Excel file dynamically based on Dynamic Parameter

Hello @nobodyukno , check out this blog article by PowerPivotPro.  This should work for you.

 

The idea is to find the value in a certain "cell"...and consider that the beginning of the table.  In your case it's either A11 or A12.  Note that P3's solution only works for 16k rows.  This is because the solution "transposes the table" (swaps columns and rows), and Power BI can only support 16k columns.

 

I wrote a comment midway through that addresses the 16k row limit:  search for this text to jump to my comment:  

 

 

let
Source =
,Starting_Value = “Label” // change as needed

// these first 3 steps are the same as Mike Ward’s solution
,Add_HeaderCheck_ROW = Table.AddColumn(Source, “Wanted”, each
List.Contains(Record.FieldValues(_), Starting_Value)),
Skip_RowsAboveHeader_ROW = Table.Skip(Add_HeaderCheck_ROW, each
[Wanted] true)
,Remove_HeaderCheck_ROW = Table.RemoveColumns(Skip_RowsAboveHeader_ROW
,{“Wanted”})

// keep only the first row to avoid transpose errors for large tables
,FirstRow_forColumns = Table.FirstN(Remove_HeaderCheck_ROW, 1)
,Transpose = Table.Transpose(FirstRow_forColumns)

// these 2 steps are the same as Mike Ward’s solution
,Add_HeaderCheck_COL = Table.AddColumn(Transpose, “Wanted”, each
List.Contains(Record.FieldValues(_), Starting_Value))
,Skip_RowsAboveHeader_COL = Table.Skip(Add_HeaderCheck_COL, each
[Wanted] true)

// count how many rows the previous step removed…this is how many columns to remove from the un-transposed table
,Rows_Removed = Table.RowCount(Add_HeaderCheck_COL) – Table.RowCount(Skip_RowsAboveHeader_COL)

// uses Datachant’s method of removing the first N columns from a table…without referencing column names
,Remove_NColumns = Table.RemoveColumns(Remove_HeaderCheck_ROW
,List.FirstN( Table.ColumnNames(Remove_HeaderCheck_ROW), Rows_Removed) )
,Headers = Table.PromoteHeaders(Remove_NColumns, [PromoteAllScalars=true])
in
Headers

 

 

I also converted this M code into a function that you can call for every table.  If you had each file as a separate row, you could do a Table.AddColumn() and create the "proper" table.  I need to dig up that code.  Let me know if you need it.

 

Cheers,

~ Chris Haas

http://www.precisiondax.com

 

View solution in original post

1 REPLY 1
Highlighted
Anonymous
Not applicable

Re: Perform Query Steps on Excel file dynamically based on Dynamic Parameter

Hello @nobodyukno , check out this blog article by PowerPivotPro.  This should work for you.

 

The idea is to find the value in a certain "cell"...and consider that the beginning of the table.  In your case it's either A11 or A12.  Note that P3's solution only works for 16k rows.  This is because the solution "transposes the table" (swaps columns and rows), and Power BI can only support 16k columns.

 

I wrote a comment midway through that addresses the 16k row limit:  search for this text to jump to my comment:  

 

 

let
Source =
,Starting_Value = “Label” // change as needed

// these first 3 steps are the same as Mike Ward’s solution
,Add_HeaderCheck_ROW = Table.AddColumn(Source, “Wanted”, each
List.Contains(Record.FieldValues(_), Starting_Value)),
Skip_RowsAboveHeader_ROW = Table.Skip(Add_HeaderCheck_ROW, each
[Wanted] true)
,Remove_HeaderCheck_ROW = Table.RemoveColumns(Skip_RowsAboveHeader_ROW
,{“Wanted”})

// keep only the first row to avoid transpose errors for large tables
,FirstRow_forColumns = Table.FirstN(Remove_HeaderCheck_ROW, 1)
,Transpose = Table.Transpose(FirstRow_forColumns)

// these 2 steps are the same as Mike Ward’s solution
,Add_HeaderCheck_COL = Table.AddColumn(Transpose, “Wanted”, each
List.Contains(Record.FieldValues(_), Starting_Value))
,Skip_RowsAboveHeader_COL = Table.Skip(Add_HeaderCheck_COL, each
[Wanted] true)

// count how many rows the previous step removed…this is how many columns to remove from the un-transposed table
,Rows_Removed = Table.RowCount(Add_HeaderCheck_COL) – Table.RowCount(Skip_RowsAboveHeader_COL)

// uses Datachant’s method of removing the first N columns from a table…without referencing column names
,Remove_NColumns = Table.RemoveColumns(Remove_HeaderCheck_ROW
,List.FirstN( Table.ColumnNames(Remove_HeaderCheck_ROW), Rows_Removed) )
,Headers = Table.PromoteHeaders(Remove_NColumns, [PromoteAllScalars=true])
in
Headers

 

 

I also converted this M code into a function that you can call for every table.  If you had each file as a separate row, you could do a Table.AddColumn() and create the "proper" table.  I need to dig up that code.  Let me know if you need it.

 

Cheers,

~ Chris Haas

http://www.precisiondax.com

 

View solution in original post

Helpful resources

Announcements
Community Blog

Community Blog

Visit our Community Blog for articles, guides, and information created by fellow community members.

Using the Community

Using the Community

Need help with the Power BI Community? Our 'Using the Community' support articles are a great place to start.

Community Summit North America

Community Summit North America

Innovate, Collaborate, Grow. The top training and networking event across the globe for Microsoft Business Applications

Power Platform 2020 release wave 2 plan

Power Platform 2020 release wave 2 plan

Features releasing from October 2020 through March 2021

Top Solution Authors
Top Kudoed Authors