cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
umesh_ram
New Member

Loop through sql server table to read a set of data in each iteration and do unpivoting

Hi PowerBI experts,

 

I am getting data from Excel files (around 400 files from a folder location). These excel files have yyyymm values starts at the cell J28. And i need to unpivot all the 12 yyyymm cell values starting from J28, e,g 202001,202002, .. to 202012.So  So, basically i am skipping first 27 rows from sample excel file and then promote 28th row as column header and then apply unpivoting. All these steps i applied in transform query and defined as a function. Then i call this function as custom function in my main query to read all the excel files in similar manner.

 

Now as a performance tunning step, i have been asked to ETL all these excel files in SSIS and load into sql server tables.

I am bit struggling on data modelling. Can i load all excel files into a single target table and maintain sequence no dynamically while loading such as 1,2,3,.. so on. And then can i read data  from the target table filter by sequence no =1 and then do the unpivoting steps and then loop the same unpivoting steps for sequence no =2 for next excel file, so on. These may be achieved through  paramter in power query as a transofrm query and called in a main query to iterate through all sequence no.

 

I am new to M Language and not able to finding the way. Kindly please help me understand the options available in M Language to achieve this.

 

BR,

Umesh.

1 ACCEPTED SOLUTION
V-pazhen-msft
Community Support
Community Support

@umesh_ram 
There is a Query Parameter feature allows you to dynamically change the excel tables based on the sequence you selected. However, I am not sure whether you can bring the parameter together with Pbi dataset back to SQL. You may try export power bi dataset to SQL afterwards: Exporting Power BI Data to SQL Server (biinsight.com).

 

Please refer to my example for using parameter on excel files:

1.  Connect to the folder that has all excel files, OK=>Transform.

V-pazhen-msft_0-1617329537834.png

 

2. Add an index column from 1, right click the index column=>add as new query.

V-pazhen-msft_6-1617329949593.png

3. Create the a parameter based on the new query.

V-pazhen-msft_5-1617329917073.png

 

4. Once parameter is created, set it as filter to the index column. 

V-pazhen-msft_8-1617330159750.png

 

V-pazhen-msft_7-1617330094469.png

5. The final step is to expand the excel table, Close and Apply.

V-pazhen-msft_9-1617330296058.png

 

You then can dynamically change the displayed table data using parameter.

V-pazhen-msft_10-1617330721445.png

 


Paul Zheng _ Community Support Team
If this post helps, please Accept it as the solution to help the other members find it more quickly.

 

View solution in original post

2 REPLIES 2
V-pazhen-msft
Community Support
Community Support

@umesh_ram 
There is a Query Parameter feature allows you to dynamically change the excel tables based on the sequence you selected. However, I am not sure whether you can bring the parameter together with Pbi dataset back to SQL. You may try export power bi dataset to SQL afterwards: Exporting Power BI Data to SQL Server (biinsight.com).

 

Please refer to my example for using parameter on excel files:

1.  Connect to the folder that has all excel files, OK=>Transform.

V-pazhen-msft_0-1617329537834.png

 

2. Add an index column from 1, right click the index column=>add as new query.

V-pazhen-msft_6-1617329949593.png

3. Create the a parameter based on the new query.

V-pazhen-msft_5-1617329917073.png

 

4. Once parameter is created, set it as filter to the index column. 

V-pazhen-msft_8-1617330159750.png

 

V-pazhen-msft_7-1617330094469.png

5. The final step is to expand the excel table, Close and Apply.

V-pazhen-msft_9-1617330296058.png

 

You then can dynamically change the displayed table data using parameter.

V-pazhen-msft_10-1617330721445.png

 


Paul Zheng _ Community Support Team
If this post helps, please Accept it as the solution to help the other members find it more quickly.

 

View solution in original post

lbendlin
Super User III
Super User III

You have been asked to do that in SSIS.  Not sure what that has to do with M?

Helpful resources

Announcements
MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

July 2021 Update 768x460.png

Check it out!

Click here to read more about the July 2021 Updates

Power Query PA Forum 768x460.png

Check it out!

Did you know that you can visit the Power Query Forum in Power BI and now Power Apps

Top Solution Authors
Top Kudoed Authors