Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
dstanisljevic
Helper I
Helper I

Dynamic SQL Tables

I've been trying to find a method to automatically have my sql query add new tables to it. Our database generates a new table at the start of the month, resulting in having to update the project to now include that table. The table does end with "_YEARMO". Would it be possible to have a formula check for the first of the month and update the query to add a join for the new table?

1 ACCEPTED SOLUTION

You don't have to loop explicitely in Power Query: Adding a column to a table in Power Query will execute a function (that you define in the last argument) that will be applied to every row. You then just expand that new column (if a table or record with multiple fields hast been created)

 

I've used a generic date-function from my blog that is pretty cody, so don't be scared by the amount of code, but once pasted into the advanced editor, you'll see that it's just a good handful of steps resulting 😉

 

let

	DateFunction = let 
			// ----------------------- Documentation ----------------------- 
		documentation_ = [
		Documentation.Name =  " Dates.ListDateIntervals
		", Documentation.Description = " Creates a list of dates according to the chosen interval between Start and End. Allowed values for 3rd parameter: ""Year"", ""Quarter"", ""Month"", ""Week"" or ""Day"".
		" , Documentation.LongDescription = " Creates a list of dates according to the chosen interval between Start and End. The dates created will always be at the end of the interval, so could be in the future if today is chosen.
		", Documentation.Category = " Table
		", Documentation.Source = " http://www.thebiccountant.com/2017/12/11/date-datesbetween-retrieve-dates-between-2-dates-power-bi-power-query/ . 
		", Documentation.Author = " Imke Feldmann: www.TheBIccountant.com . 
		", Documentation.Examples = {[Description =  " see http://www.thebiccountant.com/2017/12/11/date-datesbetween-retrieve-dates-between-2-dates-power-bi-power-query/ .
		" , Code = " 
		 ", Result = " 
		"]}],

		// ----------------------- Function Code ----------------------- 
		function_ =  
		(From as date, To as date, optional Selection as text ) =>
		let
		// Create default-value "Day" if no selection for the 3rd parameter has been made
		TimeInterval = if Selection = null then "Day" else Selection,

		// Table with different values for each case
		CaseFunctions = #table({"Case", "LastDateInTI", "TypeOfAddedTI", "NumberOfAddedTIs"},
				{   {"Day", Date.From, Date.AddDays, Number.From(To-From)+1},
					{"Week", Date.EndOfWeek, Date.AddWeeks, Number.RoundUp((Number.From(To-From)+1)/7)},
					{"Month", Date.EndOfMonth, Date.AddMonths, (Date.Year(To)*12+Date.Month(To))-(Date.Year(From)*12+Date.Month(From))+1},
					{"Quarter", Date.EndOfQuarter, Date.AddQuarters, (Date.Year(To)*4+Date.QuarterOfYear(To))-(Date.Year(From)*4+Date.QuarterOfYear(From))+1},
					{"Year", Date.EndOfYear, Date.AddYears,Date.Year(To)-Date.Year(From)+1} 
				} ),

		// Filter table on selected case
				Case = CaseFunctions{[Case = TimeInterval]},
			
		// Create list with dates: List with number of date intervals -> Add number of intervals to From-parameter -> shift dates at the end of each respective interval	
			DateFunction = List.Transform({0..Case[NumberOfAddedTIs]-1}, each Function.Invoke(Case[LastDateInTI], {Function.Invoke(Case[TypeOfAddedTI], {From, _})}))
		in
			DateFunction,
		// ----------------------- New Function Type ----------------------- 
		type_ = type function (
			  From as (type date),
			  To as (type date),
			  optional Selection as (type text meta [
									Documentation.FieldCaption = "Select Date Interval",
									Documentation.FieldDescription = "Select Date Interval, if nothing selected, the default value will be ""Day""",
									Documentation.AllowedValues = {"Day", "Week", "Month", "Quarter", "Year"}
									])
				)
			as table meta documentation_,

		// Replace the extisting type of the function with the individually defined
		Result =  Value.ReplaceType(function_, type_)
	 in 
	Result,
    
	SQL_Function = (TableDate as text) =>

                    Sql.Database("localhost", "database", 
                    [Query="select RoomName,AssetName,CFV_SummarizedEnergyUsageByHour_" & TableDate & ".AssetID,CFV_SummarizedEnergyUsageByHour_" & TableDate & ".EnergyUsage,CFV_SummarizedEnergyUsageByHour_" & TableDate & ".LogTimeStamp
                    from CFV_SummarizedEnergyUsageByHour_" & TableDate & "
                    join CRV_Assets on CFV_SummarizedEnergyUsageByHour_" & TableDate & ".AssetID=CRV_Assets.AssetID
                    join CRV_Symbols on CRV_Assets.SymbolID=CRV_Symbols.SymbolID
                    join CRV_Rooms on CRV_Symbols.RoomID=CRV_Rooms.RoomID"]),


    ListOfDates = DateFunction(#date(2017,01,01), Date.From(DateTime.LocalNow()), "Month"),
    #"Converted to Table" = Table.FromList(ListOfDates, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    FormattedDate = Table.AddColumn(#"Converted to Table", "Date", each Date.Year([Column1])*100 + Date.Month([Column1])),
    ChgType = Table.TransformColumnTypes(FormattedDate,{{"Date", type text}}),
    CallSQLFunction = Table.AddColumn(ChgType, "ExecuteSQLFunction", each SQL_Function([Date]))
in
    CallSQLFunction

I've also converted your SQL-Call to a function so that it can be applied to every row. Just expand the resulting column.

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

View solution in original post

7 REPLIES 7
Greg_Deckler
Super User
Super User

In theory, you could have a function that does this and essentially returns a list of all of the tables that you want and another function or the main query itself just do a union/join on that list of table names. In theory. People have done similar things with web pages that essentially have an unspecified number of pages like ?page=1, ?page=2, etc. @ImkeF might be able to help out here.


@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
The Definitive Guide to Power Query (M)

DAX is easy, CALCULATE makes DAX hard...

Is my understanding correct, that the table name would end with _201804 for April?

 

Then you could create filter expression like this:

 

DateStartOfMonth = #date(Number.From( Text.Start( Text.End( FileName, 6), 4)) , Number.From( Text.End( FileName, 2)) ,1) 

 

that returns the date of the first day in April.

 

You could then compare it to the current day:

 

if DateStartOfMonth >= Date.From(DateTime.LocalNow()) then ...

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

@ImkeF That is correct, the April table would end in that fashion. What I'm struggling with is making that part of the select statement when querying the data from that table. I've figured out how to do it via parameters, but this only works when utilizing the desktop app.

That surprises me: What errror-message are you getting in the service?

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

OK, worked the variable part out and it was my error. Here's what I have that is working for a basic test:

 

let

    TableDate = "201804",

    Source = Sql.Database("localhost", "Database", [Query="select RoomName,RoomDescription,AssetName,CFV_RawEnergyUsage_"&TableDate&".AssetID,CFV_RawEnergyUsage_"&TableDate&".EnergyUsage,CFV_RawEnergyUsage_"&TableDate&".LogTimeStamp#(lf)from CFV_RawEnergyUsage_"&TableDate&"#(lf)join CRV_Assets on CFV_RawEnergyUsage_"&TableDate&".AssetID=CRV_Assets.AssetID#(lf)join CRV_Symbols on CRV_Assets.SymbolID=CRV_Symbols.SymbolID#(lf)join CRV_Rooms on CRV_Symbols.RoomID=CRV_Rooms.RoomID"])

in

    Source

 

So right now, just testing it as a fixed variable. This works and returns the table correctly. Where I'm struggling is what @Greg_Deckler suggested, looping through to bring in multiple tables. So as new tables are generated, bring there data in as well as ensuring that the past tables are in there as well.

Here is where I'm at. I'm very new to this so I'm assuming what I'm trying to do is more complex than my knowledge of BI. So I've been able to figure out how to calculate the date info I need into the correct format and then append the tables together into a table that gives me the data I need. Where I'm failing is understanding how to loop the process. Here's what I expect the process to be:

 

1) Determine start of month from current date. Grab data from that table.

2) Loop backwards and grab data from each table until the EndDate table is gathered.

3) End with all data into a single table.

 

Is this doable?

 

let
EndDate = "201701",
TableDate1 = Date.ToText(#date(Date.Year(DateTime.LocalNow()),Date.Month(DateTime.LocalNow()),Date.Day(DateTime.LocalNow())), "yyyyMM"),
TableDate2 = Date.ToText(#date(Date.Year(DateTime.LocalNow()),Date.Month(DateTime.LocalNow()) - 1,Date.Day(DateTime.LocalNow())), "yyyyMM"),
Source1 = Sql.Database("localhost", "database", [Query="select RoomName,AssetName,CFV_SummarizedEnergyUsageByHour_"&TableDate1&".AssetID,CFV_SummarizedEnergyUsageByHour_"&TableDate1&".EnergyUsage,CFV_SummarizedEnergyUsageByHour_"&TableDate1&".LogTimeStamp#(lf)from CFV_SummarizedEnergyUsageByHour_"&TableDate1&"#(lf)join CRV_Assets on CFV_SummarizedEnergyUsageByHour_"&TableDate1&".AssetID=CRV_Assets.AssetID#(lf)join CRV_Symbols on CRV_Assets.SymbolID=CRV_Symbols.SymbolID#(lf)join CRV_Rooms on CRV_Symbols.RoomID=CRV_Rooms.RoomID"]),
Source2 = Sql.Database("localhost", "database", [Query="select RoomName,AssetName,CFV_SummarizedEnergyUsageByHour_"&TableDate2&".AssetID,CFV_SummarizedEnergyUsageByHour_"&TableDate2&".EnergyUsage,CFV_SummarizedEnergyUsageByHour_"&TableDate2&".LogTimeStamp#(lf)from CFV_SummarizedEnergyUsageByHour_"&TableDate2&"#(lf)join CRV_Assets on CFV_SummarizedEnergyUsageByHour_"&TableDate2&".AssetID=CRV_Assets.AssetID#(lf)join CRV_Symbols on CRV_Assets.SymbolID=CRV_Symbols.SymbolID#(lf)join CRV_Rooms on CRV_Symbols.RoomID=CRV_Rooms.RoomID"]),
Data = Table.Combine({Source1,Source2})
in
Data

You don't have to loop explicitely in Power Query: Adding a column to a table in Power Query will execute a function (that you define in the last argument) that will be applied to every row. You then just expand that new column (if a table or record with multiple fields hast been created)

 

I've used a generic date-function from my blog that is pretty cody, so don't be scared by the amount of code, but once pasted into the advanced editor, you'll see that it's just a good handful of steps resulting 😉

 

let

	DateFunction = let 
			// ----------------------- Documentation ----------------------- 
		documentation_ = [
		Documentation.Name =  " Dates.ListDateIntervals
		", Documentation.Description = " Creates a list of dates according to the chosen interval between Start and End. Allowed values for 3rd parameter: ""Year"", ""Quarter"", ""Month"", ""Week"" or ""Day"".
		" , Documentation.LongDescription = " Creates a list of dates according to the chosen interval between Start and End. The dates created will always be at the end of the interval, so could be in the future if today is chosen.
		", Documentation.Category = " Table
		", Documentation.Source = " http://www.thebiccountant.com/2017/12/11/date-datesbetween-retrieve-dates-between-2-dates-power-bi-power-query/ . 
		", Documentation.Author = " Imke Feldmann: www.TheBIccountant.com . 
		", Documentation.Examples = {[Description =  " see http://www.thebiccountant.com/2017/12/11/date-datesbetween-retrieve-dates-between-2-dates-power-bi-power-query/ .
		" , Code = " 
		 ", Result = " 
		"]}],

		// ----------------------- Function Code ----------------------- 
		function_ =  
		(From as date, To as date, optional Selection as text ) =>
		let
		// Create default-value "Day" if no selection for the 3rd parameter has been made
		TimeInterval = if Selection = null then "Day" else Selection,

		// Table with different values for each case
		CaseFunctions = #table({"Case", "LastDateInTI", "TypeOfAddedTI", "NumberOfAddedTIs"},
				{   {"Day", Date.From, Date.AddDays, Number.From(To-From)+1},
					{"Week", Date.EndOfWeek, Date.AddWeeks, Number.RoundUp((Number.From(To-From)+1)/7)},
					{"Month", Date.EndOfMonth, Date.AddMonths, (Date.Year(To)*12+Date.Month(To))-(Date.Year(From)*12+Date.Month(From))+1},
					{"Quarter", Date.EndOfQuarter, Date.AddQuarters, (Date.Year(To)*4+Date.QuarterOfYear(To))-(Date.Year(From)*4+Date.QuarterOfYear(From))+1},
					{"Year", Date.EndOfYear, Date.AddYears,Date.Year(To)-Date.Year(From)+1} 
				} ),

		// Filter table on selected case
				Case = CaseFunctions{[Case = TimeInterval]},
			
		// Create list with dates: List with number of date intervals -> Add number of intervals to From-parameter -> shift dates at the end of each respective interval	
			DateFunction = List.Transform({0..Case[NumberOfAddedTIs]-1}, each Function.Invoke(Case[LastDateInTI], {Function.Invoke(Case[TypeOfAddedTI], {From, _})}))
		in
			DateFunction,
		// ----------------------- New Function Type ----------------------- 
		type_ = type function (
			  From as (type date),
			  To as (type date),
			  optional Selection as (type text meta [
									Documentation.FieldCaption = "Select Date Interval",
									Documentation.FieldDescription = "Select Date Interval, if nothing selected, the default value will be ""Day""",
									Documentation.AllowedValues = {"Day", "Week", "Month", "Quarter", "Year"}
									])
				)
			as table meta documentation_,

		// Replace the extisting type of the function with the individually defined
		Result =  Value.ReplaceType(function_, type_)
	 in 
	Result,
    
	SQL_Function = (TableDate as text) =>

                    Sql.Database("localhost", "database", 
                    [Query="select RoomName,AssetName,CFV_SummarizedEnergyUsageByHour_" & TableDate & ".AssetID,CFV_SummarizedEnergyUsageByHour_" & TableDate & ".EnergyUsage,CFV_SummarizedEnergyUsageByHour_" & TableDate & ".LogTimeStamp
                    from CFV_SummarizedEnergyUsageByHour_" & TableDate & "
                    join CRV_Assets on CFV_SummarizedEnergyUsageByHour_" & TableDate & ".AssetID=CRV_Assets.AssetID
                    join CRV_Symbols on CRV_Assets.SymbolID=CRV_Symbols.SymbolID
                    join CRV_Rooms on CRV_Symbols.RoomID=CRV_Rooms.RoomID"]),


    ListOfDates = DateFunction(#date(2017,01,01), Date.From(DateTime.LocalNow()), "Month"),
    #"Converted to Table" = Table.FromList(ListOfDates, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
    FormattedDate = Table.AddColumn(#"Converted to Table", "Date", each Date.Year([Column1])*100 + Date.Month([Column1])),
    ChgType = Table.TransformColumnTypes(FormattedDate,{{"Date", type text}}),
    CallSQLFunction = Table.AddColumn(ChgType, "ExecuteSQLFunction", each SQL_Function([Date]))
in
    CallSQLFunction

I've also converted your SQL-Call to a function so that it can be applied to every row. Just expand the resulting column.

 

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors