Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
RMel
Frequent Visitor

How to run an R script transformation on an existing query?

Hi all, I am new to Power BI but fairly experienced with R.

 

I am training myself by working through the exercises in the Dashboard in a Day course here: https://powerbi.microsoft.com/en-us/diad/

 

I noticed there is an option in the Query Editor Transform bar to run an R script to transform data instead of/as well as using the GUI.

 

So I tried to create a very simple new column using the R script:

 

dataset$double_revenue <- dataset$Revenue*2

 

This resulted in the error message:

 

"Formula.Firewall: Query 'Sales' (step 'Run R Script') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination."

 

The automatically-generated M code for the query is as such (This is my first experience with M so do not know how to debug this):

 

let
Source = Csv.Document(File.Contents("C:\Users\mydata\Data\USSales\sales.csv"),[Delimiter=",", Columns=5, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"ProductID", Int64.Type}, {"Date", type date}, {"Zip", type text}, {"Units", Int64.Type}, {"Revenue", type number}}),
#"Appended Query" = Table.Combine({#"Changed Type", #"International Sales"}),
#"Added Conditional Column" = Table.AddColumn(#"Appended Query", "CountryName", each if [Country] = null then "USA" else [Country]),
#"Removed Columns" = Table.RemoveColumns(#"Added Conditional Column",{"Country"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"CountryName", "Country"}}),
#"Changed Type1" = Table.TransformColumnTypes(#"Renamed Columns",{{"Country", type text}, {"Revenue", Currency.Type}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type1", each true),
#"Filtered Rows1" = Table.SelectRows(#"Filtered Rows", each Date.IsInPreviousNYears([Date], 3)),
#"Run R Script" = R.Execute("# 'dataset' holds the input data for this script#(lf)#(lf)dataset$double_revenue <- dataset$Revenue*2",[dataset=#"Filtered Rows1"])
in
#"Run R Script"

 

I have read the information here for this error message: https://www.excelguru.ca/blog/2015/03/11/power-query-errors-please-rebuild-this-data-combination/ but this is specific to merges and I don't know how to apply it to the issue with this R script.

 

I am aware that the R script references the query; I wanted to use R code to transform a column in a query that I had already imported and transformed the data for rather than running all of the import and transformation steps in R (in this case I would have used Get Data -> R Script instead).

 

How do I get this query to work with the R code transform step in it?

8 REPLIES 8
Carmichael
Resolver I
Resolver I

@RMel I was having the same problem referenced here and couldnt for the life of me figure it out.  Found the following post which highlighted the solution (at least for me).  

 

By changing this setting on the file to ignore privacy levels it worked for me.  I am sure there are ramifications of publishing models with this option unchecked but for now I can at leaset continue with using R to perform necessary transformations on existing queries.

https://community.powerbi.com/t5/Desktop/Formula-Firewall-Query-references-other-queries-or-steps/td... 

dax
Community Support
Community Support

Hi @RMel , 

You could try below Code to see whether it work or not (ds[[1]] is the first column in dataset), this works well when in single dataset 

 

# 'dataset' holds the input data for this script


ds <- dataset

ds$progress <- ds[[1]]*100

dataset <- ds

 

 By the way, I have other error information(can't convert to number when I re-produce your design), I am not sure whether this is caused by query combine or others, and I will test this in my environment, I will inform you as soon as possible. 

Best Regards,
Zoe Zhi

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

RMel
Frequent Visitor

Hi @dax,

I tried this but unfortunately I am still getting the same error!

Thanks!

You need to return a dataframe back to the query.  Does this work?

 

dataset$double_revenue <- dataset$Revenue*2

dataset

 

If this works for you, please mark it as solution.  Kudos are appreciated too.  Please let me know if not.

Regards,

Pat





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


Hi @mahoneypat,

 

Unfortunately I am still getting the same error message with this R script too.

 

Thanks! 

My bad.  It's been a while.  You need to send the resulting dataframe to output

 

# 'dataset' holds the input data for this script
dataset$newcolumn <- dataset$Number * 2
output <- dataset

 

I did a test query, if you are interested.

let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTJUitWJVnICsozALGcgy1gpNhYA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Letter = _t, Number = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Letter", type text}, {"Number", Int64.Type}}),
#"Run R script" = R.Execute("# 'dataset' holds the input data for this script#(lf)dataset$newcolumn <- dataset$Number * 2#(lf)output <- dataset",[dataset=#"Changed Type"]),
#"""output""" = #"Run R script"{[Name="output"]}[Value]
in
#"""output"""

 

If this works for you, please mark it as solution.  Kudos are appreciated too.  Please let me know if not.

Regards,

Pat





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


Hi @mahoneypat,

 

I tried this and still got the same error. I wondered if this is because I merged this query with another one earlier, so I tried it with a different query that I had not merged previously instead.

 

This time I received a prompt saying "Information is required about data privacy."

 

I clicked "Ignore Privacy level checks for this file. Ignoring privacy level checks could expose sensitive or confidential data to an unauthorised person." and clicked "Save" (I wouldn't be able to do this for business data due to security issues but tried it for the purposes of testing.)

 

Then I received a different error message that starts with (I can't add the whole thing as it exceeds 20,000 characters):

 

Feedback Type:
Frown (Error)

Error Message:
Object reference not set to an instance of an object.

Stack Trace:
Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> System.NullReferenceException: Object reference not set to an instance of an object. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.PowerBI.Radio.RConnection.get_ServerVersion()
at Microsoft.Mashup.Engine1.Library.Common.DelegatingDbConnection.get_ServerVersion()
at Microsoft.Mashup.Engine1.Library.Common.DelegatingDbConnection.get_ServerVersion()
at Microsoft.Mashup.Engine1.Library.Common.DbEnvironment.LoadServerVersion()
at Microsoft.Mashup.Engine1.Library.Common.DbEnvironment.LoadCachedServerVersion()
at Microsoft.Mashup.Engine1.Library.Common.DbEnvironment.CreateConnection()
at Microsoft.Mashup.Engine1.Library.Common.DbValueBuilder.ConnectionReader..ctor(DbEnvironment environment, Func`2 readerWrapper)
at Microsoft.Mashup.Engine1.Library.Common.NativeQueryTableValue.Initialize()
at Microsoft.Mashup.Engine1.Library.Common.NativeQueryTableValue.EnsureInitialized()
at Microsoft.Mashup.Engine1.Library.Common.NativeQueryTableValue.GetEnumerator()
at Microsoft.Mashup.Engine1.Runtime.TableValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.TableValue.get_Item(Value key)
at Microsoft.Mashup.Engine1.Runtime.Library.List.ElementWithListCheckFunctionValue.Invoke(Value collection, Value key)
at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction1.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)
at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction1.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.Instruction.Execute(MembersFrame1& frame, Instruction[] instructions)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)
at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(Value frame)
at Microsoft.Mashup.Engine1.Language.IfInstruction.Execute(Value frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(Value frame)
at Microsoft.Mashup.Engine1.Language.RuntimeFunctionValue1.Invoke(Value arg0)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)
at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(MembersFrame2& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(MembersFrame2& frame)
at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame2& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
at Microsoft.Mashup.Engine1.Runtime.FunctionValue.MetaTypeFunctionValue.Invoke(Value arg0, Value arg1)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(MembersFrame2& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame2& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(MembersFrame2& frame)
at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame2& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
at Microsoft.Mashup.Engine1.Runtime.FunctionValue.Invoke(Value[] args)
at Microsoft.Mashup.Engine1.Runtime.FunctionValue.MetaTypeFunctionValue.Invoke(Value[] args)
at Microsoft.Mashup.Engine1.Runtime.Extensibility.HostRelinkingFunctionValue.ExtensionFunctionValue.InvokeN(Value[] args)
at Microsoft.Mashup.Engine1.Runtime.NativeFunctionValueN.Invoke(Value[] args)
at Microsoft.Mashup.Engine1.Runtime.NativeFunctionValueN.Invoke(Value arg0, Value arg1)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)
at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(Value key)
at Microsoft.Mashup.Engine1.Runtime.Library.Linker.BindFunctionValue.TypedInvoke(RecordValue environment, Value section, TextValue name)
at Microsoft.Mashup.Engine1.Runtime.NativeFunctionValue3`4.Invoke(Value arg0, Value arg1, Value arg2)
at Microsoft.Mashup.Engine1.Runtime.FunctionValue.Invoke(Value[] args)
at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction.Execute(MembersFrame0& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue0.Invoke()
at Microsoft.Mashup.Engine1.Language.ListInstruction.RuntimeListValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.ListInstruction.RuntimeListValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.ListValue.get_Item(Value key)
at Microsoft.Mashup.Engine1.Runtime.Library.List.ElementWithListCheckFunctionValue.Invoke(Value collection, Value key)
at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)
at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.Instruction.ExecuteCondition(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.ExecuteCondition(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.IfInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
at Microsoft.Mashup.Engine1.Runtime.RecordValue.get_Item(String field)
at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(MembersFrame0& frame)
at Microsoft.Mashup.Engine1.Language.MembersFunctionValue0.Invoke()
at Microsoft.Mashup.Engine1.Runtime.FunctionValue.Invoke(Value[] args)
at Microsoft.Mashup.Engine1.Engine.Microsoft.Mashup.Engine.Interface.IEngine.Invoke(IValue function, IValue[] arguments)
at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.BeginGetResult(DocumentEvaluationParameters parameters, Action`1 callback)
--- End of inner exception stack trace ---

 

Any ideas as to what this means?

Please see this post. 

http://officeusers.blogspot.com/2018/06/using-r-in-power-bi-object-reference.html

 

Are you using 32 bit R Studio?

 

Regards

Pat

 





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.