Dataflow - How can I stop automatic steps being added

[mpjtaylor]

I have a Datalake with multiple folders with lots of blobs that are JSON files each folder has different json structures and some are lists and others are records. I am trying to use a single dataflow to read in the content so that I can use subsequent dataflows to process specific folders and json files. The reason for this is to have a single datalake scheduled read transaction to get all the raw data.

The problem I have is I cannot find a way to save the dataflow with either the binary column or the Json [Content] column as they are automatically removed with the addtion of these frustrating steps which I want to prevent:

  #"Transform columns" = Table.TransformColumnTypes(JsonDocument, {{"JsonDocument", type text}}),

  #"Replace errors" = Table.ReplaceErrorValues(#"Transform columns", {{"JsonDocument", null}}),

  #"Remove columns" = Table.RemoveColumns(#"Replace errors", Table.ColumnsOfType(#"Replace errors", {type table, type record, type list, type nullable binary, type binary, type function}))

I have tried changing the Global and project settings but nothing I do seems to stop these being automatically added,,also tried changing the column types, duplicating the columns

How can this be achieved I just want the steps to stop at:

let

  Source = AzureStorage.Blobs("storageaccount"),

  Navigation = Source{[Name = "data"]}[Data],

  #"Filtered rows" = Table.SelectRows(Navigation, each [Extension] = ".json"),

  ChooseColumns = Table.SelectColumns(#"Filtered rows", {"Content", "Name"}),

  JsonDocument = Table.AddColumn(ChooseColumns, "JsonDocument", each Json.Document([Content]))

in

JsonDocument

Kind Regards

Marcus

[KBENN]

I'm facing exactly the same issue here. Anyone have a solution for it ?

Thanks a lot !

[yyr6]

Same issue happening for me. I keep deleting the steps, but they keep reappearing, and when pulling into Desktop, I cannot make the changes I need because of the dataflow limitations. Any update?

[SMHolck]

I may have found a way around the one that shows a fX (function) symbol and says "Remove Columns" ... 

Table.RemoveColumns(#"Previous step name", Table.ColumnsOfType(#"Removed columns 1", {type table, type record, type list, type nullable binary, type binary, type function}))
The PowerQuery editor for Dataflows seems to be adding this in an attempt to get rid of columns where every current value is null.  However, I want it to retain the column because I am writing the code ahead of the data - as in the system doesn't yet have values in it yet, but will in the future.
 
Try explicitly setting the type of the columns that you want to keep.  My best guess is that it was implicitly converting my column to the nullable binary.  But by explicitly setting the type in the previous step, it finally kept my column.  It was a little funky in my case because that column was explicitly set in the original query which was merged with my final table, so not sure why I had to tell PowerQuery the column type twice.  

[ChristianRochau]

This worked for me! thanks 🙂

[v-stephen-msft]

Hi @mpjtaylor ,

To my knowledge, your requirement about locking applied steps could not be realized currently.
But this is at idea/feedback level alone. May be ,hopes Power Bi will implement this in nearby future.

Best Regards,

Stephen Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[v-stephen-msft]

Hi @mpjtaylor ,

Based on my test, you can't cancel the automatic steps. But you can maually delete the steps which you don't want.

You can delete a step in Query Settings.

Or you can delete a step in the advanced editor. You need to keep the grammar correct.

Best Regards,

Stephen Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[mpjtaylor]

Thanks Stephen,

Deleting the steps and saving they just get recreated,

[girishks90]

i'm facing this problem too now...did you find a solution?