Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
mpjtaylor
Advocate I
Advocate I

Dataflow - How can I stop automatic steps being added

I have a Datalake with multiple folders with lots of blobs that are JSON files each folder has different json structures and some are lists and others are records. I am trying to use a single dataflow to read in the content so that I can use subsequent dataflows to process specific folders and json files. The reason for this is to have a single datalake scheduled read transaction to get all the raw data.

 

The problem I have is I cannot find a way to save the dataflow with either the binary column or the Json [Content] column as they are automatically removed with the addtion of these frustrating steps which I want to prevent:

 

  #"Transform columns" = Table.TransformColumnTypes(JsonDocument, {{"JsonDocument"type text}}),
  #"Replace errors" = Table.ReplaceErrorValues(#"Transform columns", {{"JsonDocument"null}}),
  #"Remove columns" = Table.RemoveColumns(#"Replace errors", Table.ColumnsOfType(#"Replace errors", {type tabletype recordtype listtype nullable binarytype binarytype function}))

 

I have tried changing the Global and project settings but nothing I do seems to stop these being automatically added,,also tried changing the column types, duplicating the columns

 

How can this be achieved I just want the steps to stop at:

 

let
  Source = AzureStorage.Blobs("storageaccount"),
  Navigation = Source{[Name = "data"]}[Data],
  #"Filtered rows" = Table.SelectRows(Navigation, each [Extension] = ".json"),
  ChooseColumns = Table.SelectColumns(#"Filtered rows", {"Content""Name"}),
  JsonDocument = Table.AddColumn(ChooseColumns, "JsonDocument"each Json.Document([Content]))
in
JsonDocument

 

Kind Regards

Marcus

 

14 REPLIES 14
myusernameisjon
New Member

I was doing something similar, reading in a bunch of json files from datalake and using a Dataflow to split and read so only wanted to use one read query and reference multiple times. I managed to get around it by converting the "Content" binary column to text. The first step on my referenced query converts it back to Binary, and then i can continue executing all of my transformations from there.

FireFighter1017
Advocate II
Advocate II

Someone may have pointed this out but, you should check column profiles for errors before you save your dataflows.  And make sure you do with column profiling set to entire data set:

FireFighter1017_0-1683045308171.png

 

You can add a step at the end of your dataflow that will check for errors :

(In this example I voluntarily created a record with a text value in column "Score" to cause a type casting error)

 

 

let
  Source = #table({"Name", "Score"}, {{"Betty", 90.3}, {"Carl", "a"}}),
  #"Changed column type" = Table.TransformColumnTypes(Source, {{"Name", type text}, {"Score", type number}}),
  #"Kept errors" = Table.SelectRowsWithErrors(#"Changed column type"),
  #"final-step" = if Table.RowCount(#"Kept errors") > 0 then #table({"Error count"},{{#"Kept errors"}}) else #"Changed column type"
in
  #"final-step"

 

 

 

You can add your transformation steps right after "Source" step and the "final-step" should tell you if you have errors or not.  If it returns data with expected row count, you're good.  If not, then click of the only cell showing up to see records that have errors.  Click on #"Kept errors" step to see error messages and fix them in previous steps.

This way Power BI will most certainly add steps when saving your dataflow but your table will only have one column with only one record and a null value in it.  Which makes it easy to spot somethign's wrong with your dataflow.

 

It would be better if we could just change a setting and ask Power BI Service to prevent saving if it detects errors instead of screwing with our data.  That is another forum...  Good luck with that one...

 

 

Stretch
New Member

I'm a total noob, but this is working for me.  I was having this same issue, and found this article that explains why it happens is because it is trying to make sure that all columns have an assigned format and not just the defaults.  I went into each query and clicked on a value in the results table, did ctrl-A to select all and then did a Detect Data Type.  Once I did that it stopped automatically inserting those steps.  The only thing is that you can't have a List or a Record as a column because it will still apply those steps since they aren't actual data types.  I'm adding those columns as a new query so I can expand the data (this is cleaner then just expanding in the column based on the data) and then I'm removing that column and building a relationship so I can tie the two queries together.

Anonymous
Not applicable

Hi,

 

Unfortunately also no solution here but i did find a post from RADACAD about this topic:

https://radacad.com/working-with-records-lists-and-values-in-power-bi-dataflows

 

in short, you can also avoid the extra steps by disabling the "enable load" of the flow. This of course is not an ideal solution but it did work for me as I use the list as input for another flow.

KBENN
New Member

I'm facing exactly the same issue here. Anyone have a solution for it ?

Thanks a lot !

yyr6
New Member

Same issue happening for me. I keep deleting the steps, but they keep reappearing, and when pulling into Desktop, I cannot make the changes I need because of the dataflow limitations. Any update?

SMHolck
Advocate IV
Advocate IV

I may have found a way around the one that shows a fX (function) symbol and says "Remove Columns" ... 

Table.RemoveColumns(#"Previous step name", Table.ColumnsOfType(#"Removed columns 1", {type tabletype recordtype listtype nullable binarytype binarytype function}))
The PowerQuery editor for Dataflows seems to be adding this in an attempt to get rid of columns where every current value is null.  However, I want it to retain the column because I am writing the code ahead of the data - as in the system doesn't yet have values in it yet, but will in the future.
 
Try explicitly setting the type of the columns that you want to keep.  My best guess is that it was implicitly converting my column to the nullable binary.  But by explicitly setting the type in the previous step, it finally kept my column.  It was a little funky in my case because that column was explicitly set in the original query which was merged with my final table, so not sure why I had to tell PowerQuery the column type twice.  

Could you provide more details on this solution? I can't figure out how to implement it.

  1. Select column(s) in the editor
  2. From the Transform menu, click the Data Type drop down button and select the desired data type
  3. Repeat for each column

NOTE: For step #1, you can Ctrl+Click to select multiple non-contiguous columns or Shift+Click to select multiple columns if you want to set several to the same data type at once, e.g. if you have 5 columns that are integers and 2 that are text and 3 that are decimal, you would select the 5 integers and transform them, then select the 2 text and transform them, then the 3 decimal

SMHolck_0-1670008553650.png

 

Anonymous
Not applicable

This worked for me! thanks 🙂

v-stephen-msft
Community Support
Community Support

Hi @mpjtaylor ,

 

To my knowledge, your requirement about locking applied steps could not be realized currently.
But this is at idea/feedback level alone. May be ,hopes Power Bi will implement this in nearby future.

 

 

 

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

v-stephen-msft
Community Support
Community Support

Hi @mpjtaylor ,

 

Based on my test, you can't cancel the automatic steps. But you can maually delete the steps which you don't want.

 

You can delete a step in Query Settings.

17.png

 

Or you can delete a step in the advanced editor. You need to keep the grammar correct.

18.png

 

 

 

Best Regards,

Stephen Tao

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Thanks Stephen,

 

Deleting the steps and saving they just get recreated,

Anonymous
Not applicable

i'm facing this problem too now...did you find a solution?

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors