Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn the coveted Fabric Analytics Engineer certification. 100% off your exam for a limited time only!

Reply
cparker4486
Helper III
Helper III

New column in source table not appearing in dataset after refresh?

I found this post but it doesn't play out for me. http://community.powerbi.com/t5/Service/New-column-in-source-table-not-appearing-in-dataset-after/m-...

 

The only way I've been able to get new columns to appear is to change the Source line in the Advanced Editor. Specifically, I have to change the Columns value to reflect the current amount.

 

Source = Csv.Document(File.Contents("C:\source\file.csv"),[Delimiter=",", Columns=12, Encoding=1252, QuoteStyle=QuoteStyle.None]),

Is this the expected behavior or is it specific to CSV source type?

2 ACCEPTED SOLUTIONS

Hi @cparker4486,

 

I will report this issue internally, and keep you update once I get any feedback.

 

Best Regards,

Qiuyun Yu

Community Support Team _ Qiuyun Yu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

nickjtay1
Advocate I
Advocate I

I spoke with Microsoft Support and have a fix.

 

PBI Desktop doesn't recognize the new column for March that was added:

 

1.png

 

When you open the advanced editor, you will see that the original number of columns is defined:

 

1 adv.png

 

Remove the definition of the number of columns:

 

2 adv.png

 

When you click "Done" and return to the query editor the new column appears:

 

2.png

View solution in original post

23 REPLIES 23
karpandi
Regular Visitor

As I haven't seen this solution above, I add it here.
Under Applied Steps > Removed Other Columns it is possible to tick the missing newly added column.

karpandi_0-1652198437850.png

 

melly
New Member

Thank you for this post.  Very helpful.  

obera
Frequent Visitor

Hi folks,

 

I am having this issue but I do not see a place in my code that specifies number of columns to change/delete it, and it is still not showing my additional 3 columns.

 

let
Source = Folder.Files("C:\FILEPATHHERE"),
#"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true),
#"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File from Assignment Group Duration", each #"Transform File from Assignment Group Duration"([Content])),
#"Renamed Columns1" = Table.RenameColumns(#"Invoke Custom Function1", {"Name", "Source.Name"}),
#"Removed Other Columns1" = Table.SelectColumns(#"Renamed Columns1", {"Source.Name", "Transform File from Assignment Group Duration"}),
#"Expanded Table Column1" = Table.ExpandTableColumn(#"Removed Other Columns1", "Transform File from Assignment Group Duration", Table.ColumnNames(#"Transform File from Assignment Group Duration"(#"Sample File (2)"))),

When Query a folder with CSV. files, you can get the same problem, but you have to apply the solution to the supporting query under "Transform File from (Query Name)". Here you select the query used for the fily taped marked with the table icon, and then open the advanced editor and apply above solution. I just added one to the amount of columns and worked as a charm. 

Same problem as Obera in that when using 'Source = Folder.Files" there is no 'Csv.Document' command to indicate the number of columns.

 

Any guidance or suggestions would be very welcome.

I had the same issue and could not find where to change the number of columns being picked up.

I found it in the Advanced Editor of the 'Transform Sample File.....' under 'Sample Query' and not in the main query feeding my table.

Once I had increased the number of columns and refreshed, the additional columns appeared.

nickjtay1
Advocate I
Advocate I

I spoke with Microsoft Support and have a fix.

 

PBI Desktop doesn't recognize the new column for March that was added:

 

1.png

 

When you open the advanced editor, you will see that the original number of columns is defined:

 

1 adv.png

 

Remove the definition of the number of columns:

 

2 adv.png

 

When you click "Done" and return to the query editor the new column appears:

 

2.png

Thank you so much for this. I'm sure I'm not supposed to comment simply to thank someone, but in this instance, where you saved me so completely, I will spit in that rule's face. I'm very thankful for this answer.

It was very helpful. Thanks.

Anonymous
Not applicable

One of the best answers ever I have come across. Thanks 🙂

It really needed lot of searching. @nickjtay1 

I ran into this problem and found that referencing the data as a table solved the problem. I have since adopted importing excel/ csv source data as a table as a best practice.

Hi scott, what do you mean by "referencing the data as a table"? Is it some kind of different way of importing the CSV file or is it a transformation after importing?

about_blank
New Member

The problems are still there. (Power BI Desktop)

zebing2
Advocate I
Advocate I

Did this ever got resolved? Not being able to refresh my data sources quickly when I am frequently changing columns is a major problem. My csv files aren't refreshing properly and I'm wondering if I need to go back to Tableau to get things done. 

 

Is there a fix for this yet?

v-qiuyu-msft
Community Support
Community Support

Hi @cparker4486,

 

Based on my test, when we connect to CSV file in Power BI Desktop, the added column in a CSV file will not update whether we click Refresh button in Home page or Refresh Preview button in Query Editor. As you mentioned, we need to modify Columns value then the added column will display. In my opinion, Columns parameter limits the number of returned column from CSV file, if we added columns in CSV file, Columns parameter value needs to be modified as well to make new column display.

 

Also I have tested with SQL Server database, if we haven't written query to indicator which columns need to be returned(eg: select column1, column2 from table1), the new column will update after clicking Refresh or Refresh Preview button. I thinks the issue should be related to CSV data source itself, as each data source are defined differently in Power Query. You can take a look at Advanced Editor in Query Editor.

 

If you have any question, please feel free to ask.

 

Best Regrads,

Qiuyun Yu

Community Support Team _ Qiuyun Yu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Hi @v-qiuyu-msft,

 

Then is this considered a bug and will be addressed at some point?

 

 

Thanks.

Hi @cparker4486,

 

I will report this issue internally, and keep you update once I get any feedback.

 

Best Regards,

Qiuyun Yu

Community Support Team _ Qiuyun Yu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Any update on this? It's still an issue.

Anonymous
Not applicable

go to Advanced Editor:

remove 'Columns=3, '  (probably you have a different number of columns)

click 'Refresh all'

For me it is not taking the new columns so I changed the Columns= 8 in the advance editor according to the columns I need from csv file. Then automatically new columns got loaded into Power BI

Helpful resources

Announcements
April AMA free

Microsoft Fabric AMA Livestream

Join us Tuesday, April 09, 9:00 – 10:00 AM PST for a live, expert-led Q&A session on all things Microsoft Fabric!

March Fabric Community Update

Fabric Community Update - March 2024

Find out what's new and trending in the Fabric Community.