cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
JWedge
Helper II
Helper II

Removing columns slowing refresh (.csv source)

Have a CSV source file pulling in just 15000 rows but over 1000 columns.  I only need a few of these so of course I remove most of them in Power Query.  Problem is it can't seem to deal with removing the columns.  If I remove a small number it takes a minute to think about it but then loads the data within 30seconds.  If I try and remove hundreds I end up giving up after half an hour of "Creating connection to model".

 

Yes I can modify the source file but really just want to know is this just to be expected or should it not be happening?  I have tried Table.RemoveColumns() and Table.SelectColumns with the same result.

 

Interestingly if I create a new query referencing the first one and then remove the columns it works quickly...

 

 

1 ACCEPTED SOLUTION
v-robertq-msft
Community Support
Community Support

Hi, @JWedge 

According to your description, you have a CSV file that has 15000 rows and over 1000 columns. I think it should be a huge amount of data for import mode. If you try to remove hundreds of columns in the Power query, it does take a lot of time because the transformation is operated on each column you removed. You can try to make another CSV file with the filtered files or just try the Table.Buffer() function as you said.

More info about Table.RemoveColumns() function in M language

 

Best Regards,

Community Support Team _Robert Qin

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

2 REPLIES 2
v-robertq-msft
Community Support
Community Support

Hi, @JWedge 

According to your description, you have a CSV file that has 15000 rows and over 1000 columns. I think it should be a huge amount of data for import mode. If you try to remove hundreds of columns in the Power query, it does take a lot of time because the transformation is operated on each column you removed. You can try to make another CSV file with the filtered files or just try the Table.Buffer() function as you said.

More info about Table.RemoveColumns() function in M language

 

Best Regards,

Community Support Team _Robert Qin

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

JWedge
Helper II
Helper II

Ok so I realised as I was submitting original post that I should try Table.Buffer() and it made a huge difference!

 

It appears as if the query is having to go back to the source file for every column it removes.  Can someone explain what's going on here?

 

Btw there are no complex steps in the query, literally just source, promote headers, remove columns, change types.

Helpful resources

Announcements
PBI User Groups

Welcome to the User Group Public Preview

Check out new user group experience and if you are a leader please create your group!

MBAS on Demand

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

July 2021 Update 768x460.png

Check it out!

Click here to read more about the July 2021 Updates

Power Query PA Forum 768x460.png

Check it out!

Did you know that you can visit the Power Query Forum in Power BI and now Power Apps

Top Solution Authors
Top Kudoed Authors