Go into the Fields Icon , which is on the Right End ( which is more of a Rectangle inside a Rectangle Shape). Right Click on the field which you do not want the total for "Check"/ "Select" the "Dont Summarize" option.
So I am going to add to this string because there is some confusion. Yes the solution identified turns of ALL TOTALS. But as per the original post - some of us want less sub-totals but still totals. This question is not answered.
There is a post saying that one can right click and select Do Not Summarize - - but that post is incorrect as several have noted - it applies only to the Table visual, not the Matrix visual.
What is happening is that there is a Total line for each Row. The Row fields are defined & showing in the Visual Design Pane at right. To have a less number of Total lines you need to have a less number of Row fields.
In my case I had separate fields for customer name, customer number - - so I created a new calculated field that merged those 2 fields - -so then in the matrix I could have one less Row field - thus one less Total line....