You're saying something with your appearance whether you mean to or not, so you may as well mean to. For example, on a weekend morning, you might actually mean to say, "It's Saturday and I don't care how I look, and I don't care what you think of how I look, and I don't care if I ever have sex again."
You might really mean that. But you'd better think about what you're saying, because everyone else is. The idea that we humans are good-natured, politically correct, nonjudgmental beings is pure fantasy. We are, at the very least, judgmental.
Fashion is communication, plain and simple. I don't mean to sound as though I'm telling you something you don't already know, because any self-respecting man with even a little common sense knows exactly what he's saying and to whom he's saying it as he gets dressed in the morning. We all wear uniforms of sorts that allow us to be accepted. There's no shame in that. That we have the gumption to clean up and, as we stare into our closet, care about how we'll look shows we're trying to put our best foot forward.