The truth is that dressing well is harder to accomplish in an office environment where the dress code is less formal than a suit and tie. When dressing with three parts -- suit, shirt and tie -- you have an easy formula for handsome color coordination and, thus, an easy path toward looking good. But, when you dress in a casual combination, your sartorial skills are put to more of a test. It is harder to look really sharp with just a shirt and pants ... and no tie.
A Well, I guess you really can wear the same clothes if you don't mind being the only one at outdoor fabric work who dresses that way. Dressing differently from the crowd takes a certain amount of knowledge and a great deal of self-confidence.
I'm not sure it is necessarily right or wrong; often it depends on the degree of difference. If everyone else dresses super casually, as in and jeans, then dressing as formally as a suit and tie is too much difference. But if everyone else dresses in business shirts and dress trousers or even khakis, then you have more leeway to dress up a notch or two from the others.
I always wonder why a man would willingly give up wearing the one item of clothing that helps him to stand out in a crowd and look sharp with so little effort and complication. I believe it is a serious mistake to underestimate the importance of this smallest item in a man's overall look. People may not remember what else you are wearing, but they remember your tie. No other single garment makes as much impact.
Your personal style could be to wear a sport coat, or to go without the jacket but add a tie. You don't have to be a total conformist. There is no crime against setting yourself somewhat apart to express your own individuality. You may just be one who believes that dressing the part helps you to play the part and to project your professional image. In your rather "traditional" position as office manager, a tie might be not be so unusual; but if the president or CEO comes in wearing athletic wear, you need to take care not to come off as making a statement of complaint.
