As a dude with broad shoulders I have this problem often. At this point I buy a size larger and shrink it in the wash when it comes to polo shirts. More comfortable and less fabric just hanging there.