Well... they're kind of right.  I can't do ten push ups.  It doesn't mean I'm a fatty.  But that's because I have ligament damage in my elbow and that exact motion causes searing pain and eventually loss of sensation below the elbow.  But I'll bet I'm in better shape than them - all the injury means is that I find other exercises to do.