Not that it costs less fabric, but it's economies of scale.  If you make 10,000 shirts in XXL, and only 1,000 in Small, the XXL will be cheaper because you'll spend less time switching things around, you have more machines dedicated to that size, and so on and so forth.  This is thin privilege, heh.