Eagle Scout here, fitness requirements used to be strict, but not really anymore.  Also, you get to choose between different courses.  For example, if you don't want to get the swimming badge, you can go for canoeing instead.