Actually, it's not exactly true that all embryos "start out" as female. Gender, aka whether your genome possesses an XY or XX  set of sex chromosomes is determined at the moment of conception - when the haploid male gamete (sperm) joins with the female haploid gamete (ovum) to form the diploid zygote. All embryos do, however, start out with the same set of paired ducts during the "Sexually indifferent stage"; the Wolffian and Mullerian ducts. At this point an embryo has the potential to develop either female or male structures adjacent to the already developing gonads (testes and ovaries).  At some point in embryogenesis the hormones produced from the triggering of genes on the Y chromosome will lead to the regression of the female  (or Mullerian structures) and Wolffian structures will develop into male genitals; the opposite happens in females. This is ofcourse a very condensed simplified version of what actually happens during embryogenesis, but my basic point is that this notion of all embryos being female by default that i hear repeated so often isn't exactly true. Even in cases like androgen insensitivity syndrome, where the body's androgen receptors are defective and the individual develops with an almost entirely female habitus, the person still possess undescended testes where ovaries should be and they do not have a uterus; because of that Y chromosome that's been there since conception. Anyway, the more you know. If this kind of thing interests you here's a link to site that explains it much better and in much more detail: https://courses.washington.edu/conj/bess/differentiation/differentiation.htm