It’s the testosterone - it changes your facial structure and body a lot more than estrogen does, so it’s harder to cover up initial testosterone with estrogen but super easy the other way around. That said once you go on estrogen your face does soften a lot and you lose or mitigate a lot of secondary male characteristics.