Many of those differences are due to hormones. The SRY is a trigger for sexual differentiation, but it’s the hormones and the other genes activated by those that make the difference during development, and hormones are required to maintain many of them.
Estrogen hormone therapy can cause physical, sexual, reproductive, and emotional changes. Learn more about how it affects the body here.
www.medicalnewstoday.com
But it does make things complicated as some things are reversible under HRT given enough time, and others aren’t. Muscle mass (and the ease that someone can build it) and fat distribution are (although we don’t have good research other than general steroid research). Testosterone is a steroid after all. Skeletal growth doesn’t change as much after puberty though. But if someone does get access to blockers, and doesn’t have to go through two puberties, then these non-reversible effects don’t happen, and HRT is more effective on the whole.
Some of these effects are even specific to one hormone or another, lowering of the voice for example. A trans man will have their voice deepen on testosterone, but a trans woman will not have their voice affected on estrogen if they went through a masculine puberty first. Breasts grow because of estrogen and progesterone, but cannot be “undone” by testosterone.
There really isn’t a black and white simple answer here. Starting HRT after puberty or getting access to blockers, trans masc vs trans femme, it all plays out differently.
If your concern is testosterone giving folks an advantage, I don’t think making trans men compete with cis women will have the effect you think it will.