This is how I imagine it, but I am opened minded about having my mind changed. 
If you look at the scale of physicality and size for the spectrum of both sexes, my impression is that males significantly outnumber females on the muscular and large size end of the scale. I imagine if there are women who could compete in NFL pro football there would be few of them, and if they are there, they should get a shot at it, if they wanted it. Now this is probably the most physical team sport that relies on size and muscle mass.
Look at weight lifting. My impression is that woman can’t compete head to head with men (not said with a derogatory tone), and therefore have separate classes of competition.
There are other sports that are not so reliant on size and muscle mass, but then the question becomes why are most teams gender specific, baseball men, softball women? I think arguably there is a degree of gender bias/sexism involved. I see no reason why a woman can’t try out for a men’s baseball team.
So for trans women, a justifiable question becomes does being a biological male bestow a physical advantage after changing over? Is there an hormonal effect of the process of a male transitioning to a female that would remove this advantage?
