You're right that "the public doesn't care about trans athletes competing."
But huge swathes of the public across the political spectrum care about male athletes competing in female sports.
That's the issue here: male athletes competing in female sports.
Your contention that the only people concerned about males using gender identity claims to horn in on female sports are a "small minority of bigoted Q-anon followers" because to everyone else "this is a complete non-issue" just goes to show that you have no clue what's going on.