Because those ("keep politics out of my games") people don't see privileged groups as political. A male protagonist isn't political. A heterosexual couple in a game isn't political. Only women and minorities are. It's just an argument to keep women and minorities in their place. They see the pleas for representation as unnecessary and uncomfortable, so they try to shuffle into something separate from their hobby. So when people say that, they're just trying to shut down those tired bigoted arguments by saying "we exist just like you, this isn't controversial or something that should be disputed". I think it's just as simple as that, though I get where you're coming from.