For me it was more the fact that it makes light of the serious themes by undermining them later. Like, Ann being constantly sexually harassed by her own teammates after her arc (and this being played for laughs). Or the theme of the Phantom Thieves being the underdogs fighting against the corrupt adults, but then shitting on one of the most marginalised groups in the world (LGBT+ people). It just made the story fall apart for me, and took me out of the game.
I did end up finishing it, and it placed somewhere on the bottom end of my top 10 games of that year, but I was really disappointed by this aspect of the game. There are of course some things it does well, and I enjoyed them.