While fucked up, most people see it as a 1v1. He chose to not retaliate so any hard feelings are on him, not the general audience.
Nobody standing up for him when it was a 1v1.
There's that but I also think it hurt him to see he's not as loved as he thought he was in the Black community. And you see that happen with some comics when they embrace the identity of being a comic more than being Black. Chappelle is toeing close to that line too if he keeps up with the "woe is me I can't say what I want" stand up specials