I think the reason white people are not allowed to form white-only clubs or associations is that it is assumed that when they do so, they think of themselves as better than people of every other race. They are haughty and exclusive. Bad taste, that.
When black people form their numerous black-only groups, this idea never occurs to people. Black people thinking themselves superior? Ridiculous. Who could possibly think something that silly? They just seek comfort in each other’s company in the face of their obvious inferiority! They are poor miserable ugly sons of bitches, and they come together to cry over being losers. We feel sorry for them, so eh, it’s no big deal if they exclude non-blacks. What white person would want to join them, anyway?
And the same goes for every other non-European group.
Again, that’s the unspoken assumption of both whites and non-whites; I myself need not have it; I am just clarifying the situation.