Folks live in a suburb of choice because of how it is....how it is now.
When that image changes enough that they don't like the image; in their mind it's time to move out.
Sadly; the non-PC explanation is they only want what they had before - if that was a mostly white neighborhood, then that's the reason.
Is it right?
Probably no, but it is one of the main reasons.
While we all want to live in the PC world of equality and tolerance and all that; at the end of the day it's a rose coloured glasses view to a large degree.
Just look at golf clubs for example (my career for 30 years); many have rules where you can only join if you've known a certain amount of Members for a set minimum period of time - eg; 6 Members for a minimum of 6 years. These Members have to propose you, or you have no chance of joining.
After this; someone may still "blackball" you and you won't get in.
These rules are put in place so they only get "like-minded" folks as Members in their "suburb" if you like...they don't want folks who are deemed "different".
Is it right?
Sorry; but yes. When you join a club or organisation such as these; one of the reasons is to spend time with only the folks you want to.
This is more yer exclusive private Club scenario; many private Clubs are much more open and accepting (they usually need the money from Members' fees), but even in these two types of Clubs I observed Clubs within Clubs, where some Members would not associate with other Members - not necessarily race or colour - often it is age related, or skill level related, or vocation related, etc.
Fernwood womens' gym would be another example of selective associations.
It's human nature in many of life's ways to have these behaviors..
With regards to where you live; you haven't the luxury of excluding folks from it; so you have to move; or adapt.