The area called the "Gray Quarter" was never on par with the rest of the city. At least, my understanding is that it was always a slum. Perhaps I'm wrong. Furthermore, Ulfric does nothing to make it remotely equal to the rest of the city. Let alone the awful situation the Argonians find themselves in. That's the thing. It's willful neglect and blatant unequal treatment. Any concerns raised by the Dunmer fall on deaf ears and the Argonians have basically no voice at all.
It's easy to say "if you don't like it just move" (in my mind, that's victim blaming) but it's the same reason people get "stuck" in real life. Picking up and moving is never easy, especially when you have to do it by like... a horse-drawn cart.
http://elderscrolls.wikia.com/wiki/Gray_Quarter
Essentially, it's a makeshift refugee camp. The area was a slum, the interior of the buildings are in disrepair, and it was the only option afforded to the Dunmer. Previously it was called the "Snow Quarter" and once the Dunmer came in the Nords changed the name to make it about skin color. The fact that it's a slum named after the inhabitants who are treated unequally, that all the Nords avoid I might add... I mean how do you not see that as xenophobic or racist? If Nords residents won't do business with the Dunmer, then the chance that your average Dunmer living there would be able to improve their circumstances are significantly diminished. Fewer customers, less business, less money, and therefore reduced access to resources for improving their homes or relocating. The deck is stacked against them and the Argonians have it even worse.
Also a five minute real time walk is so much longer than that in Skyrim time. I mean it's a country so I feel like distance between cities is significant.
Yeah, but...
Sure, it's possible, but what causes problems are those arrogant, individual leaders who supposedly speak for an entire race that get people into trouble. As a whole, the races would probably be able to get along with each other. It is the foolish military and spiritual leaders who incite war and racial division.
The attitude of many of the citizens there says everything. When I went into Windhelm for the first time on my initial playthrough with my Dunmer character... well, let's just say it the first city where I earned a murder bounty.
I always avoid Windhelm, no matter what kind of character I'm playing. What a hateful hellhole.
The Dunmer inspire hatred of themselves, the Khajiit and the Argonians were enslaved by them, so no racial harmony there, most Nordic enemy that you come across will yell "Skyrim belongs to the Nords" to which I reply before lopping off his/her head fine keep it then, Nordic Khajiit relations are on such a high that they wont even tolerate them within cities. Guards will tell you quite freely" stay out of trouble Elf" , Imperial.
Wouldn't it be just grand if everyone shook hands, paws, fins or whatever and started hating on the Thalmor, destroyed them totally, sell them at slave markets, or give them all work in a grand capacity of course within Divith Fyrs Corprusarium.