Supposedly, yes, though breeding with other races along the Hammerfell border may have introduced some darker skin tones. But, from a development standpoint, I think Bethesda was going for an olive skin Mediterranean look to really drive home the idea: "Imperials are like Romans!!"
Imperials are not a 'race'. Imperials are both Colovians and Nibenese. To complicate matters further the Nibenese are not one race either, but rather different people formally enslaved by the Ayleids. Because they are from many different cultures, one can assume that skin tones varied from people to people. This may help explain some of the 'darker' skin tones in the genes.
Source: The Imperial Divide...by Vix (of course)
(Then again, maybe it's from being in the sun too long without sunscreen )
Which really doesn't make any sense as most Imperials in Skyrim should be of Colovian descent since they inhabit the northern and northwestern half of Cyrodiil. I suppose there might be a few Nibenese in there somewhere, but they're probably not the majority of Imperials you meet (given that most are in the military).
Yeah, Kyrielle and RuneRed sum it up pretty well. I'd add that they're different from Nords in the same way that Bosmer and Altmer are distinct races while sharing many physical features. Personally I'm glad they look different from Nords and even Bretons, otherwise it would be very hard to tell the races apart. Hell, I have a difficult enough time as it is. At least with Bretons you can tell because of their names, and because they're very short.