If you're playing a melee/archery character, you can pretty much scale your damage to match any difficulty, for example, around 100 damage per hit is good for Expert. However, one of the problems is with magic in general - mages don't actually get any better as they level up, while their enemies do.
I tend not to play as low as Adept, because I get carried away with taking perks and become incredibly overpowered. Even a simple steel sword with a few perks in One-Handed can kill a dragon in a couple shots on Adept, and that doesn't seem right, seeing as they're meant to be the "world-eaters".
I wouldn't go that far...
Once I'd played on master once I didn't look back, anything lower feels too easy, barely worth even thinking about my build...
At least on master I have to think a little more before engaging enemies to avoid death.
Legendary difficulty on the other hand doesn't sound much fun, you have to hit things a good few times to get them to drop on master, legendary must become a pain in the ass...
I think it also depends how you play...
I tend to avoid high elevations and bandit infested forts more on master difficulty. Certain caves are also deemed too dangerous to set foot in and I'll stay away...
The Rift is another area I'll steer away from until at least level 15+ usually. (Bear packs!)
I stick to paths and consider what quests I can handle much more carefully. Also pay close attention to my perks so I don't spread too thin until I have a solid base set-up.
Dangerous areas on Master also include anywhere where there is a boss-level enemy instead of a group of lower level ones. The worst thing about high difficulties is the Kill-cams, not the actual fighting. I think the game just deciding to kill you when you're in no danger of legitimate death is just stupid.
I find most of the play styles easy, especially when combined with shouts. I find the hardest is a non-dragon-born, non-combatant, non-magic-using, thief. Although I've played a few, I don't usually care to play thieves - preferring to play a more honest 'hero'.
Also I find the first few levels the hardest, but as I gain a few levels my chances of survival goes way up; I estimate about 80% of my dead characters died before reaching 10th level. I rarely lose a character after hitting level 20, and I've only lost a few (c. 3 ?) after 30th. Now I rarely lose a character, period, but that has to do with my knowledge and proficiency of a game that I've played for nearly two years.
Each class can be hard and easy. A pure mage with enchanted gear that reduces casting cost of 2 schools of magic to 0 and the impact perk is unlocked it is easy even on Master (I had one, and I didn't finish the game because it got boring very soon). It's all about overpowering or not :)
Off the top of my head, I'd say a Bard is pretty hard to play. Especially because there isn't one defined or agreed upon skillset or armor type for a bard. If you play it the way I'd play it, you'd be using swords, staves, illusion, speech, alteration and perhaps alchemy. It's hard to do unless you get your illusion skill high enough early on.