The seems no ultimate all-in-one handheld for retro games.
The experience on the new Odroid GO Advance is very good, ie for GB/GBA/NES/Arcade/PS1 games, for some PSP games (eg Virtual Tennis) it doesn’t perform well.
The PSP GO is of my top choice apart that for some arcade games (eg Hook) it’s losing frames. The grip is essential to improve its ergonomics.
For GB/GBC/GBA games I prefer the GBA SP with backlight, but the K101+ is also a good alternative.
If one doesn’t care too much about the portability, the PS Vita Slim will be the best option (good performance, larger screen, sleep function).
Of course for DS and 3DS games no emulators on other devices.
Android-based handeld is also promising, eg. the ZPG.