Here's what you do...
Start with Darwin, build your own Display PostScript or Display PDF layer so that you can provide a Quartz-like graphics layer, then get a nice, cushy UI framework (KDE, Gnome, et cetera) and make said framework pervasive the way Aqua is with OS X.
Remember, OS X isn't just X11 with a pretty window manager, it's a whole new graphics engine combined with the Carbon/Cocoa frameworks which handle all the UI stuff.
Really, the graphics layer would be the fun part, after that, you'd just have to either adapt your framework to it, or design your graphics layer to emulate X11 calls.
Fiddle around with OpenStep, or see if you can get your mits on Rhapsody and hack the holy living fuck out of it.
If you could get some programmers together and make that work... actually pull off an open source implemenation of OS X, or something like it... you'd be the next Linus Torvalds!