About tightening the API, it could be done without breaking backwards compatibility. Here's how:
First, let applications to provide manifests for what they want from the APIs, exactly. After that, things other than these wouldn't just work. This wouldn't break anything, since applications would themselves state what they want and what they don't. This is important because it's theoretically impossible to determine if an application does action X during its runtime or not, without actually running it. If the application states that it's not going to do it, then the mechanism doesn't need to be provided and the application is known to be safe in that regard.
Provide a developer version of windows which requires that all applications manifest what they need, and by default applications run in a very restricted environment with limited API availability, registry and filesystem scope and so on. Developers don't mind it as much as the masses do. Developers can fix things. Developers can hack. Since this version of windows would be extremely security restricted, it should be made "free" and installable alongside any other properly licensed windows system. Give enough powertoy goodies with this release to make developers prefer it, and all future applications will come with proper manifests shipped.
In few years, once there's a significant amount of manifested applications in the market (or perhaps even earlier) release a tool for manifest auditing. The tool would be used to scan system for any non-manifested software, and to determine which applications require privileged API functionality such as unlimited GetAsyncKeyState, SetWindowsHookEx, AttachThreadInput which are used for keyloggers. Properly behaving applications don't need to GetAsyncKeyState while they aren't active, or do global hooks, or attach themselves to threads they don't own. Non-manifested applications thus simply couldn't act as keyloggers, as they wouldn't have access to the required APIs to implement them. Same applies for many other sorts of malware, they still need APIs to function.
Once most commonly used applications are manifested, release Server edition which allows for manifest signing. So, no applications will run on the server unless they come with a manifest signed by the administrator. The signed manifest will specify exactly what APIs the application is allowed to access and so on, and everything will be fine and happy. If this works fine, release same and similar functionality for home systems, which warn users if they're trying to run application that requires risky functionality. They might still click yes, but at least they'll have the choice.
Finally, this will need some design changes as well to allow for finer granularity of the manifest checking. A text editor needs to save the file, but otherwise it won't need to read/write anything on its own. An ideal solution would be that the text editor saves data to a stream, and the stream is saved by a common control which lets the user choose where to write the data. This way the files in text editor could only be opened through user interaction, with exception of special files in manifested directories. I'm sure Microsoft has resources to make this design change if it wants to, and it's only necessary to let applications manifest strict scopes while allowing broad access to the system for the user.
Oh my, that's long. I hope the idea got through, I've been thinking about this for a few days now :p
