An illegal operation is when a program is trying to access more memory than Windows wants to give it. If the program can create a scratch or temp space, it will use that instead - that is why Illustrator and Photoshop almost never give this error. Otherwise, the program tries to steal some memory from Windows itself.
In Windows XP, the core OS is constantly on the lookout for a memory theif, and simply turns it away, and your computer goes into thrashing mode, which is basically an infinite loop where the program wants to take memory and Windows won't give it. When a program in XP is not responding, often, this is what has happened.
In Windows 98, Windows only checks its memory allocation sporadically, and sometimes a program can get in there and take some. Eventually, though, the core OS will catch on and kill the process responsible. Now, when I say core OS, I mean something like the kernel or IO system, because as we all know, Windows Explorer and the Desktop and IE can all pull illegal operations. Actually, IE was the king of illegal operations, in my experience.
Ways to fix this - get Linux
Seriously, though, the more memory you have to spare, the better. And set your paging file size correctly. Adobe recommends a min paging file size of (RAM/2)+12, and max size of 2*RAM. So for 512MB, that's 268 min and 1024 max.
Also, even more esoteric, you can adjust the memory allocation directly. One thing that you can do in the config.sys file and autoexec.bat file is allocate and deallocate low and hi memory. Fucking with this without knowing exactly what you're doing can screw things up bad, so I wouldn't recommend this until you've had read a whole book on DOS.
btw, this perspective of the "illegal operation" is childish and generalized - therefore, its only purpose is to illustrate concepts to people who aren't familiar with them. Anyone who already understands memory allocation procedures in Windows will probably find this version is ridiculously dumbed down.