When you defrag you need alot of free space. Your basically taking a bunch of files and putting them "off to the side" until thier proper place is opened for them to be placed into.
And if you don't have enough space in the HD for the files to swap back and forth then it will go into memory .
Also as with FS9 make sure you have no unneeded back ground processes running while defraging, as mentioned above you'll need as much free resources available.
Take Gixers advice and remove any unneeded stuff and then give it a shot.
Might also want to run a "scandsk" or "chkdsk" if your system has that.