No, ATi was not known for problems back then. I never experienced any glitches or freezing, also nobody i know complained about that, nor i noticed anything about this when reading reviews -- and i usually check reviews before buying anything. Sure, you will always find people with different problems for any hardware, but this doesn't mean product is unstable. I don't know how many hours i spent on DOOM 3 and other games, and if it was unstable i would surely know about it. I am not defending any brand, i am just surprised how quickly people jump to conclusions.
You can't expect miracles from 2004 lower mid-end card, which 9600 is. If game makes mouse lag, that is coding problem not graphic card's fault. It is how game is designed, nothing to do with hardware. Perhaps that game is more optimized for GeForce, so you have better response times. I don't play GTA, but i think this is most probably connected to fps you get. And mouse in-game (not windows) will move as fast or as slow as rendering system can do. Sell it and buy 9800 or better. Also 40 GB HDD can be serious performance bottleneck, make sure it is set UDMA, or even better -- give it away for free and buy 200GB+ HDD at least.
So you see the problem yet? You are saying all ATi products have problems, just because you are getting sluggish performance from one game with value-level video card from 2004. So should i install GeForce 2 MX100 from 2000 and then try to run DOOM 3 with it and say all nVidia products have "problems"?

About general performance, check reviews and you will see for yourself. 9xxx series cards many times were faster than nV 5xxx, but it also depends on games and which range of cards you compare. OpenGL at that time was faster with nV, which includes DOOM 3. However later things changed and for long period ATi dominated the market.