No, server is not lagged by players, we ran 30+ player games fine (and 64 with bots) and COD4 had 62 players at one time. But in COD4 there are other issues with 50+ players, not related to connection. Players cannot lag others, because server accepts game data from all clients, processes this data and then sends it to you. No client is able to send anything to you server doesn't permit -- all data you get is from the server only and server won't lag you.
There is one issue and this is too low rate and packets settings at clients (/rate, /cl_maxpackets). You may experience a bit higher ping, but NOT lag or connection issues, when we play on open maps and there is a lot of action. Server is sending you updates on all stuff visible to you in 360 degrees radius, plus some more, which sometimes requires rates of around 30000. Keep in mind this is rather low really, it means merely a 30k downstream and upstream from client is usually 2-12k per second, on top of that Q3 uses compression, so rate 30000 can push same as 50000 uncompressed or similar. I recommend
/rate 40000 (or more) and
/cl_maxpackets 60-90 and /snaps 30 (40 is same as 30). Use
/toggle cg_lagometer to see if your connection is good, few yellow dots now and then can still be good, if you have red lines or yellow triangles then connection is not good.

So, even dialup can handle Q3, it's no problem at all with cable or adsl or ftth. If you get disconnected for no reason then try playing Q3 with clean OS. Remove stuff you don't need, there must be something, either not allowing packets thru or disrupting your connection. Maybe there are wifi connectivity issues. If you test if internet is working properly then test download speeds with download manager and make sure source is fast enough to send at full speed of your connection. Download manager because they use multiple streams/connections to single file. Only then you can see if line is ok. Tests like speedtest.net are useless most of the time, because not all clients can receive single download at full speed and their servers are stressed, not always sending to you at full speed. If you just open few pages to 'see if it works', that won't tell you much because TCP protocol (for www) handles bad connections very good, while UDP which is used for games needs good unsaturated connectivity. UDP is only sending packets once and if there are line errors those packets are lost, so for games your connection must be reliable, while for TCP client can always demand new packets in case some get lost.
One more thing, if you test your max speed it's good to see graph as well, for example
http://www.dumeter.com/ can show you that. Always try multiple sites to download something, or download from multiple sources at once, so line is really utilized 100%. You also need to know what is your max speed @ ISP, so you can call them in case connection is not optimal. Many times problems are easily fixed.