Our server at work, rooted

[xyle_one]

First, a little background. I work for a small design firm in Carson City, Nevada. We do print and web design, basically anything marketing related. We host our clients on a server rented from Media Temple. The people I work for are not that savy when it comes to server maintainance, and they rely on another company to do basic sys admin stuff on the machine.

Well, it turns out that, 1) there was not firewall running. for the last year, 2) someone, expoloited it (of course) through some unsafe scripts that were running. Now our machine is being used as a spam relay (their words, not mine, I'm not too sure if that is correct terminology). Their recommendation is to start fresh, with a clean restore, which means we have to rebuild 100 hosting clients, and more importantly, thousands of email accounts. I refuse to believe that is our only option. But it looks like we have no choice.

Is there anything I can look for, maybe to track down the exploited scripts, or at least to stop the spam? We are already prepared to start fresh.

I put this in the RedHat forums as the media temple server is running redhat enterprise 3.

We are fscked aren't we?

[Void Main]

No your not fscked. All of your configuration data is probably still good. What you need to do is boot from an alternate source and track down the naughty bits. There probably is some sort of root kit installed. The reason you have to boot from an alternate source is becuase good root kits modify the system programs and can hide themselves from things like "ps", "find" etc. When I've cleaned stuff up like this I usually find a trace of the problem, then I search for other files on the system with a time/date stamp close to the times on the pieces I did find. If it's a SPAM relay then there should be an extra port open on your server (nmap it) or your sendmail server has been opened up to allow mail to be relayed through it. How do you know you've been rooted? You must have some evidence of it. Start there. I also usually go through and do an "rpm -V" on each installed package and find out what's been messed with that I can't account for and investigate those pieces that have changed.

However, if you truly have been rooted then just copy all of your configuration files off (httpd.conf, vhost configurations, mail configurations, etc) and build a new server and restore those configurations. It IS important to find out how they got in or they'll just get in again. If it's poorly written inhouse code then that needs to be identified and fixed. If not, you'll just be doing it all over again. I'm sure all packages were kept up to date so no known exploits were exploited right? :) It's very important to keep your system up to date and reduce the number of listening processes to the bare necessities, and apply proper permissions and policies on those. And of course it is exremely wise to have this sort of server running in a DMZ.

[xyle_one]

When you say boot from an alternate source, what do you mean? Boot from a live cd? We do not have physical access to the box.

I have been in contact sorta with the company doing server maintainance for us. They aren't being as open about what is happening as I would like, and am still quite unsure what is going on. They say there are signs of root level access, and that there are exploitable scripts on the machine. About 95% of our clients have a contact form pointing to the same formmail.cgi script, one that has proven to be vulnerable.

I am waiting to hear back from them.

I did run nmap and i didnt see anything that raised any flags.

I have no idea why I posted here. It's just frustrating.

[Void Main]

Yes, alternate source would mean from a CD like KNOPPIX or put the hard drive into another machine. I guess not having physical access sort of hampers both of those thoughts though. :) Make sure you get backups of all your configs before doing anything drastic. You can still do some tracking down.. I think it's actually sort of fun analyzing some of these twit's breakins and figuring out exactly what they have done. Most of them leave a pretty good trail.

[xyle_one]

I have 4-5 days to do whatever I please on the machine. I am going to use this time to learn whatever I can. Thanks Voidmain

[Basher52]

wouldnt this be nice little thing to run?
http://www.rootkit.nl/projects/rootkit_hunter.html

this can detect alot of nasty things as Ive been told and since i installed it i run it almos every day