My server has been hacked.

[Void Main]

First what software is paging you? Yes, I have already covered how to find hidden processes (sniffers etc). You have to make sure you have a good copy of "ps" and other utilities as I mentioned in an earlier message. Sound like you have a root kit installed in which the "ps" command that is currently on your system is set to hide the nasty processes. Get a good statically linked version of ps and any other utility (ls, etc) and upload to your server and only run those uploaded utilities. Using your uploaded "ps" command you should be able to see all of the processes. It is possible that a kernel module was inserted to do some of the hiding as well so you'll need a static version of modutils to upload (lsmod).

If it were me, I would shut the machine down and yank the drive, put a new drive in and install from scratch, and restore the client content. But you *must* fiigure out how the person got in so you can take measures to prevent it from happening again.

Well, it's past my bed time. Good luck!

[Webdiggity]

From what I can tell on the forums is that when you get like this it's time to reload. I don't have redundant drives on this box. I had been planning to upgrade to a new box with dual drives at the end of the month. It looks like this just moved my time table up a few weeks.

Lesson learned. Keep your system updated. I just hope it doesn't melt down in the next couple of weeks.

[Void Main]

But I thought you said you were keeping your system updated. If you haven't been then I can completely understand how this happened. But if you *have* been keeping your system up to date then your lesson is not yet learned. It is very important that you find out how this person rooted your box, not just for you, but for the rest of us. Even if you haven't been keeping your system up to date it is good to find out what they exploited and verify that it is a known vulnerability. If you don't keep up with CERT advisories, you should (but keep your system up to date daily is a good base).

It certainly would be your best option to rebuild from scratch, although I have cleaned up similar rooted systems in the past for people. It is possible to verify the integrity of everything on your system (but as I said, it takes booting from alternate source to be 100%). Of course you could rebuild your system from scratch, then do a restore from backup from a day prior to the breakin (if you can determine that time). You definitely should have good full and incremental backups if you are hosting people and charging them money for doing so.

[Webdiggity]

As I said earlier, there was a period of about 1 week where I didn't get to my computer to upgrade the kernel and I'm thinking that is when it happened. I'm still working on it. I was up til 3:00am this morning looking over logs. sheesh.

I also installed chkrootkit and it didn't detect anything. I know that probably doesn't mean anything tho.

[Void Main]

I don't know of any security issues in the last couple of weeks that would have allowed such an exploit. Which means they may have exploited a new hole that no one knows about. Again, I think it's important that you track down the hole. If you want me to help you can upload stuff to my ftp server (voidmain.is-a-geek.net). Things like the tar of your log directory and the txt files from your find and your rpm -V commands. (might want to do a "find -mtime 14"). I would be more than happy to look for unusual things in those files this evening.

[Webdiggity]

Do I need a username or password to upload to your server or just anonymous?

[Void Main]

Anonymous, but you have to change into the "uploads" directory.

[Webdiggity]

Ok, I dumped all the rpms to a txt file and ran find / -mtime -14 > /tmp/files.txt and tarred them up with my whole /var/log directory.

I gotta warn ya, it's 130mb. hehe Duh, I zipped it. haha Now its only 20mb. haha

[Void Main]

No problem, on the rpm list you did a "-Va" right?

[Webdiggity]

Yeah, I ran

# rpm -Va > /tmp/rpmchk.txt

If you would like to get in my server and poke around, I can arrange that too. Just let me know.

[Void Main]

I may be willing to do that, can't promise anything though. I was pretty busy last night, might be tonight as well.

[Webdiggity]

I fully understand. You have already been more than helpful.

[Void Main]

There may not be anything in them but have you saved off all the ".bash_history" files from everyone's home directories, most importantly root's? Look through the file for commands that you know you didn't perform and look for suspicioius activity in the user histories (if you find something in one of them that doesn't mean that user did it, could have been someone got their password, etc).

From your find I see there are some files that have been modified recently:

/usr/bin/cvsbug
/usr/bin/cvs
/usr/bin/rcs2log
/usr/bin/xmlwf
/usr/bin/annotate
/usr/bin/gdlib-config
/usr/bin/gdtopng
/usr/bin/pear
/usr/bin/php
/usr/bin/php-config
/usr/bin/phpextdist
/usr/bin/phpize
/usr/bin/bdftogd
/usr/bin/gd2copypal
/usr/bin/gd2topng
/usr/bin/gdparttopng
/usr/bin/pngtogd
/usr/bin/pngtogd2
/usr/bin/webpng

/usr/sbin/crond
/usr/sbin/imapd

And according to your rpm check you have some file where permissions are set differently than what they were set to at install time:

.M...... /usr/bin/chfn
.M...... /usr/bin/chsh
.M...... /usr/bin/newgrp
.M...... /usr/bin/write

and this one concerns me unless your web software installed a custom cron:

S.5....T /usr/sbin/crond

and these show up again with different sizes and times:

.......T /usr/bin/bdftogd
S.5....T /usr/bin/gd2copypal
S.5....T /usr/bin/gd2topng
S.5....T /usr/bin/gdparttopng
S.5....T /usr/bin/gdtopng
S.5....T /usr/bin/pngtogd
S.5....T /usr/bin/pngtogd2
S.5....T /usr/bin/webpng

SM5....T c /etc/bashrc

.M...... /bin/ping
.M...... /usr/sbin/ping6
.M...... /usr/sbin/traceroute6

S.5....T /usr/bin/GET
S.5....T /usr/bin/HEAD
S.5....T /usr/bin/POST
S.5....T /usr/bin/lwp-download
S.5....T /usr/bin/lwp-mirror
S.5....T /usr/bin/lwp-request
S.5....T /usr/bin/lwp-rget
.M...... /usr/bin/rcp
.M...... /usr/bin/rlogin
.M...... /usr/bin/rsh

This one has probably changed for a reason but check to make sure something hasn't been added unbeknownst to you:

S.5....T c /etc/crontab

I just can't imagine why you have so many files with permissions that have changed, these can be dangerous if not set right:

.M...... /usr/bin/sudo
.M...... /usr/sbin/userhelper

These should be checked out:

S.5....T c /etc/inittab
S.5....T c /etc/rc.d/rc.local
S.5....T c /etc/sysctl.conf

S.5....T c /etc/pam.d/system-auth

.M....G. /bin/su

Now your hosting software may be responsible for a lot of the changes from the defaults and if that is the case, nothing really jumps out at me. It doesn't appear that a root kit is installed, but then you used the find command from the tainted system correct? If you ran a statically linked find command that you uploaded from another system and came up with the same files then it almost looks as if there is no root kit installed. Of course a kernel module running can hide a lot of things too, so to be really sure you have to boot from alternate media and mount the drive so you know you have a good running system to do your analysis from.

It's also hard for me to see anything suspicious since I don't know the history of the machine. Again, good luck!

[Webdiggity]

Just recently, I updated all of the system software and whm so that could possibly be alot of the reason, right?

My provider seems to think it might have been a script kiddie using an exploit off of a 3rd party php package like nuke or something.

I downloaded a new PS and another package that has find, ls and a bunch of others. I can upload them to a dirctory call audit and 'make' them static from what I learned reading up today. However, how do I make sure I'm running those commands and not the ones on the system?

Sorry about all the questions, I also have a new server coming this weekend complete with dual 80gig drives. I'll be using rsync to make daily incremental backups on the second drive unless you have a better suggestion. This has been something that I planned to do for a few weeks now anyway so no harm at this point.

This has definitely been a big lesson for me. Can you point me at some good resources for security? I think I need brushing up in that area. I can upload packages and edit files and do searches comfortably now but I think, no, I KNOW, I have to bone up on my security practices.

Once I get the accounts copied to the new box, I'll have the old server until the end of the first week in March. Does anyone want a crack at this box in the event that I cannot find anything? Suggestions?

Thanks again,
Matt

[Void Main]

Just recently, I updated all of the system software and whm so that could possibly be alot of the reason, right?

Most certainly, depending on what system software you updated. If they were RPMS then it wouldn't account for all the permissions problems.

My provider seems to think it might have been a script kiddie using an exploit off of a 3rd party php package like nuke or something.

Very likely. One of the members here alerted me to a security issue with my phpMan pages. You could read any file on my system using the right URL Well, not any file, only files that the apache user had read access to, which still can give out key information.

I downloaded a new PS and another package that has find, ls and a bunch of others. I can upload them to a dirctory call audit and 'make' them static from what I learned reading up today. However, how do I make sure I'm running those commands and not the ones on the system?

Couple of ways. One way is to preface each command with a "./" if you have changed into the directory where they reside, or better yet just type this on the command line after logging in:

# export PATH=/path/to/your/static/commands

Sorry about all the questions, I also have a new server coming this weekend complete with dual 80gig drives. I'll be using rsync to make daily incremental backups on the second drive unless you have a better suggestion.

That is better than nothing, but it would be far better if the backups could be moved off to a separate machine, or to tape, etc. A person that has root can wipe out your entire system with a single bound.

This has definitely been a big lesson for me. Can you point me at some good resources for security? I think I need brushing up in that area. I can upload packages and edit files and do searches comfortably now but I think, no, I KNOW, I have to bone up on my security practices.

Security is one of the most important things and something you can never stop trying to be better at. This will get you started:

http://www.sans.org/
http://www.cert.org/

It's a humbling experience no? :)

EDIT: I forgot to mention that if it is a script kiddie that exploited a flaw in a php script etc, then if you build the new system just like the old you'll still have that hole open. I wouldn't personally feel comfortable until I found out exactly how the person was able to modify those files. For instance, I would want to know what the owner/group and the mode (permissions) on the files that were changed. If the permissions were such that the user your web server runs under could not have modified them then it's likely they were in with root access, that's more than exploiting a PHP script vulnerability. The user the web server runs under should not have permission to do squat.