My server has been hacked.

[Webdiggity]

Would it be better for me to build a dedicated Linux box here at home and rsync them to this one instead? I'll keep the 2nd drive as primary backup and the box at home as emergency backup?

What kind of system would I need. I could load Fedora on it with an 80 gig drive (cheap) and 256mb of ram on an AMD 1500 or something. Won't need a monitor, keyboard or mouse once I set it up.

I'll try my best to do it on my own and only bother you guys if I get stuck.

hehe

Thanks again guys.

[Void Main]

Well, I have a DLT tape changer attached to my server at the hosting facility. I did also rsync files to another server and had an automatic failover to the second server if ther was a problem on the first server, or it wasn't reachable. But at least keeping backups of critical data off-server would certainly be wise. rsync is pretty efficient (I do rsync via ssh) as far as being able to transfer just parts of files that have changed so cable/DSL should be just fine for keeping a good backup on your home machine. I used to be pretty anal about that and keep like 3 copies in 3 different places, on top of backing up on tape. :) That might be considered "overkill".

[Webdiggity]

Yeah, I like the way the first back up is a one for one copy and the rest are just snapshots of the changes so you can roll the server back if you have to. I'll probably get the rsync up and running before I start moving accounts over so the first download is as small as possible (probably still the biggest)

So it is possible to have my box at home automatically access my production box and download the backups or visa versa. The SB box dials home and downloads the backups?

Anyway, do you want a crack at the old box once I get situated on the new system? Just let me know. You get first dibs. haha

Thanks again,
Webby

[Void Main]

At home I would have a cron job that runs and connects to the server to do the rsync.

As far as the server, what model is it, what are the specs and what do you want for it? I'm not really in the market for one, especially one that's been cracked. :)

[Webdiggity]

No, I meant do you want to poke around on it and try to dig deeper. It's actually going back to SB. I'm sorry if I didn't make that clear. Hehe

[Void Main]

You've already done most of the things I would have done, other than get a process list with a static linked ps and rerun the find with a static linked find. If it were my server there is a lot more that I would do, but that is only because I would be intimately familiar with what is on it and what sort of changes I have made. I have been browsing through your logs and haven't come up with anything as of yet. Either they are good at covering their tracks or they didn't do much to your system.

[Webdiggity]

I have a copy of ps downloaded to my pc. I deleted the copy on the server. I didn't know how to make it run statically. I"ll be doing that tonight to see if anything glaring pops out. Thanks for the info. There is a good article I found on creating and using a statically linked audit "kit" that I'll be sending to you to see if it would be of value to others.

Webby

[Webdiggity]

So I should clamp down the user the web server runs under? I'll look into that asap. Also, since I already cleaned the files, is there a log that lists file modification history for my /home directory?

As I get the new server on line, would it be a good idea to have my logs periodically backup up and dumped to a safe place. I don't mind going overkill on the new rig.

[Void Main]

I have a copy of ps downloaded to my pc. I deleted the copy on the server. I didn't know how to make it run statically. I"ll be doing that tonight to see if anything glaring pops out. Thanks for the info. There is a good article I found on creating and using a statically linked audit "kit" that I'll be sending to you to see if it would be of value to others. ;)

Webby

What do you mean by you "deleted the copy on the server"? You just want to compile and static link a new copy and uplaod to your server (don't forget to make it executable if it isn't already with a "chmod +x ps").

[Void Main]

So I should clamp down the user the web server runs under?

No, not specifically. It's just that I see some people (even on tutorials <gasp>) say that you should change the ownership of your web content to match the user that the web server runs under (usually the user is "apache", "httpd", or "nobody"). This is the exact wrong thing to do. The web user should not be able to write to any directory or file. There are very few limited exceptions to this (a counter data file for example needs to be written to by the web user). When someone exploits a vulnerability in your web server or in a PHP script they will be connected with the rights that the web server user has which means if that user owns all of your content then all of your content can potentially be changed by the cracker. That's why I asked you to list the permissions on the index.html files and the directories that they are contained in (ls -l index.html).

Also, since I already cleaned the files, is there a log that lists file modification history for my /home directory?

Got any incremental backups? :)

As I get the new server on line, would it be a good idea to have my logs periodically backup up and dumped to a safe place. I don't mind going overkill on the new rig.

Normally it's a good idea to have a remote syslog server so all of the system logs actually get written directly to another server entirely. That's really not feasible in your case.

[Webdiggity]

I aim to fix all that. The new rig is one the way and I'm feeling like a complete dumbass! I did learn that my system does do a daily update of installed packages however there was a one week period where I didn't have the kernel upgraded when I should have. I fear that my system has been compromised.

I'm still getting the trojan horse errors and it says the binary is in SK?
Hidden PID

Does this make any sense?

[Void Main]

I agree that it seems your system has been comprimised but I don't believe not upgrading the kernel in a week is the cause of it. There have been no security alerts of vulnerabilities in the kernel in the last week that I am aware of (unless you know of one that I haven't heard about).

[Webdiggity]

Well, you were right. It was a rootkit. I was able to find it because cPanel did a good job of letting me know where it was. I looked in my E-mail account off server where the root messages also get sent and right there is was pointing to that wierd sk directory.

File modifications on it coincide with the date and time the files on my home directory were modified.

I'm uploading it to your server now, Void. It should reveal what was actually done to my system right?

Thanks,
Webby

[Void Main]

I'm uploading it to your server now, Void. It should reveal what was actually done to my system right?

Well, it might at least tell us what was done with that portion of it. It's hard to tell if there isn't more to it at this point though. I'll start looking it over and see what I can come up with.

EDIT: I guess you haven't uploaded it yet. Let me know when you do...

[Webdiggity]

Yeah, I uploaded it. archive.tar in the /upload directory