Linux - Finding Port Listeners
As a web host I am having trouble with CE5.5 and IRC bots making it onto the server. Hours and hours of liooing brought me to this example but it only comes close.
First we need to check the status of our ports:
netstat -na | grep "ESTABLISHED"
tcp 0 0 192.168.0.1:39701 4.3.2.1:6667 ESTABLISHED
I See and not a single 6667:
tcp 0 0 xx.xx.xx71:22 202.75.219.248:41251 ESTABLISHED
tcp 0 0 xx.xx.xx88:25 222.124.115.159:11202 ESTABLISHED
tcp 0 0 xx.xx.xx71:110 174.252.195.168:4036 ESTABLISHED
tcp 0 0 xx.xx.xx95:22 202.75.219.248:41833 ESTABLISHED
tcp 0 0 xx.xx.xx85:22 202.75.219.248:56912 ESTABLISHED
tcp 0 0 xx.xx.xx79:22 202.75.219.248:51889 ESTABLISHED
tcp 0 326 xx.xx.xx71:36584 66.42.239.82:25 ESTABLISHED
tcp 0 326 xx.xx.xx71:36585 66.42.239.82:25 ESTABLISHED
tcp 0 326 xx.xx.xx71:36586 66.42.239.82:25 ESTABLISHED
tcp 0 0 xx.xx.xx71:993 107.10.100.129:49181 ESTABLISHED
tcp 0 0 xx.xx.xx71:25 66.42.239.94:43499 ESTABLISHED
tcp 0 23 xx.xx.xx87:22 202.75.219.248:56040 ESTABLISHED
tcp 0 0 xx.xx.xx83:22 202.75.219.248:35805 ESTABLISHED
tcp 0 0 xx.xx.xx86:22 202.75.219.248:49188 ESTABLISHED
tcp 0 0 xx.xx.xx88:22 202.75.219.248:58410 ESTABLISHED
tcp 0 352 xx.xx.xx71:22 71.84.230.166:1858 ESTABLISHED
tcp 0 0 xx.xx.xx80:22 202.75.219.248:46419 ESTABLISHED
tcp 0 0 xx.xx.xx71:143 174.252.200.2:6640 ESTABLISHED
tcp 0 0 xx.xx.xx71:143 174.253.132.110:5165 ESTABLISHED
tcp 0 0 xx.xx.xx71:143 174.253.132.110:5182 ESTABLISHED
udp 0 0 127.0.0.1:39144 127.0.0.1:39144 ESTABLISHED
lsof -Pnl +M -i4 | grep 6667
modcluste 3959 10224 1u IPv4 12002 TCP 192.168.0.1:39701->4.3.2.1:6667 (ESTABLISHED)
I see:
[root@cis-secure ~]# lsof -Pnl +M -i4 | grep 6667
perl 7184 48 90u IPv4 607481368 TCP xx.xx.xx.71:36610->38.108.80.105:6667 (SYN_SENT)
The program name listed above is ?perl? but that name may be cut short. To find out it?s full name
we can run the following name using the process id (from the above output) ?3959?.
ps aux 3959
sally 3959 0.0 0.0 1988 852 ? Ss Feb05 0:00 perl
I see the fill output of PS not just the PID indicated.
Now we can see the complete name. We can also see that the user ?sally? started the application.
Most likely her account was hacked. We can find where that application is located by using either (or both) of the following commands:
NONE OF THIS APPLIES OR WORKS:
locate modclusterd
find / | grep modclusterd
The output might be like follows:
/var/tmp/.a/.kde/modclusterd
The above looks like a cracker was here. So lets kill it and remove it.
kill -9 3959
rm -rf /var/tmp/.a
These damn IRC bots are the death of me!
Any help would be gratefully accepted.
Cheers.
Finding IRC Bots and other port listeners
Re: Finding IRC Bots and other port listeners
You need to take the machine offline ASAP. You've been hacked and you have no idea what they've installed on your machine nor what data is being siphoned off. In the worst case your machine might currently be hosting illegal material and you would be liable for it. Take it offline and work out a plan.
For a start, running CentOS 5.5 is probably the cause of the problem - current release is 5.8 and earlier releases have numerous security related bugs including kernel root exploits. The best way to proceed at this point is to back up any data that you care about and reformat and reinstall on the current release. Also review any non-CentOS software that you have installed and make sure that they're at the latest levels.
It's not safe to attempt to cleanup whatever was done to your machine.
For a start, running CentOS 5.5 is probably the cause of the problem - current release is 5.8 and earlier releases have numerous security related bugs including kernel root exploits. The best way to proceed at this point is to back up any data that you care about and reformat and reinstall on the current release. Also review any non-CentOS software that you have installed and make sure that they're at the latest levels.
It's not safe to attempt to cleanup whatever was done to your machine.
Re: Finding IRC Bots and other port listeners
Thank you for the reply but that is quite an over reaction. An IRC bot is generally harmless. I also mentioned that I black-holed the IP's so even though they think they are communicating the firewall is stopping the connection. I'm not sure that 5.5 vs. 5.8 is all that much of a difference. Besides, if you give up tun the machine off that makes it virtually impossible to try to locate the problem. IRC hacks are generally uploaded to a directory with -perm 0777. As a general rule that act as a spam gateway. Especially since ps tells us so:
perl 7184 apache 90u IPv4 608934271 TCP cis-secure.xcvbnjhgf.net:51094->mail.cdconsultants.ca:ircd (SYN_SENT)
I wonder if we might get back to the original question and the commands previously listed.
perl 7184 apache 90u IPv4 608934271 TCP cis-secure.xcvbnjhgf.net:51094->mail.cdconsultants.ca:ircd (SYN_SENT)
I wonder if we might get back to the original question and the commands previously listed.
-
DaemonProgrammr
- Posts: 78
- Joined: 2011/12/12 12:49:46
Re: Finding IRC Bots and other port listeners
Paranoid as it might seem: TrevorH night have a point.
Those IRC bots don't pop-up from mid-air. Somebody / Something has installed them and if you're hacked, there's a 100 to 1 chance (add zero's at your own leisure) that there's a rootkit
you're not detecting. Killing bots is just erasing the symptoms.
Back-up any code, databases and other data and rebuild a server. (Nuke and pave)
Those IRC bots don't pop-up from mid-air. Somebody / Something has installed them and if you're hacked, there's a 100 to 1 chance (add zero's at your own leisure) that there's a rootkit
you're not detecting. Killing bots is just erasing the symptoms.
Back-up any code, databases and other data and rebuild a server. (Nuke and pave)
Re: Finding IRC Bots and other port listeners
You are both partially correct but you miss several key points. The most important point is that backing up the serving, doing a clean install, and then restoring the backup only restores the compromised site!. One has to be able to locate it before backing up. We are actually about to upgrade to Pesk 10.4.4 from Plesk 8.x and 9.x along with Virtuozzo 4. Plesk actually has admitted that a compromise was discovered dating back to 2007 and is in all versions except the absolute latest. It is an injection hack that allows the bots to discover the user FTP password and then they upload GootKit DDoS scripts designed to cripple each server.
We know that the bots obtain the FTP user password and upload the content to directories demanded to be chmod 0777 (Joomla and others) not caring about security. We located all of the compromised sites using scripts that I wrote, deleted the files, and then turned off CGI/Perl.
The 6667 hack is problematic because we cannot locate where it is on two servers. We can lsof -p or -i and still find nothing. I was hoping a more experienced admin would be able share some tricks. These are pissy little buggers.
Our firewall blocks the entire range and the attack, although calling an IP off the server, sin't actually able to dump its payload.
We know that the bots obtain the FTP user password and upload the content to directories demanded to be chmod 0777 (Joomla and others) not caring about security. We located all of the compromised sites using scripts that I wrote, deleted the files, and then turned off CGI/Perl.
The 6667 hack is problematic because we cannot locate where it is on two servers. We can lsof -p or -i and still find nothing. I was hoping a more experienced admin would be able share some tricks. These are pissy little buggers.
Our firewall blocks the entire range and the attack, although calling an IP off the server, sin't actually able to dump its payload.
Re: Finding IRC Bots and other port listeners
[quote]
The 6667 hack is problematic because we cannot locate where it is on two servers. We can lsof -p or -i and still find nothing. I was hoping a more experienced admin would be able share some tricks
[/quote]
This means that they have the ability to hide processes and you can no longer trust [u]anything[/u] on this system. They have root access and have loaded kernel modules to hide things. The system is no longer yours, it's theirs.
The 6667 hack is problematic because we cannot locate where it is on two servers. We can lsof -p or -i and still find nothing. I was hoping a more experienced admin would be able share some tricks
[/quote]
This means that they have the ability to hide processes and you can no longer trust [u]anything[/u] on this system. They have root access and have loaded kernel modules to hide things. The system is no longer yours, it's theirs.
Re: Finding IRC Bots and other port listeners
I would like to be respectfully candid. I asked a question about locating what resources were being used by an IRC bot and over a weeks time not one person has attempted to answer the question. As a web host with 25 shared servers and another 33 dedicated servers I can tell you from my vast experi4ence that IRC bots are uploaded to the server using malformed URL's that are attacking weaknesses in phpBB, WordPress, calendars, and any number of personally coded scripts by end users that don't know how to write a script with security in mind. Since 1997 I have never had a server root by a IRC bot script. Port 6667 is blocked full time on our firewall. Our firewall is an extremely robust and effect Watchguardguard X8000 Peak loaded with all features.
[quote]
This means that they have the ability to hide processes and you can no longer trust anything on this system. They have root access and have loaded kernel modules to hide things. The system is no longer yours, it's theirs.
[/quote]
When asking more experience users for help I find that arguing with them is counter-productive. But my question on HOW TO remains unanswered. I've even gone as far as stating that we are running a hardware firewall that is trapping all content in or out from port 6667, yet one of you writes that the box is rooted. Format, reload the OS and restore backups. You are supposed to be more experienced and one would assume that you'd realize that if the site affected was backed up before the last backup restoring it on a clean server will only restore the breach.
I am going to say that I believe you are incorrect and have assumed. The way the script makes it onto the server is quite simple and made even simpler by apps like WordPress demanding DIR's be 0777 - stupid! Ridiculously stupid. It is my opinion that such a script should 644 the DIR and only change it to 0777 when actually needed and the program controls the change. But heck, the users want to freely upload photos. As you know the upload goes to /tmp or /var/tmp. All of our servers are on Virtuozzo 3 for the base OS but each partition is Centos 5.5 - fairly current. The hack is inside the container. Once the file(s) is uploaded it is expanded in /tmp that then writes it back to the actual hacked site and executable. Even running
[code]
BINDMOUNT="/tmp,nosuid,noexec,nodev /var/tmp,nosuid,noexec,nodev "
[/code]
So, as you may see your deduction that the server is rooted is flawed. It is far easier. Back to my original question.
If I lsof -i :6667 I get:
[code]
netstat -na | grep "ESTABLISHED"
tcp 0 0 192.168.0.1:39701 4.3.2.1:6667 ESTABLISHED
lsof -Pnl +M -i4 | grep 6667
modcluste 3959 10224 1u IPv4 12002 TCP 192.168.0.1:39701->4.3.2.1:6667 (ESTABLISHED)
ps aux 3959
sally 3959 0.0 0.0 1988 852 ? Ss Feb05 0:00 modclusterd
[/code]
This exercise yields no useful information. I recall that when on Ensim years ago it did work. Now Plesk 8.x, 9.x or 10.x -- mature software the latter is, the hack still makes it on board. It could also be as simple as a PHP flaw not yet detected. All of this bleeding edge makes being a web host a lot of work. I can isolate the affected server and just shut the port off on the manged switch. Our 192.168 network still gives me access to troublehoot over the private connection. All of these toys di little good if I cannot find the source.
I need to humbly and respectfully ask that more experienced users offer some guidance of tracing :6667 to its actual web site.
Lastly, we run numerous hack catching scripts including a daily run of rkhunter that is built into Plesk. Sometimes it will reveal a password-less account and an easy fix. Usually it finds noting wrong. I've tried writing a script that will check the crontab for each site but the code escapes my Neophyte ways. Are there more ways, perhaps direct in nature, that will immediately locate the hack? I can read logs until I am blue. I can run grep commands for the easy like WGET or LWP. Sometimes I get lucky. I can often sort by bandwidth and it is more effective. Sometimes watching top helps. Buy thus far no practical solution.
If I can I'd reallt like a fair shot at talking with you guru's that aren't predisposed to a root server. Virtuozzo already has a build-in detect and restore system so that oif WGET were modified it would as a threat.
All of our servers are being upgraded to Virtuozzo 4.x, Plesk 10.4.4. The Virtuozzo base will be CE 5.x or 6 if it is allowed. The same with the containers.
[quote]
This means that they have the ability to hide processes and you can no longer trust anything on this system. They have root access and have loaded kernel modules to hide things. The system is no longer yours, it's theirs.
[/quote]
When asking more experience users for help I find that arguing with them is counter-productive. But my question on HOW TO remains unanswered. I've even gone as far as stating that we are running a hardware firewall that is trapping all content in or out from port 6667, yet one of you writes that the box is rooted. Format, reload the OS and restore backups. You are supposed to be more experienced and one would assume that you'd realize that if the site affected was backed up before the last backup restoring it on a clean server will only restore the breach.
I am going to say that I believe you are incorrect and have assumed. The way the script makes it onto the server is quite simple and made even simpler by apps like WordPress demanding DIR's be 0777 - stupid! Ridiculously stupid. It is my opinion that such a script should 644 the DIR and only change it to 0777 when actually needed and the program controls the change. But heck, the users want to freely upload photos. As you know the upload goes to /tmp or /var/tmp. All of our servers are on Virtuozzo 3 for the base OS but each partition is Centos 5.5 - fairly current. The hack is inside the container. Once the file(s) is uploaded it is expanded in /tmp that then writes it back to the actual hacked site and executable. Even running
[code]
BINDMOUNT="/tmp,nosuid,noexec,nodev /var/tmp,nosuid,noexec,nodev "
[/code]
So, as you may see your deduction that the server is rooted is flawed. It is far easier. Back to my original question.
If I lsof -i :6667 I get:
[code]
netstat -na | grep "ESTABLISHED"
tcp 0 0 192.168.0.1:39701 4.3.2.1:6667 ESTABLISHED
lsof -Pnl +M -i4 | grep 6667
modcluste 3959 10224 1u IPv4 12002 TCP 192.168.0.1:39701->4.3.2.1:6667 (ESTABLISHED)
ps aux 3959
sally 3959 0.0 0.0 1988 852 ? Ss Feb05 0:00 modclusterd
[/code]
This exercise yields no useful information. I recall that when on Ensim years ago it did work. Now Plesk 8.x, 9.x or 10.x -- mature software the latter is, the hack still makes it on board. It could also be as simple as a PHP flaw not yet detected. All of this bleeding edge makes being a web host a lot of work. I can isolate the affected server and just shut the port off on the manged switch. Our 192.168 network still gives me access to troublehoot over the private connection. All of these toys di little good if I cannot find the source.
I need to humbly and respectfully ask that more experienced users offer some guidance of tracing :6667 to its actual web site.
Lastly, we run numerous hack catching scripts including a daily run of rkhunter that is built into Plesk. Sometimes it will reveal a password-less account and an easy fix. Usually it finds noting wrong. I've tried writing a script that will check the crontab for each site but the code escapes my Neophyte ways. Are there more ways, perhaps direct in nature, that will immediately locate the hack? I can read logs until I am blue. I can run grep commands for the easy like WGET or LWP. Sometimes I get lucky. I can often sort by bandwidth and it is more effective. Sometimes watching top helps. Buy thus far no practical solution.
If I can I'd reallt like a fair shot at talking with you guru's that aren't predisposed to a root server. Virtuozzo already has a build-in detect and restore system so that oif WGET were modified it would as a threat.
All of our servers are being upgraded to Virtuozzo 4.x, Plesk 10.4.4. The Virtuozzo base will be CE 5.x or 6 if it is allowed. The same with the containers.
Re: Finding IRC Bots and other port listeners
Since you insist on posting nothing useful I find it very difficult to even try to help. Posting the output of a bunch of commands that you've found on a web page from someone else's system does absolutely no good since the output from those commands is dependent on [u]their systemp[/u]. I'm basing this on the fact that google finds your [u]exact[/u] command output on http://agix.com.au/blog/?p=2776 and reposting that is useless since it has nothing whatsoever to do with your system.
I am presuming that you have tracked back the outbound port 6667 connection from your firewall to the host that you are looking at. If so then you need to check that system. I'm going to guess that you are the person who replied on that agix web page with at least some details of the real system and that the `lsof -Pnl +M -i4 | grep 6667` shows the connection outbound to 38.108.80.105:6667. The ps command posted on that page will always list all processes because that's what "aux" asks it to do! You'll be more likely to find something useful in `ps 7184` assuming that it is still the same PID or in /proc/7184 and it's files and subdirectories.
The reason that you can run `netstat -na | grep ?ESTABLISHED?` and not see the connection to port 6667 is because on your system it is not ESTABLISHED, it's in SYN_SENT so no successful connection has been established.
What you do see in your output on that other web page is a process
[quote]
7184 ? S 0:00 /usr/sbin/apache/log
[/quote]
which means that's what is running and trying to connect to port 6667 on 38.108.80.105.
Unfortunately CentOS 5.5 may be "fairly" current but the kernels that go with it still have local root exploits so once someone can run a process on your local machine they can use those to get root access.
There is no file /usr/sbin/apache/log supplied with any version of CentOS 5. Searching google for that leads to several web pages like http://pdw.weinstein.org/2010/06/damn-script-kiddies-get-off-my-lawn.html
I still stand by what I said about your system no longer being yours - it belongs to the crackers. There's no point in repeating myself - I know what I'd be doing right now and I still advise you to do the same.
I am presuming that you have tracked back the outbound port 6667 connection from your firewall to the host that you are looking at. If so then you need to check that system. I'm going to guess that you are the person who replied on that agix web page with at least some details of the real system and that the `lsof -Pnl +M -i4 | grep 6667` shows the connection outbound to 38.108.80.105:6667. The ps command posted on that page will always list all processes because that's what "aux" asks it to do! You'll be more likely to find something useful in `ps 7184` assuming that it is still the same PID or in /proc/7184 and it's files and subdirectories.
The reason that you can run `netstat -na | grep ?ESTABLISHED?` and not see the connection to port 6667 is because on your system it is not ESTABLISHED, it's in SYN_SENT so no successful connection has been established.
What you do see in your output on that other web page is a process
[quote]
7184 ? S 0:00 /usr/sbin/apache/log
[/quote]
which means that's what is running and trying to connect to port 6667 on 38.108.80.105.
Unfortunately CentOS 5.5 may be "fairly" current but the kernels that go with it still have local root exploits so once someone can run a process on your local machine they can use those to get root access.
There is no file /usr/sbin/apache/log supplied with any version of CentOS 5. Searching google for that leads to several web pages like http://pdw.weinstein.org/2010/06/damn-script-kiddies-get-off-my-lawn.html
I still stand by what I said about your system no longer being yours - it belongs to the crackers. There's no point in repeating myself - I know what I'd be doing right now and I still advise you to do the same.
-
pschaff
- Retired Moderator
- Posts: 18276
- Joined: 2006/12/13 20:15:34
- Location: Tidewater, Virginia, North America
- Contact:
Finding IRC Bots and other port listeners
[quote]
krazybob wrote:
... each partition is Centos 5.5 - fairly current. ...[/quote]
[code]Release Initial kernel Release Date
5.0 2.6.18-8.el5 2007-04-12
5.1 2.6.18-53.el5 2007-12-02
5.2 2.6.18-92.el5 2008-06-24
5.3 2.6.18-128.el5 2009-03-31
5.4 2.6.18-164.el5 2009-10-21
[color=ff0000] 5.5 2.6.18-194.el5 2010-05-14[/color]
5.6 2.6.18-238.el5 2011-04-08
5.7 2.6.18-274.el5 2011-09-13
5.8 2.6.18-308.el5 2012-03-07[/code]
The latest update kernel-2.6.18-308.1.1.el5 is the 23rd kernel released since the original 5.5, there have been 14 additional glibc releases up to glibc-2.5-81.el5_8.1, and literally hundreds of other errata releases, many of Important or Critical severity. Hardly "fairly current".
krazybob wrote:
... each partition is Centos 5.5 - fairly current. ...[/quote]
[code]Release Initial kernel Release Date
5.0 2.6.18-8.el5 2007-04-12
5.1 2.6.18-53.el5 2007-12-02
5.2 2.6.18-92.el5 2008-06-24
5.3 2.6.18-128.el5 2009-03-31
5.4 2.6.18-164.el5 2009-10-21
[color=ff0000] 5.5 2.6.18-194.el5 2010-05-14[/color]
5.6 2.6.18-238.el5 2011-04-08
5.7 2.6.18-274.el5 2011-09-13
5.8 2.6.18-308.el5 2012-03-07[/code]
The latest update kernel-2.6.18-308.1.1.el5 is the 23rd kernel released since the original 5.5, there have been 14 additional glibc releases up to glibc-2.5-81.el5_8.1, and literally hundreds of other errata releases, many of Important or Critical severity. Hardly "fairly current".
Re: Finding IRC Bots and other port listeners
[quote]
TrevorH wrote:
Since you insist on posting nothing useful I find it very difficult to even try to help.
[/quote]
This has got to be the rudest site I have ever had to deal with. I did not just find commands. I use commands. Just because I use suggested commands form another web page does not mean they are not useful to me. They just don't go far enough and even stated that they were of little help. Rather than actually trying to help you put down those seeking to learn. For obvious reasons I am not posting my own IP's.
I use lsof -i :6667 usually followed lsof -p followed by whatever clue may exist. At least I am trying to find resources (answers) before coming here to get swmacked around.
Not one of you has actually answered the question. None of you. Let me remind you again what it was. What are some methods of tracking down the the ingress of the 6667 hack? You all assumed the server was rooted but cannot refute that in a web hosting environment it came in just as I said. Through a web site with a crappy script or 0777 perms on a DIR. Perhaps other damage has been done, although rkhunter says not. [b]So please try to answer the actual question without editorializing on what you think of me.[/b] No, I am not at your level. That's why I am here. Seeking knowledge that you ridicule and put me down for.
What would you be doing right now? Reformatting and restoring from backups -- the web site(s) that contain the hack included. The goal is to find that site, remove the hack and then do a partial backup before reinstalling a fresh OS.
Please take serious note that Parallels releases patches for Virtuozzo and one does not just grab a kernel off a web page and apply it. One applies the kernel update from Parallels.
TrevorH wrote:
Since you insist on posting nothing useful I find it very difficult to even try to help.
[/quote]
This has got to be the rudest site I have ever had to deal with. I did not just find commands. I use commands. Just because I use suggested commands form another web page does not mean they are not useful to me. They just don't go far enough and even stated that they were of little help. Rather than actually trying to help you put down those seeking to learn. For obvious reasons I am not posting my own IP's.
I use lsof -i :6667 usually followed lsof -p followed by whatever clue may exist. At least I am trying to find resources (answers) before coming here to get swmacked around.
Not one of you has actually answered the question. None of you. Let me remind you again what it was. What are some methods of tracking down the the ingress of the 6667 hack? You all assumed the server was rooted but cannot refute that in a web hosting environment it came in just as I said. Through a web site with a crappy script or 0777 perms on a DIR. Perhaps other damage has been done, although rkhunter says not. [b]So please try to answer the actual question without editorializing on what you think of me.[/b] No, I am not at your level. That's why I am here. Seeking knowledge that you ridicule and put me down for.
What would you be doing right now? Reformatting and restoring from backups -- the web site(s) that contain the hack included. The goal is to find that site, remove the hack and then do a partial backup before reinstalling a fresh OS.
Please take serious note that Parallels releases patches for Virtuozzo and one does not just grab a kernel off a web page and apply it. One applies the kernel update from Parallels.