SSH issue, Cannot allocate memory

General support questions
Post Reply
roadtodebian
Posts: 3
Joined: 2020/06/25 06:43:02

SSH issue, Cannot allocate memory

Post by roadtodebian » 2020/06/29 03:52:51

Hello all,

I have azure subscription, and build centos 7 vm. I'm installed apache, postgresql, and php laravel on top of it. For remote administration and file transfer i'm using ssh and sftp. But i got a problem. SSHD service suddenly closed in random time, for average one in 1.5 day. I checked the secure log, it shows:
Jun 28 23:50:43 HCIS-App sshd[5016]: Did not receive identification string from 172.30.18.10 port 34477
Jun 29 00:00:43 HCIS-App sshd[1198]: error: fork: Cannot allocate memory
Jun 29 00:10:43 HCIS-App sshd[1198]: error: fork: Cannot allocate memory
Jun 29 00:20:43 HCIS-App sshd[1198]: error: fork: Cannot allocate memory
i try to add swap partition, install telnet service for secondary remote, disable selinux and firewall, but no luck. The error message also seen on other services, like cron, postfix. But, the website (httpd) still working.

Postfix
Jun 28 07:50:08 HCIS-App postfix/postfix-script[21501]: starting the Postfix mail system
Jun 28 07:50:08 HCIS-App postfix/master[21505]: daemon started -- version 2.10.1, configuration /etc/postfix
Jun 29 00:30:48 HCIS-App postfix/master[21505]: warning: master_spawn: fork: Cannot allocate memory -- throttling
Jun 29 00:31:48 HCIS-App postfix/master[21505]: warning: master_spawn: fork: Cannot allocate memory -- throttling
Jun 29 00:32:29 HCIS-App postfix/master[21505]: warning: unix_trigger_event: read timeout for service public/pickup
Jun 29 00:32:48 HCIS-App postfix/master[21505]: warning: master_spawn: fork: Cannot allocate memory -- throttling
Jun 29 00:33:48 HCIS-App postfix/master[21505]: warning: master_spawn: fork: Cannot allocate memory -- throttling

Cron
Jun 28 23:52:01 HCIS-App CROND[15165]: (hcisuser) CMD (php /var/www/html/hcconnQG/artisan schedule:run >> /dev/null 2>&1)
Jun 28 23:53:01 HCIS-App CROND[22174]: (hcisuser) CMD (php /var/www/html/hcconnQG/artisan schedule:run >> /dev/null 2>&1)
Jun 28 23:53:01 HCIS-App CROND[28456]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jun 28 23:53:01 HCIS-App CROND[11524]: (CRON) CAN'T FORK (child_process): Cannot allocate memory
Jun 28 23:54:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:54:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:55:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:55:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:56:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:56:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:57:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:57:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:58:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:58:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:59:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 28 23:59:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 29 00:00:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
Jun 29 00:00:01 HCIS-App crond[19020]: (CRON) CAN'T FORK (do_command): Cannot allocate memory
------------------etc--------

httpd
[Sun Jun 28 03:14:02.172761 2020] [mpm_prefork:notice] [pid 1199] AH00163: Apache/2.4.6 (CentOS) PHP/7.0.33 configured -- resuming normal operations
[Sun Jun 28 03:14:02.172768 2020] [core:notice] [pid 1199] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sun Jun 28 11:22:07.323874 2020] [cgi:error] [pid 31090] [client aaa.bbb.ccc.ddd:59666] script not found or unable to stat: /var/www/cgi-bin/mainfunction.cgi
[Sun Jun 28 13:19:50.245031 2020] [:error] [pid 11882] [client aaa.bbb.ccc.ddd:55412] script '/var/www/html/setup.php' not found or unable to stat
[Mon Jun 29 00:08:17.833962 2020] [mpm_prefork:error] [pid 1199] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Mon Jun 29 02:01:35.258657 2020] [:error] [pid 12368] [client aaa.bbb.ccc.ddd:53840] script '/var/www/html/index.php' not found or unable to stat
[Mon Jun 29 03:06:49.993991 2020] [cgi:error] [pid 15964] [client aaa.bbb.ccc.ddd:42317] script not found or unable to stat: /var/www/cgi-bin/mainfunction.cgi
[Mon Jun 29 03:06:56.919207 2020] [cgi:error] [pid 6717] [client aaa.bbb.ccc.ddd:42348] script not found or unable to stat: /var/www/cgi-bin/mainfunction.cgi
[Mon Jun 29 03:07:10.343073 2020] [cgi:error] [pid 5685] [client aaa.bbb.ccc.ddd:42379] script not found or unable to stat: /var/www/cgi-bin/mainfunction.cgi
[Mon Jun 29 05:04:17.019705 2020] [mpm_prefork:error] [pid 1199] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Mon Jun 29 05:08:53.304324 2020] [mpm_prefork:error] [pid 1199] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Mon Jun 29 05:14:29.648909 2020] [mpm_prefork:error] [pid 1199] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Mon Jun 29 05:36:47.059635 2020] [mpm_prefork:error] [pid 1199] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Mon Jun 29 05:41:05.320885 2020] [mpm_prefork:error] [pid 1199] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process
[Mon Jun 29 05:51:55.920251 2020] [:error] [pid 6717] [client aaa.bbb.ccc.ddd:45158] script '/var/www/html/azenv.php' not found or unable to stat
[Mon Jun 29 05:54:47.186817 2020] [mpm_prefork:error] [pid 1199] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: SSH issue, Cannot allocate memory

Post by TrevorH » 2020/06/29 09:12:18

i try to add swap partition, install telnet service for secondary remote, disable selinux and firewall, but no luck. The error message also seen on other services, like cron, postfix. But, the website (httpd) still working.
You installed *telnet*? Is this 1970? Do not do that. Or better yet, if you did it, undo it ASAP.

If you're running a website then selinux is your friend and you shouldn't disable it. So is the firewall. If you've left your system exposed like this for a long time - more than a few hours - then you should probably consider starting over on a new machine as this one is quite possibly now compromised.

Everything else you've posted appears to show that you are suffering from a lack of memory. How much RAM does this VM have?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

roadtodebian
Posts: 3
Joined: 2020/06/25 06:43:02

Re: SSH issue, Cannot allocate memory

Post by roadtodebian » 2020/06/29 09:58:11

I though when sshd error, i can still remote via telnet, but the telnet daemon got error too. I set ssh remote access only available from private ip by using vpn. My CentOS have 16GB of ram, and i'm using SNMP to monitor cpu, memory, disk space. Memory still available around 70% during peak time, and swap memory still 100% free.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: SSH issue, Cannot allocate memory

Post by TrevorH » 2020/06/29 13:01:37

Running a telnet daemon is unsafe. Do not do it. All data is transmitted in plain text and is inherently unsafe. Stop the service, remove the package, step away from the telnet.
Memory still available around 70% during peak time, and swap memory still 100% free.
Apparently whatever tells you that is lying.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

roadtodebian
Posts: 3
Joined: 2020/06/25 06:43:02

Re: SSH issue, Cannot allocate memory

Post by roadtodebian » 2020/07/01 06:28:56

I'm already uninstall telnetd. "Cannot allocate memory" error still persisted. Any suggestion or log files need to be check?

Post Reply