webserver running out of memory after updating from httpd-2.4.6-97.el7.centos.x86_64 to httpd-2.4.6-97.el7.centos.1.x86

General support questions
Post Reply
leifh
Posts: 24
Joined: 2009/02/24 21:19:12
Location: Norway

webserver running out of memory after updating from httpd-2.4.6-97.el7.centos.x86_64 to httpd-2.4.6-97.el7.centos.1.x86

Post by leifh » 2021/11/09 10:05:13

Hi
A CentOS7 machine of mine has started acting troubled after an update last Tuesday.

The server is a webserver and until the update it worked without issues.

httpd version working ok:
httpd-2.4.6-97.el7.centos.x86_64

httpd version now running (with issues)
httpd-2.4.6-97.el7.centos.1.x86_64

Tried initially to increase memory from 80GB to 125GB, but it still ate it all.

Before the update the server always had 30+GB available (free memory + buffers/cache).

After the update all memory is gone in 24 hours.
It seems the number of running threads just keeps growing.
The memory gets freed when stopping the webserver.

The server uses the mpm worker.
Below are the mpm worker config and it's been running for years (but I am now starting to wonder if some values are to high)

<IfModule worker.c>
ServerLimit 120
StartServers 20
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestWorkers 3000
MaxConnectionsPerChild 6000
</IfModule>

Anyone having had similar trouble or a tip how to troubleshoot?

pjsr2
Posts: 614
Joined: 2014/03/27 20:11:07

Re: webserver running out of memory after updating from httpd-2.4.6-97.el7.centos.x86_64 to httpd-2.4.6-97.el7.centos.1

Post by pjsr2 » 2021/11/09 13:35:34

Sounds like something is leaking memory. Reduce the value of MaxConnectionsPerChild, this will make that children get restarted more often, thus reducing the impact of a memory leak.

leifh
Posts: 24
Joined: 2009/02/24 21:19:12
Location: Norway

Re: webserver running out of memory after updating from httpd-2.4.6-97.el7.centos.x86_64 to httpd-2.4.6-97.el7.centos.1

Post by leifh » 2021/11/09 14:06:11

Hi, thanks for the headsup.
Have changed the configuration, now running

<IfModule worker.c>
ServerLimit 100
StartServers 20
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestWorkers 2000
MaxConnectionsPerChild 500
</IfModule>

If problem still persist I will try downgrading back to httpd-2.4.6-97.el7.centos.x86_64 to verify the new version is to blame.

leifh
Posts: 24
Joined: 2009/02/24 21:19:12
Location: Norway

Re: webserver running out of memory after updating from httpd-2.4.6-97.el7.centos.x86_64 to httpd-2.4.6-97.el7.centos.1

Post by leifh » 2021/11/10 09:03:28

The new config did not seem to help.
Downgraded to httpd-2.4.6-97.el7.centos.x86_64 12 hours ago memory has been freed since once (something that did not happen once using httpd-2.4.6-97.el7.centos.1.x86_64). Will keep monitoring.

If memory consumption now stabilizes will try with the old mpm worker config so the only change on the webserver is the apache version.

leifh
Posts: 24
Joined: 2009/02/24 21:19:12
Location: Norway

Re: webserver running out of memory after updating from httpd-2.4.6-97.el7.centos.x86_64 to httpd-2.4.6-97.el7.centos.1

Post by leifh » 2021/11/12 11:00:24

With the current configuration using httpd-2.4.6-97.el7.centos.x86_64 it now seems the webserver is behaving.
Will let it run for a few more days to be sure.

Saw httpd-2.4.6-97.el7.centos.x86_64 was released 2021-11-11 so might try it out.
https://access.redhat.com/errata/RHBA-2021:4606

leifh
Posts: 24
Joined: 2009/02/24 21:19:12
Location: Norway

Re: webserver running out of memory after updating from httpd-2.4.6-97.el7.centos.x86_64 to httpd-2.4.6-97.el7.centos.1

Post by leifh » 2021/12/09 07:42:56

Everything ok with the updated version httpd-2.4.6-97.el7.centos.2.x86_64

Post Reply