mysql crashes Out of memory: Kill process 5290

Issues related to applications and software problems
Post Reply
bigklay
Posts: 5
Joined: 2021/05/12 15:00:53

mysql crashes Out of memory: Kill process 5290

Post by bigklay » 2021/05/12 15:24:01

I have a lamp server that is having an issue with mysql crashing. I monitor the server and watch the oom_score of mysql as that seems to predict the failure. I backup databases on the server weekly. Every time I backup one of the databases the oom_score increases substantially from say 40 which is normal post service restart to 55, then if I back up a second database, it raises to 75-76. If I restart mysql it returns to 40.

+-----------------+
| version() |
+-----------------+
| 10.0.38-MariaDB |
+-----------------+
Error code when it crashes.
May 8 02:25:36 www kernel: Out of memory: Kill process 5290 (mysqld) score 49 or sacrifice child
May 8 02:25:36 www kernel: Killed process 5290 (mysqld), UID 996, total-vm:5360004kB, anon-rss:125876kB, file-rss:28kB, shmem-rss:0kB
May 8 02:25:36 www kernel: httpd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
May 8 02:25:37 www kernel: httpd cpuset=/ mems_allowed=0

Has anyone seen anything like this?

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: mysql crashes Out of memory: Kill process 5290

Post by TrevorH » 2021/05/12 16:10:33

What does your /etc/my.cnf (and /etc/my.cnf.d/*) file look like?

And where did you get mariadb 10.0.38 from? It's not something we ship and nor is it in any of my installed repos (which have 5.5.68 from base/updates, 10.1.41, 10.2.30, 10.3.21 and 10.4.17 from IUS, 10.1.29, 10.2.33, 10.3.27 and 10.5.8 from SCL)
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

bigklay
Posts: 5
Joined: 2021/05/12 15:00:53

Re: mysql crashes Out of memory: Kill process 5290

Post by bigklay » 2021/05/12 16:17:40

MariaDB [(none)]> select version();
+-----------------+
| version() |
+-----------------+
| 10.0.38-MariaDB |
+-----------------+
1 row in set (0.00 sec)

my.cnf looks like this. I added the innodb_buffer_pool_size=4G after increasing the vm's ram to 8 gb which I know is overkill but the error persists.

[root@www etc]# free -h
total used free shared buff/cache available
Mem: 7.6G 1.1G 5.3G 10M 1.2G 6.3G
Swap: 2.0G 0B 2.0G
[root@www etc]# cat /proc/1537/oom_score
77 <current score


[root@www etc]# cat my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[mysqld]
innodb_buffer_pool_size=4G
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: mysql crashes Out of memory: Kill process 5290

Post by TrevorH » 2021/05/12 16:40:30

I saw what version you were running in the first post but you still didn't say where it came from. It's not a version we supply. And since it's from Jan 2019 you should probably look at it and find out what's been fixed in 10.0.x since then.

Do you have any config files under /etc/my.cnf.d as those will be included from your my.cnf and could add/override things.

Also, what makes you think that 77 is a high score? I just checked 4 of my mariadb servers and they have cores of 96, 96, 233 and 237. None of them have ever been killed by the OOM killer in more than 10 years of use.

You can also use oom_score_adj to make it less likely that a process will be killed by the OOM killer.

cat /proc/$(pidof mysqld)/oom_score
echo "-100" > /proc/$(pidof mysqld)/oom_score_adj
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

bigklay
Posts: 5
Joined: 2021/05/12 15:00:53

Re: mysql crashes Out of memory: Kill process 5290

Post by bigklay » 2021/05/12 17:49:28

77 is not that bad, but doing two db backups raised it from 35-40 to 77. I am not sure what the score is before it dies, as it happens randomly.

I saw what version you were running in the first post but you still didn't say where it came from. It's not a version we supply. And since it's from Jan 2019 you should probably look at it and find out what's been fixed in 10.0.x since then.

I installed it when I built the server many moons ago. and as far as I recall, it was from the yum repositories. Do you think my repos are bad? I update monthly but to be honest, I don't know the last time mysql was actually updated. Guess I am not paying enough attention to it. SHould I be updating mysql?

Determining fastest mirrors
* base: repos.lax.quadranet.com
* extras: mirror.san.fastserv.com
* updates: mirror.centos.lax1.serverforge.org
repo id repo name status
!base/7/x86_64 CentOS-7 - Base 10,072
!extras/7/x86_64 CentOS-7 - Extras 451
!mariadb MariaDB 43
!updates/7/x86_64 CentOS-7 - Updates 1,640
repolist: 12,206

bigklay
Posts: 5
Joined: 2021/05/12 15:00:53

Re: mysql crashes Out of memory: Kill process 5290

Post by bigklay » 2021/05/12 20:36:09

It is definitely out of date. I already moved to the 10.2.xxx on a test system, I will begin by putting it on the dev box for the web developer and see if it is stable. Then I can plan the upgrade for the rest of the systems.

BShT
Posts: 583
Joined: 2019/10/09 12:31:40

Re: mysql crashes Out of memory: Kill process 5290

Post by BShT » 2021/05/13 18:15:12

i had never seen mysql OOM except bad code and undersized server

there are some my.cnf confs that you need to set if you have a really loaded environment

pjsr2
Posts: 614
Joined: 2014/03/27 20:11:07

Re: mysql crashes Out of memory: Kill process 5290

Post by pjsr2 » 2021/05/14 09:51:09

Are you sure the mysql server is to blame? As I understand it: the OOM killer kicks in when the total memory usage becomes to large. It then tries to kill as few processes as possible while regaining as much memory as possible. This makes a process like the mysql server a likely candidate for killing, even when the actual shortage of memory is due to a number of other processes which each use less memory, together cause the excessive memory usage.
I could think of a situation where your web server starts to fork a higher number of working processes that each consume a sizeable amount of memory.

bigklay
Posts: 5
Joined: 2021/05/12 15:00:53

Re: mysql crashes Out of memory: Kill process 5290

Post by bigklay » 2021/05/21 18:36:09

These are all valid thoughts, I decided to grab the lowest hanging fruit and since the mysql was so out of date, I tested and deployed the upgrade to mysql. Unfortunately, the only time the failure has been seen is on the PROD server. So I updated it after testing in dev and QA and we are monitoring the situation now.

Post Reply