This morning my webserver running a wordpress website with mariadb and apache was not responding to anything - I had to manually switch it off and on with the power switch to get it responsive again.
The wordpress website has been running for a long time without issues.
My centos 8 is fully updated, and no updates were done during the last weeks, but still the problem occurred last night.
My php version is also up to date:
Code: Select all
[root@server1 mariadb]# rpm -qa | grep php
php-xml-7.2.31-1.el8.remi.x86_64
php-fpm-7.2.31-1.el8.remi.x86_64
php-cli-7.2.31-1.el8.remi.x86_64
php-common-7.2.31-1.el8.remi.x86_64
php-pecl-zip-1.18.2-1.el8.remi.7.2.x86_64
php-pdo-7.2.31-1.el8.remi.x86_64
php-mbstring-7.2.31-1.el8.remi.x86_64
php-json-7.2.31-1.el8.remi.x86_64
php-imap-7.2.31-1.el8.remi.x86_64
php-mysqlnd-7.2.31-1.el8.remi.x86_64
[root@server1 mariadb]#
**Unmatched Entries**
phpMyAdmin: user denied: ADMIN (mysql-denied) from 149.129.69.214: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 168.227.182.21: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 31.155.239.140: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 31.215.19.37: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 84.2.253.146: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 119.95.184.15: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 197.36.27.245: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 223.206.128.204: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 39.41.162.91: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 82.202.226.51: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 82.78.49.33: 1 Time(s)
phpMyAdmin: user denied: USER (mysql-denied) from 142.93.251.224: 1 Time(s)
phpMyAdmin: user denied: USER (mysql-denied) from 183.89.78.235: 1 Time(s)
phpMyAdmin: user denied: USER (mysql-denied) from 197.202.11.195: 1 Time(s)
Today, when going to my wordpress website, it repsonds with "Error establishing a database connection"; httpd is running without any issues though; I can see the login window for phpMyAdmin in my browser - but not login.
So I checked the mariadb. It is constantly crashing, dumping core and trying to restart etc :
Code: Select all
[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: activating (start) since Sun 2020-05-17 10:24:27 CEST; [b]1s ago[/b]
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 7921 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
Process: 7892 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 7959 (mysqld)
Tasks: 15 (limit: 26213)
Memory: 55.6M
CGroup: /system.slice/mariadb.service
├─7959 /usr/libexec/mysqld --basedir=/usr
└─7978 addr2line -C -f -e /usr/libexec/mysqld
May 17 10:24:27 server1.hartings.se systemd[1]: Starting MariaDB 10.3 database server...
May 17 10:24:27 server1.hartings.se mysql-check-socket[7892]: Socket file /var/lib/mysql/mysql.sock exists.
May 17 10:24:27 server1.hartings.se mysql-check-socket[7892]: [b]No process is using /var/lib/mysql/mysql.sock, which means it is a garbage[/b], s>
May 17 10:24:27 server1.hartings.se mysql-prepare-db-dir[7921]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing>
May 17 10:24:27 server1.hartings.se mysql-prepare-db-dir[7921]: If this is not the case, make sure the /var/lib/mysql is empty before runni>
May 17 10:24:27 server1.hartings.se mysqld[7959]: 2020-05-17 10:24:27 0 [Note] /usr/libexec/mysqld (mysqld 10.3.17-MariaDB) starting as pro>
May 17 10:24:27 server1.hartings.se mysqld[7959]: 2020-05-17 10:24:27 0 [Warning] Could not increase number of max_open_files to more than >
May 17 10:24:27 server1.hartings.se mysqld[7959]: 2020-05-17 10:24:27 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 15>
[root@server1 mariadb]# systemctl restart mariadb
Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: core-dump) since Sun 2020-05-17 10:25:30 CEST; 3s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 9251 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=dumped, signal=ABRT)
Process: 9213 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
Process: 9184 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 9251 (code=dumped, signal=ABRT)
Tasks: 0 (limit: 26213)
Memory: 1.6M
CGroup: /system.slice/mariadb.service
May 17 10:25:30 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 10:25:30 server1.hartings.se systemd[1]: Failed to start MariaDB 10.3 database server.
[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: core-dump) since Sun 2020-05-17 10:29:09 CEST; 4s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 14005 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=dumped, signal=ABRT)
Process: 13967 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
Process: 13936 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 14005 (code=dumped, signal=ABRT)
Tasks: 0 (limit: 26213)
Memory: 1.7M
CGroup: /system.slice/mariadb.service
May 17 10:29:09 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 10:29:09 server1.hartings.se systemd[1]: Failed to start MariaDB 10.3 database server.
[root@server1 mariadb]#
Code: Select all
root@server1 ~]# journalctl -xe
#0 0x00007fe9a10d699d syscall (libc.so.6)
#1 0x00007fe9a2db0b7e __io_getevents_0_4 (libaio.so.1)
#2 0x0000564599843042 _ZN15LinuxAIOHandler7collectEv (mysqld)
#3 0x00005645998433d0 _ZN15LinuxAIOHandler4pollEPP10fil_node_tPPvP9IORequest >
#4 0x000056459984649a _Z14os_aio_handlermPP10fil_node_tPPvP9IORequest (mysqld)
#5 0x000056459998dd27 _Z12fil_aio_waitm (mysqld)
#6 0x00005645998c2a88 io_handler_thread (mysqld)
#7 0x00007fe9a31d82de start_thread (libpthread.so.0)
#8 0x00007fe9a10dc133 __clone (libc.so.6)
Stack trace of thread 15869:
#0 0x00007fe9a10d699d syscall (libc.so.6)
#1 0x00007fe9a2db0b7e __io_getevents_0_4 (libaio.so.1)
#2 0x0000564599843042 _ZN15LinuxAIOHandler7collectEv (mysqld)
#3 0x00005645998433d0 _ZN15LinuxAIOHandler4pollEPP10fil_node_tPPvP9IORequest >
#4 0x000056459984649a _Z14os_aio_handlermPP10fil_node_tPPvP9IORequest (mysqld)
#5 0x000056459998dd27 _Z12fil_aio_waitm (mysqld)
#6 0x00005645998c2a88 io_handler_thread (mysqld)
#7 0x00007fe9a31d82de start_thread (libpthread.so.0)
#8 0x00007fe9a10dc133 __clone (libc.so.6)
Stack trace of thread 15866:
#0 0x00007fe9a10d699d syscall (libc.so.6)
#1 0x00007fe9a2db0b7e __io_getevents_0_4 (libaio.so.1)
#2 0x0000564599843042 _ZN15LinuxAIOHandler7collectEv (mysqld)
#3 0x00005645998433d0 _ZN15LinuxAIOHandler4pollEPP10fil_node_tPPvP9IORequest >
#4 0x000056459984649a _Z14os_aio_handlermPP10fil_node_tPPvP9IORequest (mysqld)
#5 0x000056459998dd27 _Z12fil_aio_waitm (mysqld)
#6 0x00005645998c2a88 io_handler_thread (mysqld)
#7 0x00007fe9a31d82de start_thread (libpthread.so.0)
#8 0x00007fe9a10dc133 __clone (libc.so.6)
-- Subject: Process 15860 (mysqld) dumped core
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- Documentation: man:core(5)
--
-- Process 15860 (mysqld) crashed and dumped core.
--
-- This usually indicates a programming error in the crashing program and
-- should be reported to its vendor as a bug.
May 17 10:04:27 server1.hartings.se systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 17 10:04:27 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 10:04:27 server1.hartings.se systemd[1]: Failed to start MariaDB 10.3 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit mariadb.service has failed.
--
-- The result is RESULT.
Code: Select all
[root@server1 mariadb]# more mariadb.log
2020-05-16 3:16:31 61116 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 3:55:05 61123 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 5:13:37 61127 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16 5:55:02 61133 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 6:34:25 61136 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 7:19:49 61142 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 7:28:10 61144 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 8:01:01 61147 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 8:38:41 61151 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 9:56:38 61157 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 10:30:18 61164 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 11:02:05 61167 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 11:35:58 61171 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 12:09:35 61188 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 12:39:52 61205 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 13:11:01 61230 [Warning] Access denied for user 'admin'@'localhost' (using password: YES)
2020-05-16 13:29:02 61247 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 14:07:09 61283 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 14:32:38 61298 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 15:02:23 61528 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16 16:05:05 61585 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 16:29:38 61612 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 17:01:17 61637 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:29:22 61664 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:56:55 61689 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 18:25:28 61713 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 19:02:20 61745 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-17 10:18:03 0 [Note] InnoDB: Using Linux native AIO
2020-05-17 10:18:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-05-17 10:18:03 0 [Note] InnoDB: Uses event mutexes
2020-05-17 10:18:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-05-17 10:18:03 0 [Note] InnoDB: Number of pools: 1
2020-05-17 10:18:03 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-05-17 10:18:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-05-17 10:18:03 0 [Note] InnoDB: Completed initialization of buffer pool
2020-05-17 10:18:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man pa
ge of setpriority().
2020-05-17 10:18:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=79381745
2020-05-17 10:18:03 0 [ERROR] [FATAL] InnoDB: Trying to read page number 4294934527 in space 0, space name innodb_system, which is outside t
he tablespace bounds. Byte offset 0, len 16384Please check that the configuration matches the InnoDB system tablespace location (ibdata file
s)
200517 10:18:03 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
[root@server1 mariadb]#
I stopped the mariadb, to avoid any more crashes and it tells me this:
Code: Select all
[root@server1 mariadb]# systemctl stop mariadb
[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: failed (Result: core-dump) since Sun 2020-05-17 11:03:45 CEST; 9s ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 24125 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=dumped, signal=ABRT)
Process: 24087 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
Process: 24058 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 24125 (code=dumped, signal=ABRT)
Tasks: 0 (limit: 26213)
Memory: 1.8M
CGroup: /system.slice/mariadb.service
May 17 11:03:44 server1.hartings.se mysql-check-socket[24058]: Socket file /var/lib/mysql/mysql.sock exists.
May 17 11:03:44 server1.hartings.se mysql-check-socket[24058]: No process is using /var/lib/mysql/mysql.sock, which means it is a garbage, >
May 17 11:03:44 server1.hartings.se mysql-prepare-db-dir[24087]: Database MariaDB is probably initialized in /var/lib/mysql already, nothin>
May 17 11:03:44 server1.hartings.se mysql-prepare-db-dir[24087]: If this is not the case, make sure the /var/lib/mysql is empty before runn>
May 17 11:03:44 server1.hartings.se mysqld[24125]: 2020-05-17 11:03:44 0 [Note] /usr/libexec/mysqld (mysqld 10.3.17-MariaDB) starting as pr>
May 17 11:03:44 server1.hartings.se mysqld[24125]: 2020-05-17 11:03:44 0 [Warning] Could not increase number of max_open_files to more than>
May 17 11:03:44 server1.hartings.se mysqld[24125]: 2020-05-17 11:03:44 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 1>
May 17 11:03:45 server1.hartings.se systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 17 11:03:45 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 11:03:45 server1.hartings.se systemd[1]: Stopped MariaDB 10.3 database server.
lines 1-23/23 (END)
Code: Select all
[root@server1 log]# tail -n 500 messages-20200517
May 17 10:43:33 server1 mysql-prepare-db-dir[31120]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
May 17 10:43:33 server1 mysql-prepare-db-dir[31120]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.
May 17 10:43:33 server1 mysqld[31158]: 2020-05-17 10:43:33 0 [Note] /usr/libexec/mysqld (mysqld 10.3.17-MariaDB) starting as process 31158 ...
May 17 10:43:33 server1 mysqld[31158]: 2020-05-17 10:43:33 0 [Warning] [b]Could not increase number of max_open_files to more than 1024 (request: 4186)
May 17 10:43:33 server1 mysqld[31158]: 2020-05-17 10:43:33 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000)[/b]
May 17 10:43:33 server1 systemd[1]: Started Process Core Dump (PID 31196/UID 0).
My mariadb rpm's are:
Code: Select all
[root@server1 ~]# rpm -qa | grep maria
mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-3.0.7-1.el8.x86_64
mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-config-3.0.7-1.el8.noarch
mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
What is really my problem here and any clue as to what I should do?
Re-install mariadb (php updated this morning after the crash)?
Your input is really appreciated, as my website is down !
Thanks!
/Ralf