Mariadb could be hijacked?

[ralf]

Hi,

This morning my webserver running a wordpress website with mariadb and apache was not responding to anything - I had to manually switch it off and on with the power switch to get it responsive again.

The wordpress website has been running for a long time without issues.
My centos 8 is fully updated, and no updates were done during the last weeks, but still the problem occurred last night.
My php version is also up to date:

Code: Select all

[root@server1 mariadb]# rpm -qa | grep php
php-xml-7.2.31-1.el8.remi.x86_64
php-fpm-7.2.31-1.el8.remi.x86_64
php-cli-7.2.31-1.el8.remi.x86_64
php-common-7.2.31-1.el8.remi.x86_64
php-pecl-zip-1.18.2-1.el8.remi.7.2.x86_64
php-pdo-7.2.31-1.el8.remi.x86_64
php-mbstring-7.2.31-1.el8.remi.x86_64
php-json-7.2.31-1.el8.remi.x86_64
php-imap-7.2.31-1.el8.remi.x86_64
php-mysqlnd-7.2.31-1.el8.remi.x86_64
[root@server1 mariadb]# 

My daily logwatch has mentioned for some days now, that I am exposed for attempts to enter phpMyAdmin, with comments like:

**Unmatched Entries**
phpMyAdmin: user denied: ADMIN (mysql-denied) from 149.129.69.214: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 168.227.182.21: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 31.155.239.140: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 31.215.19.37: 1 Time(s)
phpMyAdmin: user denied: ADMIN (mysql-denied) from 84.2.253.146: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 119.95.184.15: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 197.36.27.245: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 223.206.128.204: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 39.41.162.91: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 82.202.226.51: 1 Time(s)
phpMyAdmin: user denied: ROOT (mysql-denied) from 82.78.49.33: 1 Time(s)
phpMyAdmin: user denied: USER (mysql-denied) from 142.93.251.224: 1 Time(s)
phpMyAdmin: user denied: USER (mysql-denied) from 183.89.78.235: 1 Time(s)
phpMyAdmin: user denied: USER (mysql-denied) from 197.202.11.195: 1 Time(s)

Today, when going to my wordpress website, it repsonds with "Error establishing a database connection"; httpd is running without any issues though; I can see the login window for phpMyAdmin in my browser - but not login.

So I checked the mariadb. It is constantly crashing, dumping core and trying to restart etc :

Code: Select all

[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: activating (start) since Sun 2020-05-17 10:24:27 CEST; [b]1s ago[/b]
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 7921 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
  Process: 7892 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 7959 (mysqld)
    Tasks: 15 (limit: 26213)
   Memory: 55.6M
   CGroup: /system.slice/mariadb.service
           ├─7959 /usr/libexec/mysqld --basedir=/usr
           └─7978 addr2line -C -f -e /usr/libexec/mysqld

May 17 10:24:27 server1.hartings.se systemd[1]: Starting MariaDB 10.3 database server...
May 17 10:24:27 server1.hartings.se mysql-check-socket[7892]: Socket file /var/lib/mysql/mysql.sock exists.
May 17 10:24:27 server1.hartings.se mysql-check-socket[7892]: [b]No process is using /var/lib/mysql/mysql.sock, which means it is a garbage[/b], s>
May 17 10:24:27 server1.hartings.se mysql-prepare-db-dir[7921]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing>
May 17 10:24:27 server1.hartings.se mysql-prepare-db-dir[7921]: If this is not the case, make sure the /var/lib/mysql is empty before runni>
May 17 10:24:27 server1.hartings.se mysqld[7959]: 2020-05-17 10:24:27 0 [Note] /usr/libexec/mysqld (mysqld 10.3.17-MariaDB) starting as pro>
May 17 10:24:27 server1.hartings.se mysqld[7959]: 2020-05-17 10:24:27 0 [Warning] Could not increase number of max_open_files to more than >
May 17 10:24:27 server1.hartings.se mysqld[7959]: 2020-05-17 10:24:27 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 15>
[root@server1 mariadb]# systemctl restart mariadb
Job for mariadb.service failed because a fatal signal was delivered causing the control process to dump core.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: core-dump) since Sun 2020-05-17 10:25:30 CEST; 3s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 9251 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=dumped, signal=ABRT)
  Process: 9213 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
  Process: 9184 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 9251 (code=dumped, signal=ABRT)
    Tasks: 0 (limit: 26213)
   Memory: 1.6M
   CGroup: /system.slice/mariadb.service

May 17 10:25:30 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 10:25:30 server1.hartings.se systemd[1]: Failed to start MariaDB 10.3 database server.
[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: core-dump) since Sun 2020-05-17 10:29:09 CEST; 4s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 14005 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=dumped, signal=ABRT)
  Process: 13967 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
  Process: 13936 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 14005 (code=dumped, signal=ABRT)
    Tasks: 0 (limit: 26213)
   Memory: 1.7M
   CGroup: /system.slice/mariadb.service

May 17 10:29:09 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 10:29:09 server1.hartings.se systemd[1]: Failed to start MariaDB 10.3 database server.
[root@server1 mariadb]# 

I read the output from journalctl -xe :

Code: Select all

root@server1 ~]# journalctl -xe
                                                             #0  0x00007fe9a10d699d syscall (libc.so.6)
                                                             #1  0x00007fe9a2db0b7e __io_getevents_0_4 (libaio.so.1)
                                                             #2  0x0000564599843042 _ZN15LinuxAIOHandler7collectEv (mysqld)
                                                             #3  0x00005645998433d0 _ZN15LinuxAIOHandler4pollEPP10fil_node_tPPvP9IORequest >
                                                             #4  0x000056459984649a _Z14os_aio_handlermPP10fil_node_tPPvP9IORequest (mysqld)
                                                             #5  0x000056459998dd27 _Z12fil_aio_waitm (mysqld)
                                                             #6  0x00005645998c2a88 io_handler_thread (mysqld)
                                                             #7  0x00007fe9a31d82de start_thread (libpthread.so.0)
                                                             #8  0x00007fe9a10dc133 __clone (libc.so.6)
                                                             
                                                             Stack trace of thread 15869:
                                                             #0  0x00007fe9a10d699d syscall (libc.so.6)
                                                             #1  0x00007fe9a2db0b7e __io_getevents_0_4 (libaio.so.1)
                                                             #2  0x0000564599843042 _ZN15LinuxAIOHandler7collectEv (mysqld)
                                                             #3  0x00005645998433d0 _ZN15LinuxAIOHandler4pollEPP10fil_node_tPPvP9IORequest >
                                                             #4  0x000056459984649a _Z14os_aio_handlermPP10fil_node_tPPvP9IORequest (mysqld)
                                                             #5  0x000056459998dd27 _Z12fil_aio_waitm (mysqld)
                                                             #6  0x00005645998c2a88 io_handler_thread (mysqld)
                                                             #7  0x00007fe9a31d82de start_thread (libpthread.so.0)
                                                             #8  0x00007fe9a10dc133 __clone (libc.so.6)
                                                             
                                                             Stack trace of thread 15866:
                                                             #0  0x00007fe9a10d699d syscall (libc.so.6)
                                                             #1  0x00007fe9a2db0b7e __io_getevents_0_4 (libaio.so.1)
                                                             #2  0x0000564599843042 _ZN15LinuxAIOHandler7collectEv (mysqld)
                                                             #3  0x00005645998433d0 _ZN15LinuxAIOHandler4pollEPP10fil_node_tPPvP9IORequest >
                                                             #4  0x000056459984649a _Z14os_aio_handlermPP10fil_node_tPPvP9IORequest (mysqld)
                                                             #5  0x000056459998dd27 _Z12fil_aio_waitm (mysqld)
                                                             #6  0x00005645998c2a88 io_handler_thread (mysqld)
                                                             #7  0x00007fe9a31d82de start_thread (libpthread.so.0)
                                                             #8  0x00007fe9a10dc133 __clone (libc.so.6)
-- Subject: Process 15860 (mysqld) dumped core
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- Documentation: man:core(5)
-- 
-- Process 15860 (mysqld) crashed and dumped core.
-- 
-- This usually indicates a programming error in the crashing program and
-- should be reported to its vendor as a bug.
May 17 10:04:27 server1.hartings.se systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 17 10:04:27 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 10:04:27 server1.hartings.se systemd[1]: Failed to start MariaDB 10.3 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Unit mariadb.service has failed.
-- 
-- The result is RESULT.

I also read the mariadb logfile:

Code: Select all

[root@server1 mariadb]# more mariadb.log
2020-05-16  3:16:31 61116 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16  3:55:05 61123 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16  5:13:37 61127 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16  5:55:02 61133 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16  6:34:25 61136 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16  7:19:49 61142 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16  7:28:10 61144 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16  8:01:01 61147 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16  8:38:41 61151 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16  9:56:38 61157 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 10:30:18 61164 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-16 11:02:05 61167 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 11:35:58 61171 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-05-16 12:09:35 61188 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 12:39:52 61205 [Warning] Access denied for user 'ROOT'@'localhost' (using password: YES)
2020-05-16 13:11:01 61230 [Warning] Access denied for user 'admin'@'localhost' (using password: YES)
2020-05-16 13:29:02 61247 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 14:07:09 61283 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 14:32:38 61298 [Warning] Access denied for user 'ADMIN'@'localhost' (using password: YES)
2020-05-16 15:02:23 61528 [Warning] Access denied for user 'USER'@'localhost' (using password: YES)
2020-05-16 16:05:05 61585 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 16:29:38 61612 [Warning] Access denied for user 'user'@'localhost' (using password: YES)
2020-05-16 17:01:17 61637 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:29:22 61664 [Warning] Access denied for user 'hartings'@'localhost' (using password: YES)
2020-05-16 17:56:55 61689 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 18:25:28 61713 [Warning] Access denied for user 'hartings.se'@'localhost' (using password: YES)
2020-05-16 19:02:20 61745 [Warning] Access denied for user '[asDomaincom]'@'localhost' (using password: YES)
2020-05-17 10:18:03 0 [Note] InnoDB: Using Linux native AIO
2020-05-17 10:18:03 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-05-17 10:18:03 0 [Note] InnoDB: Uses event mutexes
2020-05-17 10:18:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-05-17 10:18:03 0 [Note] InnoDB: Number of pools: 1
2020-05-17 10:18:03 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-05-17 10:18:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-05-17 10:18:03 0 [Note] InnoDB: Completed initialization of buffer pool
2020-05-17 10:18:03 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man pa
ge of setpriority().
2020-05-17 10:18:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=79381745
2020-05-17 10:18:03 0 [ERROR] [FATAL] InnoDB: Trying to read page number 4294934527 in space 0, space name innodb_system, which is outside t
he tablespace bounds. Byte offset 0, len 16384Please check that the configuration matches the InnoDB system tablespace location (ibdata file
s)
200517 10:18:03 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
[root@server1 mariadb]#

I stopped the mariadb, to avoid any more crashes and it tells me this:

Code: Select all

[root@server1 mariadb]# systemctl stop mariadb
[root@server1 mariadb]# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: core-dump) since Sun 2020-05-17 11:03:45 CEST; 9s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 24125 ExecStart=/usr/libexec/mysqld --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=dumped, signal=ABRT)
  Process: 24087 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
  Process: 24058 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
 Main PID: 24125 (code=dumped, signal=ABRT)
    Tasks: 0 (limit: 26213)
   Memory: 1.8M
   CGroup: /system.slice/mariadb.service

May 17 11:03:44 server1.hartings.se mysql-check-socket[24058]: Socket file /var/lib/mysql/mysql.sock exists.
May 17 11:03:44 server1.hartings.se mysql-check-socket[24058]: No process is using /var/lib/mysql/mysql.sock, which means it is a garbage, >
May 17 11:03:44 server1.hartings.se mysql-prepare-db-dir[24087]: Database MariaDB is probably initialized in /var/lib/mysql already, nothin>
May 17 11:03:44 server1.hartings.se mysql-prepare-db-dir[24087]: If this is not the case, make sure the /var/lib/mysql is empty before runn>
May 17 11:03:44 server1.hartings.se mysqld[24125]: 2020-05-17 11:03:44 0 [Note] /usr/libexec/mysqld (mysqld 10.3.17-MariaDB) starting as pr>
May 17 11:03:44 server1.hartings.se mysqld[24125]: 2020-05-17 11:03:44 0 [Warning] Could not increase number of max_open_files to more than>
May 17 11:03:44 server1.hartings.se mysqld[24125]: 2020-05-17 11:03:44 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 1>
May 17 11:03:45 server1.hartings.se systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
May 17 11:03:45 server1.hartings.se systemd[1]: mariadb.service: Failed with result 'core-dump'.
May 17 11:03:45 server1.hartings.se systemd[1]: Stopped MariaDB 10.3 database server.
lines 1-23/23 (END)

Just to print the complete text in the unclipped text in the /var/log/messages file:

Code: Select all

[root@server1 log]# tail -n 500 messages-20200517
May 17 10:43:33 server1 mysql-prepare-db-dir[31120]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
May 17 10:43:33 server1 mysql-prepare-db-dir[31120]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.
May 17 10:43:33 server1 mysqld[31158]: 2020-05-17 10:43:33 0 [Note] /usr/libexec/mysqld (mysqld 10.3.17-MariaDB) starting as process 31158 ...
May 17 10:43:33 server1 mysqld[31158]: 2020-05-17 10:43:33 0 [Warning] [b]Could not increase number of max_open_files to more than 1024 (request: 4186)
May 17 10:43:33 server1 mysqld[31158]: 2020-05-17 10:43:33 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)[/b]
May 17 10:43:33 server1 systemd[1]: Started Process Core Dump (PID 31196/UID 0).

My mariadb rpm's are:

Code: Select all

[root@server1 ~]# rpm -qa | grep maria
mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-3.0.7-1.el8.x86_64
mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-config-3.0.7-1.el8.noarch
mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64

[root@server1 ~]#

What is really my problem here and any clue as to what I should do?
Re-install mariadb (php updated this morning after the crash)?
Your input is really appreciated, as my website is down !
Thanks!

/Ralf

[ralf]

An update/additional info:

Just read the general CENTOS8 announcement: viewtopic.php?f=9&t=72492&sid=94ef9edfd ... a495fb92e9
Don't understand I missed that before. I followed the recommended update and rebooted. Though many new packages were installed, still the same problem. No difference.

I re-installed via dnf all mariadb packages.

Code: Select all

mariadb-backup-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-common-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-errmsg-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-config-3.0.7-1.el8.noarch
mariadb-server-utils-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64
mariadb-connector-c-3.0.7-1.el8.x86_64
mariadb-gssapi-server-10.3.17-1.module_el8.1.0+257+48736ea6.x86_64

--> Still the same problem.

My website is still down!! I don't know what to do any more
Any hint is really appreciated!

/Ralf

[gerry666uk]

The first sentence jumps out at me:

"This morning my webserver running a wordpress website with mariadb and apache was not responding to anything - I had to manually switch it off and on with the power switch to get it responsive again."

When you say "not responding" do you mean
1. It's a pure console server running on bare metal, and you could not type anything
2. You were unable to use SSH from another client machine
3. It's a vm
4. It's a GUI server with a screen and mouse pointer and the pointer was frozen
etc, etc,

Switching off power to a database server is BAD BAD BAD, it might corrupt the data if some page had not been written from memory.
Could it be a hardware failure, if it was frozen to that degree?
Check /var/log/messages from the day/evening before it froze to see if anything else was going on

Were the files in 'datadir' constructed "from nothing" by this instance of MariaDB, or did you try to place some existing data files into this new instance? You may be able to get it restarted by removing all files from 'datadir' and then start it up, and then do a restore of just the WordPress database into the new instance.

[gerry666uk]

In Addition,

Regarding phpMyAdmin:

You should not expose the phpMyAdmin login to the internet. For a start, don't use a URL with the word '/phpMyAdmin', use something like '/s_skfkse', and then use access restrictions e.g. localhost only, or better still use a separate server that can only be reached by the internal network.

Regarding WordPress:
Be careful of people attacking /wp-login.php
Check your existing web logs, and see if there any repeated requests to /wp-login.php

[ralf]

Hi gerry666uk

Thanks for the feedback. It's a console server, and it did not repsond to anything I typed; couldn't do anything else but switching off/on.
I know that this is "not recommended", but it was in my view allready dead.

The way forward you suggest is indeed what I planned to do, as I could not find any other way to repair the corrupt Innodb:
1. Copy the orginal db settings (for just in case....):
cp -a /etc/my.cnf /root/_etc_my.cnf_backup
cp -aR /etc/my.cnf.d/ /root/_etc_my.cnf.d_backup
backup the "wordpress" db-folder in /var/lib/mysql/ to somewhere else
backup the "phpmyadmin" folder in /var/lib/mysql/ to somewhere else

2. Follow the suggestions in https://stackoverflow.com/questions/333 ... -or-rhel-7
meaning:
Remove the mariadb packages and delete the directory /var/lib/mysql; delete the /etc/my.cnf and delete /etc/mycnf.d/
Then reinstall the mariadb packages.
Then try to put back the copied "wordpress"-folder and the copied "phpmyadmin" folder to /var/lib/mysql/
If it is not possible to copy back (cp -a) those folders, I have to first reinstall phpmyadminand then import, using phpmyadmin, my backup wordpress db. Some more work but should be feasible.

Any comments to this? Would this work for CentOS8 too?

You should not expose the phpMyAdmin login to the internet. For a start, don't use a URL with the word '/phpMyAdmin', use something like '/s_skfkse', and then use access restrictions e.g. localhost only, or better still use a separate server that can only be reached by the internal network.

Regarding WordPress:
Be careful of people attacking /wp-login.php
Check your existing web logs, and see if there any repeated requests to /wp-login.php

Agree! I had phpmyadmin actually exposed to the internet, which I will certainly not do again . I learned my lesson when looking at the numerous attempts to enter via phpmyadmin. Exactly where do I choose the "name" and the login limitation to localhost only?

WordPress is protected as good as I can. I searched the internet before and found lots of suggestions on how to protect it. I also used httpd to password protect the wp-admin (on top of the WP login itself). But I will do a second search....

Thanks for yout valuable input!

/Ralf

[gerry666uk]

Regarding it being a console server:

It's unusual for it to stop responding, as there's not much to go wrong (compared with a GUI server). Was the cursor still flashing?
There are some hot keys you can try in this situation to switch to a different console, but I don't know the details.
You still need to find out what was happening in the few hours before it stopped responding, you should see log messages cease at some point, unless it was still running, (and just the console broke). You should be able to see the gap between where it went wrong and where the power was turned back on. Be sure to check smartmon on all drives and do fs check on all filesystems.

Regarding the "cnf" files:

Yes, it's a good idea to make a backup, but the bigger question is did you ever change them?

Regarding the "WordPress db folder" in /var/lib/mysql:

Yes, you should back up the folder, but this is not the normal way to keep a backup of your WordPress database. You should instead be doing a database dump each night, and then if it breaks, you restore it from the dump, NOT from the binary files.

Regarding the "phpmyadmin" folder in /var/lib/mysql:

That doesn't sound right to me, unless phpmyadmin has it's own database, it should not be anywhere near /var/lib/mysql?? I thought it was just a bunch of PHP files?

Other than that:

What you are saying makes sense, e.g. clean up everything and let MariaDB re-create a new set of database files in 'datadir', run some tests with a simple home made database "my_test_db", and make sure it's working properly. Stop and Start the service a few times, and check the logs on each, then finally restore the WordPress database (preferably from a dump, because the binary files might be damaged).

Note, don't delete the actual directory '/var/lib/mysql', but instead empty it completely, otherwise you might lose selinux

[ralf]

Hi again,

Regarding it being a console server:

This is exactly why it scared me. To my knowledge nothing was changed, at least not by me. I did check /var/log/messages for the day of the crash and it said nothing special. Same for mariad.log. I checked the secure.log files. Apart from lots of attemps via phpmyadmin and ssh not particular - no successfull login is logged. Very strange; perhaps someone cleaned the logs, or it is a hardware issue, i.e. bad memory.
So I checked the RAM with MEMTEST for 2 hours - 2 runs - and I got one single error during the hammer test, which is quite severe. So I guess the RAM is not likely the reason for my problems.

Code: Select all

Regarding the "cnf" files:

No, to my knowledge I didn't change anything...

Code: Select all

Regarding the "phpmyadmin" folder in /var/lib/mysql:

That doesn't sound right to me, unless phpmyadmin has it's own database, it should not be anywhere near /var/lib/mysql?? I thought it was just a bunch of PHP files?

phpmyadmin dir in /var/lib/mysql, it contains what seems to be its own database:

Code: Select all

[root@server1 mysql]# cd phpmyadmin
[root@server1 phpmyadmin]# ll
total 2004
-rw-rw----. 1 mysql mysql     54 Oct 17  2019 db.opt
-rw-rw----. 1 mysql mysql   3323 Oct 17  2019 pma__bookmark.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__bookmark.ibd
-rw-rw----. 1 mysql mysql   2678 Oct 17  2019 pma__central_columns.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__central_columns.ibd
-rw-rw----. 1 mysql mysql   6880 Oct 17  2019 pma__column_info.frm
-rw-rw----. 1 mysql mysql 114688 Oct 17  2019 pma__column_info.ibd
-rw-rw----. 1 mysql mysql   1157 Oct 17  2019 pma__designer_settings.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__designer_settings.ibd
-rw-rw----. 1 mysql mysql   1954 Oct 17  2019 pma__export_templates.frm
-rw-rw----. 1 mysql mysql 114688 Oct 17  2019 pma__export_templates.ibd
-rw-rw----. 1 mysql mysql   1150 Oct 17  2019 pma__favorite.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__favorite.ibd
-rw-rw----. 1 mysql mysql   2129 Oct 17  2019 pma__history.frm
-rw-rw----. 1 mysql mysql 114688 Oct 17  2019 pma__history.ibd
-rw-rw----. 1 mysql mysql   1995 Oct 17  2019 pma__navigationhiding.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__navigationhiding.ibd
-rw-rw----. 1 mysql mysql   1812 Oct 17  2019 pma__pdf_pages.frm
-rw-rw----. 1 mysql mysql 114688 Oct 17  2019 pma__pdf_pages.ibd
-rw-rw----. 1 mysql mysql   1150 Oct 17  2019 pma__recent.frm
-rw-rw----. 1 mysql mysql  98304 Nov  2  2019 pma__recent.ibd
-rw-rw----. 1 mysql mysql   2721 Oct 17  2019 pma__relation.frm
-rw-rw----. 1 mysql mysql 114688 Oct 17  2019 pma__relation.ibd
-rw-rw----. 1 mysql mysql   2108 Oct 17  2019 pma__savedsearches.frm
-rw-rw----. 1 mysql mysql 114688 Oct 17  2019 pma__savedsearches.ibd
-rw-rw----. 1 mysql mysql   1419 Oct 17  2019 pma__table_coords.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__table_coords.ibd
-rw-rw----. 1 mysql mysql   1560 Oct 17  2019 pma__table_info.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__table_info.ibd
-rw-rw----. 1 mysql mysql   1621 Oct 17  2019 pma__table_uiprefs.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__table_uiprefs.ibd
-rw-rw----. 1 mysql mysql   1812 Oct 17  2019 pma__tracking.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__tracking.ibd
-rw-rw----. 1 mysql mysql   1186 Oct 17  2019 pma__userconfig.frm
-rw-rw----. 1 mysql mysql  98304 Nov  5  2019 pma__userconfig.ibd
-rw-rw----. 1 mysql mysql   1363 Oct 17  2019 pma__usergroups.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__usergroups.ibd
-rw-rw----. 1 mysql mysql   1336 Oct 17  2019 pma__users.frm
-rw-rw----. 1 mysql mysql  98304 Oct 17  2019 pma__users.ibd
[root@server1 phpmyadmin]# 

Code: Select all

Note, don't delete the actual directory '/var/lib/mysql', but instead empty it completely, otherwise you might lose selinux

Good point, thanks, I will try to remove/uninstall the directories below and remove the packages, then put back the phpmyadmin dir. If this doesn't work I need to install pgpmyadmin from scratch again. Then I will secure/not expose phpmyadmin to the internet (change name, limit IP access, and put extra password)
Then with the phpmyadmin working, I can put back the orginal wordpress.sql db (I have a recent backup, created using phpmyadmin).

I'll let you know what happened.

Thanks!!

[ralf]

OK. So my website is up and running again!

As to the procedure:
Deleting the directories in /var/lib/mysql worked fine.
Re-instaling mariadb and mariadb-server worked fine too --> now mysql is running again
Copying back the old /var/lib/mysql/phpmyadmin did not work ; I had to reinstall it from scratch again.
I used this https://www.howtoforge.com/install-and- ... -centos-8/ to install phpmyadmin.

When that was done:
I could use phpmyadmin.
I then created an empty db, called wordpress using phpmyadmin
I reinstalled the backup of my original wordpress.sql db, using phpmyadmin
Now the website was up and running!

I also implemented the section "Secure phpMyAdmin" from the same website:
Adding the exra layer of security, by adding a user "admin" worked great.
However, limiting the IP range to access phpmyadmin , doesn't seem to work.....

I modified the content of phpmyadmin.con into:

Code: Select all

[root@server1 conf.d]# more phpmyadmin.conf 
Alias /phpmyadmin /usr/share/phpmyadmin

<Directory /usr/share/phpmyadmin/>
   AddDefaultCharset UTF-8

Options  +FollowSymLinks +Multiviews +Indexes
    AllowOverride None
    AuthType basic
    AuthName "Authentication Required"
    AuthUserFile /etc/phpmyadmin/.htpasswd
    Require valid-user

   <IfModule mod_authz_core.c>
     # Apache 2.4
     <RequireAny> 
      Require ip 127.0.0.1
      Require ip ::1
     </RequireAny>
   </IfModule>
   <IfModule !mod_authz_core.c>
     # Apache 2.2
     Order Deny,Allow
     Deny from All
     Allow from 127.0.0.1
     Allow from ::1
   </IfModule>
</Directory>

<Directory /usr/share/phpmyadmin/setup/>
   <IfModule mod_authz_core.c>
     # Apache 2.4
     <RequireAny>
       Require ip 127.0.0.1
       Require ip ::1
     </RequireAny>
   </IfModule>
   <IfModule !mod_authz_core.c>
     # Apache 2.2
     Order Deny,Allow
     Deny from All
     Allow from 127.0.0.1
     Allow from ::1
   </IfModule>
</Directory>
[root@server1 conf.d]#

Then I restarted apache: systemctl restart httpd to activate the new version.
However, I can still login (though now seeing the extra security layer - user: admin), but from any local IP address, like 192.168.1.x
I thought this was now not possible anymore?

Any suggestions as to what is wrong?

I think I want to change the "phpmyadmin" adress/name as well. Is this done by changing in the phpmyadmin.conf:
Alias /phpmyadmin /usr/share/phpmyadmin into
Alias /whatever_name /usr/share/phpmyadmin

Thanks for your help here!

[gerry666uk]

Good news to hear that the database and web site are both back up now.

The original issue of why the console was not responding still seems unresolved to me, and I don't know if you did the smartmon and filesystem checks.

You might want to install and enable 'sar', as it will quickly show you the exact point of any downtime, e.g. if the server were to freeze at 02:00, you'd see a big gap in the 'sar' output between that time and the time it was powered off and then powered on again. it will also show big peaks if there's a hacking attempt.

Regarding the phpMyAdmin "pma" database, I do remember this now, it's to do with offering some extra feartures, but as you've discovered, it's not ideal in a disaster recovery scenario. I currently run it without the 'pma' database.

phpMyAdmin is handy for end users, but as you're the server admin, using a console, you might be better off becoming a Database Administrator (DBA), where you would do everything on the command line, create database, set perms, backup, restore etc. This is many time better when it comes to disaster recovery.

I'll leave the other general questions about web server restrictions to someone else, (or a different forum) as it's not specific to CentOS 8.