Mariadb 10.2 suddenly stopped working in Centos 7 error code=exited 203/EXEC

Hi to all,

Unfortunately my production server suddenly encountered this error in Mariadb 10.2.Due to this issue
my server is down for 3 days.Theres no specific error on logs that contains information about this issue.

[root@mydb ~]# systemctl status mysqld
● mariadb.service - MariaDB 10.2.44 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Sun 2023-12-10 03:19:00 PST; 2s ago
     Docs: man:mysqld(
           https://mariadb.com/kb/en/library/systemd/
  Process: 163260 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS --basedir=/usr $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=203/EXEC)
  Process: 163231 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 163228 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 163260 (code=exited, status=203/EXEC)

Dec 10 03:19:00 mydb systemd[1]: Starting MariaDB 10.2.44 database server...
Dec 10 03:19:00 mydb systemd[1]: mariadb.service: main process exited, code=exited, status=203/EXEC
Dec 10 03:19:00 mydb systemd[1]: Failed to start MariaDB 10.2.44 database server.
Dec 10 03:19:00 mydb systemd[1]: Unit mariadb.service entered failed state.
Dec 10 03:19:00 mydb systemd[1]: mariadb.service failed.

status=203/EXEC is usually a "File Not Found" error. 

please post the last lines of "  journalctl -u mariadb -n 100  "

Hi thanks for the reply, the file mysql and mysqld files are existing.Also here is the logs.

[root@mydb ~]# journalctl -xe mariadb
Failed to add match 'mariadb': Invalid argument
Failed to add filters: Invalid argument
[root@mydb ~]# journalctl -u mariadb
-- Logs begin at Tue 2023-12-12 18:21:58 PST, end at Tue 2023-12-12 20:53:53 PST. --
Dec 12 18:22:10 mydb.mydb systemd[1]: Starting MariaDB 10.2.44 database server...
Dec 12 18:22:10 mydb.mydb systemd[1]: mariadb.service: main process exited, code=exited, status=203/EXEC
Dec 12 18:22:10 mydb.mydb systemd[1]: Failed to start MariaDB 10.2.44 database server.
Dec 12 18:22:10 mydb.mydb systemd[1]: Unit mariadb.service entered failed state.
Dec 12 18:22:10 mydb.mydb systemd[1]: mariadb.service failed.
[root@mydb ~]#


also  ,if I run only the /usr/sbin/mysqld command, the mariadb instance will start the database and the CWP7 dashboard is accessible.
But unfortunately it is not working properly maybe due the systemd mariadb service is required to make the  CWP fully working?
we are really confused right now on this issue because this issue just occured and we dont have any changes on the system.

Try journalctl -x -u mariadb -n 50

Also make sure one of your instances of mysql isn't already running on port 3306.

netstat -nlp | grep 3306

heres 50 lines journalctl

[root@mydb ~]# journalctl -x -u mariadb -n 50
-- Logs begin at Tue 2023-12-12 18:21:58 PST, end at Tue 2023-12-12 21:58:33 PST. --
Dec 12 18:22:10 mydb systemd[1]: Starting MariaDB 10.2.44 database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has begun starting up.
Dec 12 18:22:10 mydb systemd[1]: mariadb.service: main process exited, code=exited, status=203/EXEC
Dec 12 18:22:10 mydb systemd[1]: Failed to start MariaDB 10.2.44 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Dec 12 18:22:10 mydb systemd[1]: Unit mariadb.service entered failed state.
Dec 12 18:22:10 mydb systemd[1]: mariadb.service failed.




and yes, 3306 is not in used unless as I mentioned that I can run the mysql using this command /usr/sbin/mysqld

Verify that you /etc/systemd/system/(maridbd/mysqld).service file matches the settings below

https://pastebin.com/QZ8m6Ubt

This might help.

https://fedoraproject.org/wiki/How_to_debug_MySQL_and_MariaDB

I compared it and looks like this is the major difference since your given link content is from 10.4 and mine is 10.2

ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION

But I tried to used that and still encountered the same error.

https://mariadb.com/kb/en/what-to-do-if-mariadb-doesnt-start/   Lots of insightful information directly from mariadb.

MariaDB 10.2.x isn't supported any longer.

The lowest LTS is 10.4.x

For further support you should upgrade to this version, or higher.

https://wiki.centos-webpanel.com/mariadb-upgrade-from-10-2-to-10-4