Author Topic: how to debug mariadb crash?  (Read 446 times)

0 Members and 1 Guest are viewing this topic.

Offline
*
how to debug mariadb crash?
« on: April 09, 2019, 09:03:35 AM »
Hi

on my server mariadb is often being crashed and not restarted.

how to find a problem why it's crashed and is there any watchdog which will restart the service if any problem?

status shows the following

[root@cwp2 scripts]# service mariadb status
Redirecting to /bin/systemctl status mariadb.service
Б≈▐ mariadb.service - MariaDB 10.1.38 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           Б■■Б■─migrated-from-my.cnf-settings.conf
   Active: active (running) since Tue 2019-04-09 11:49:09 EEST; 11min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 7664 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 7616 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 7611 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
 Main PID: 7630 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /system.slice/mariadb.service
           Б■■Б■─7630 /usr/sbin/mysqld

Apr 09 11:59:11 cwp2 mysqld[7630]: 2019-04-09 11:59:11 140547070831360 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'
Apr 09 11:59:17 cwp2 mysqld[7630]: 2019-04-09 11:59:17 140547070831360 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'
Apr 09 11:59:33 cwp2 mysqld[7630]: 2019-04-09 11:59:33 140547070831360 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'
Apr 09 11:59:36 cwp2 mysqld[7630]: 2019-04-09 11:59:36 140547070831360 [ERROR] Got error 127 when reading table './ida_sport/cache'
Apr 09 11:59:36 cwp2 mysqld[7630]: 2019-04-09 11:59:36 140547070831360 [ERROR] Got error 127 when reading table './ida_sport/cache'
Apr 09 11:59:43 cwp2 mysqld[7630]: 2019-04-09 11:59:43 140547070831360 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'
Apr 09 11:59:49 cwp2 mysqld[7630]: 2019-04-09 11:59:49 140547070831360 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'
Apr 09 11:59:57 cwp2 mysqld[7630]: 2019-04-09 11:59:57 140547070831360 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'
Apr 09 12:00:04 cwp2 mysqld[7630]: 2019-04-09 12:00:04 140547070831360 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'
Apr 09 12:00:11 cwp2 mysqld[7630]: 2019-04-09 12:00:11 140547088923392 [ERROR] Got error 127 when reading table './geekshop_shop/oc_product'

Offline
*
Re: how to debug mariadb crash?
« Reply #1 on: April 09, 2019, 11:23:41 AM »
cwp has two monitoring tools, recommend is cwppro monit, but this is not the solution!

A database must be fully stable and it can't crash so you must find and resolve the issue, in some cases that could mean export all with mysqldump and then reinstall database and import all data back again.
VPS & Dedicated server provider with included FREE Managed support for CWP.
http://www.studio4host.com/

*** Don't allow that your server or website is down, choose hosting provider with included expert managed support for your CWP.

Offline
*
Re: how to debug mariadb crash?
« Reply #2 on: April 09, 2019, 02:49:57 PM »
I think too that is is must be stable of course, my question was how to understand why it is being crashed. Right now process was dead again but can locate no info why in logs (

Offline
*****
Re: how to debug mariadb crash?
« Reply #3 on: April 10, 2019, 05:55:48 AM »
Hi, are you sure you checking correct logs? Too often it crashes due not enough RAM.
Check the MariaDB configuration file for the log file location.
You can ask me to solve any problem with your server for some money in pm  ;)
Services Monitoring & RBL Monitoring
http://centos-webpanel.com/services-monitor
Join our Development Team and get paid !
http://centos-webpanel.com/develope-modules-for-cwp

Installation Instructions
http://centos-webpanel.com/installation-instructions
Get Fast Support Here
http://centos-webpanel.com/support-services

Offline
*
Re: how to debug mariadb crash?
« Reply #4 on: April 11, 2019, 06:43:47 AM »
Which logs to check is a my main question )

journalctl -u mariadb shows

Apr 11 06:05:19 cwp2 systemd[1]: mariadb.service: main process exited, code=killed, status=9/KILL
Apr 11 06:05:19 cwp2 systemd[1]: Unit mariadb.service entered failed state.
Apr 11 06:05:19 cwp2 systemd[1]: mariadb.service failed.
Apr 11 06:05:24 cwp2 systemd[1]: mariadb.service holdoff time over, scheduling restart.
Apr 11 06:05:50 cwp2 systemd[1]: Stopped MariaDB 10.1.38 database server.
Apr 11 06:07:49 cwp2 systemd[1]: Starting MariaDB 10.1.38 database server...
Apr 11 06:07:49 cwp2 systemd[1]: mariadb.service start-pre operation timed out. Terminating.
Apr 11 06:09:21 cwp2 systemd[1]: mariadb.service stop-final-sigterm timed out. Skipping SIGKILL. Entering failed mode.
Apr 11 06:09:22 cwp2 systemd[1]: Failed to start MariaDB 10.1.38 database server.
Apr 11 06:09:22 cwp2 systemd[1]: Unit mariadb.service entered failed state.
Apr 11 06:09:22 cwp2 systemd[1]: mariadb.service failed.

I've added 4gb extra memory...

Offline
*****
Re: how to debug mariadb crash?
« Reply #5 on: April 11, 2019, 06:12:56 PM »
Hi, the path to log must be set in /etc/my.cnf.d/server.cnf
Also, check if all files in /var/lib/mysql owned by mysql. The command below will change the owner to correct:
Code: [Select]
chown -R mysql:mysql /var/lib/mysql
You can ask me to solve any problem with your server for some money in pm  ;)
Services Monitoring & RBL Monitoring
http://centos-webpanel.com/services-monitor
Join our Development Team and get paid !
http://centos-webpanel.com/develope-modules-for-cwp

Installation Instructions
http://centos-webpanel.com/installation-instructions
Get Fast Support Here
http://centos-webpanel.com/support-services