MySql auto shutdown at night

Hello,

Until Sunday last everything works very well. In the 1st of August a big backup eat all my root's partition space what stops mysql from working. I deleted that backup and restart mysql, then everything works fine again.

BUT..
when i woke up the next day, i find that all websites have an mysql connexion error, i connected to do CWP admin to see what happen, than i had this message on the index :

mysqld (pid 16175) is running...

Trying to start mysql server, please reload browser!

Could not connect: No such file or directory

I reload the page, and the CWP works fine (and all other website by the way), just by refreshing the CWP index it seems that there is no Mysql error anymore.
I open the log file to see what happen and i noticed that in 4H00 am, the mysql was shutted down and it start again on the exact time when i reloead my CWP index page.

I ignored this probleme hopping that will not happen again, but the day after the same thing happen, mysql shutting down at late night, and when connecting to CWP everything works fine after realoding this page, the only difference was the time when Mysql was Shutdown, it was 03h11AM and not 04H00AM (so i don't think it's a cron).

I added today this line to the crontab file as a mini solution : checking mysql service every 15min, and start it if it's down, and i hope it will fix temporary this problem, until i find the real cause and solve it.

Code: [Select]

*/15 * * * * /sbin/service mysqld status || service mysqld start
The server is a good machine, an 8 core xeon with 32gb of ram, and there is only 3 websites in. The estimated traffic for the 3 website is 20k/25k visitor per day, and we use cache. i don't think at all that the server is over 20% charge all the day.

this is the log file, i hope someone can help me, because the server is a production one:

Code: [Select]

2016-08-02 20:30:01 4361 [Warning] IP address '123.249.45.132' could not be resolved: Name or service not known
2016-08-03 00:13:21 4361 [Warning] IP address '210.212.98.235' could not be resolved: Name or service not known
2016-08-03 00:17:13 4361 [Warning] IP address '114.112.1.117' could not be resolved: Name or service not known
2016-08-03 03:11:01 4361 [Note] /usr/sbin/mysqld: Normal shutdown

2016-08-03 03:11:01 4361 [Note] Giving 1 client threads a chance to die gracefully
2016-08-03 03:11:01 4361 [Note] Event Scheduler: Purging the queue. 0 events
2016-08-03 03:11:01 4361 [Note] Shutting down slave threads
160803 03:11:02 mysqld_safe A mysqld process already exists
2016-08-03 03:11:03 4361 [Note] Forcefully disconnecting 0 remaining clients
2016-08-03 03:11:03 4361 [Note] Binlog end
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'partition'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'ARCHIVE'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_METRICS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_CMPMEM'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_CMP'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_LOCKS'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'INNODB_TRX'
2016-08-03 03:11:03 4361 [Note] Shutting down plugin 'InnoDB'
2016-08-03 03:11:03 4361 [Note] InnoDB: FTS optimize thread exiting.
2016-08-03 03:11:03 4361 [Note] InnoDB: Starting shutdown...
2016-08-03 03:11:04 4361 [Note] InnoDB: Shutdown completed; log sequence number 919189441
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'BLACKHOLE'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'CSV'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'MyISAM'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'MEMORY'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'MRG_MYISAM'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'sha256_password'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'mysql_old_password'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'mysql_native_password'
2016-08-03 03:11:04 4361 [Note] Shutting down plugin 'binlog'
2016-08-03 03:11:04 4361 [Note] /usr/sbin/mysqld: Shutdown complete

160803 03:11:05 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

you should check all other logs at that time to determinate the possible issue or you can contact our support to assist you with that.

Hello,

I have the same problem since 2 days.
It's seems to be the backup cron at 4AM who stop the mysql server. However, I have no clue on why the mysql server didn't restart after. 

backup is NOT stopping mysql...so its probably related with some issue with one of the databases/tables try to run mysqldumpo manually for each database to see which has issues.

I SOLVED IT !! 

i dont think it's a table or DB probleme, because there is no error in log file and also (what make me sure) i noticed that the problem happen when i restart server, mysql dont start with other services, until i start it.

This is the way i solved it :

i add this line to the hourly and daily cron file (check if mysql service is dawn and restart it):

Code: [Select]

/sbin/service mysqld status || /sbin/service mysqld start
and i executed this command only one time (connected as root) :

Code: [Select]

$ sudo /sbin/chkconfig --level 3 mysqld on
i found this solution in this post (it seems like a Centos issue) :  http://dba.stackexchange.com/a/44742/103387
They saied :

I ran into this problem on a managed system. Took me a while to spot the very obvious cause, which is that the system as a whole had gone down for scheduled maintenance -- I guess because an updated kernel package was available.

So obvious!

The fix was of course to change the system configuration so that MySQL was automatically started when the system comes up. This was on CentOS

I hope it can help someone if had this problem

I have that same problem.

MySQL closes every night between at 3 and 4.





No attack server....

emresycn i writted the solution.

Connect to you server with putty as root and execute this command line

Code: [Select]

sudo /sbin/chkconfig --level 3 mysqld on
And (optionnaly) add this line to the crontab:
*/30 * * * * /sbin/service mysqld status || /sbin/service mysqld start

I use it

If you need to auto restart Apache and MySQL in case one or both are crashed, just schedule this SH script on Centos (RHEL) in your cron section.

Save the code section below as restart_apache_mysql.sh in your home directory.
Give the file permissions, so root can execute the script and schedule it in Cron.

Code: [Select]

chmod 755 /home/restart_apache_mysql.sh

Code: [Select]

#!/bin/bash
 
# MYSQL SECTION
RESTARTM="/etc/rc.d/init.d/mysqld restart"
MYSQLD="mysqld"
$PGREP ${MYSQLD}
if [ $? -ne 0 ]
then
$RESTARTM
fi


But most people have this problem. Always at the same times... 

But most people have this problem. Always at the same times... 

Yes, i think it's not a probleme about CWP, but it's an Centos system's update or verification that works early at night and stop mysql service before staring.
I hope they will fix this in their future versions/updates

With 0.9.8.48 when cron.php runs i can see in MariaDB log
[Note] /usr/sbin/mysqld: Normal shutdown
but MariaDB service never restarts.

i can replicate the issue by manually running

/usr/local/cwp/php54/bin/php -d max_execution_time=1000000 -q /usr/local/cwpsrv/htdocs/resources/admin/include/cron.php

any thoughts on this?

Happened to me last night, 4am "normal shutdown" in the mysql log file. CWP version: 0.9.8.50 MySQL didn't restart on it's own but came up when I manually restarted it. 

/var/log/cron
Sep 13 03:36:01 srv1 run-parts(/etc/cron.daily)[31458]: starting cwp
Sep 13 03:36:04 srv1 run-parts(/etc/cron.daily)[31561]: finished cwp

/var/log/mysqld
160913  3:36:03 [Note] /usr/libexec/mysqld: Normal shutdown
160913  3:36:03 [Note] Event Scheduler: Purging the queue. 0 events
160913  3:36:05 [Note] /usr/libexec/mysqld: Shutdown complete
160913 03:36:05 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Maybe the problem occurs if it takes longer for mysqld to shut down than it does for the script to run?

I got the same error in me tonight I'm MySQL problem.

Another error;

panel we want to throw restart MySQL gives the following error.

Code: [Select]

Warning: Error while sending QUERY packet. PID=11313 in /usr/local/cwpsrv/htdocs/resources/admin/include/functions.php on line 0
MySQL server has gone away
Warning: [obfuscated]() expects parameter 1 to be resource, boolean given in /usr/local/cwpsrv/htdocs/resources/admin/include/functions.php on line 0

I got the same error in me tonight I'm MySQL problem. is there a solution to this problem? I have also implemented cron but it did not help and still error.

On my system the problem vanished after enabling chkconfig on MySql.

sudo chkconfig mysqld on

Best regards