Custom Backup

[ejsolutions]

Run this as a nightly cron task, calling it for example custom-backup.sh

• Ensure that you have /backup/custom and /home/tmp_bak directories (change to suit).
• Adjust the retention value to suit.
• Optional: create a /home/username/backup_exclude.conf to ignore cache, tmp etc. Just list each on a different line.

Caveats: currently untested for mail forwarders or subdomains.

This will create local backups only which you can then FTP/rsync (whatever) to a remote location, with a different cron task running, say an hour later.
Backups are stored in three directories: home_dir, mysql and vmail - this way you can restore only a portion of a backup if needs be. Just unzip/untar to a temporary location and grab the files that you need.

Code Select Expand

#!/usr/bin/bash
tmp_dir=/home/tmp_bak/
backup_dir=/backup/custom/
retention=2
# -------------------
mysql root_cwp -B -N -s -e "SELECT username,domain FROM user WHERE backup='on'" | while read -r username domain
do

echo Custom backup task starting for $username at $domain
mkdir -p ${tmp_dir}${username}/home_dir
echo Copying home directory
ionice -c 3 nice -n +19 rsync -a /home/${username}/ ${tmp_dir}${username}/home_dir
echo Backing up databases
mkdir -p ${tmp_dir}${username}/mysql/
mysql --defaults-extra-file=/root/.my.cnf -e "show databases LIKE '${username}%';" | grep -v Database | while read databasename
do
     echo Dumping $databasename
     mysqldump --defaults-extra-file=/root/.my.cnf "$databasename" > ${tmp_dir}${username}/mysql/"$databasename.sql" \
               2> ${tmp_dir}${username}/mysql/errors.txt

done
if [ -d /var/vmail/${domain} ]; then
mkdir -p ${tmp_dir}${username}/vmail/
echo Copying email
ionice -c 3 nice -n +19 cp -fR /var/vmail/${domain} ${tmp_dir}${username}/vmail/
fi
echo Consolidating files
if [ -f /home/${username}/backup_exclude.conf ]; then
ionice -c 3 nice -n +19 tar -cjf ${tmp_dir}${username}.tar.bz2 --exclude-from=/home/${username}/backup_exclude.conf ${tmp_dir}${username}
else
ionice -c 3 nice -n +19 tar -cjf ${tmp_dir}${username}.tar.bz2 ${tmp_dir}${username}
fi
mv ${tmp_dir}${username}.tar.bz2 ${backup_dir}${username}-$(date -d "today" +"%Y%m%d%H%M").tar.bz2
echo Cleaning up
/usr/bin/find ${backup_dir} -name "*.bz2" -mtime +${retention} -delete > /dev/null 2>&1
rm -Rf ${tmp_dir}${username}

done
echo Custom Backup Job Finished



Use at your own risk!
I currently use this to supplement the CWP (new) backup function.

[PiotrE]

Thank You!! Script finally give CWP the ability to backup databases. With both buildin options I culdnt do this at all!

[Teo]

Thanks for sharing this script.
What about moving the "if backup_exclude.conf" before to speedup the process and limiting the disk usage?
(the backup_exclude.conf lines have to be relative to user home, so if I need to exclude the folder /home/myuser/public_html/var/cache, I'll write public_html/var/cache/)

Code Select Expand

#!/usr/bin/bash
tmp_dir=/home/tmp_bak/
backup_dir=/backup/custom/
retention=2
# -------------------
mysql root_cwp -B -N -s -e "SELECT username,domain FROM user WHERE backup='on'" | while read -r username domain
do

echo Custom backup task starting for $username at $domain
mkdir -p ${tmp_dir}${username}/home_dir
echo Copying home directory
if [ -f /home/${username}/backup_exclude.conf ]; then
ionice -c 3 nice -n +19 rsync -a --links --exclude-from=/home/${username}/backup_exclude.conf /home/${username}/ ${tmp_dir}${username}/home_dir
else
ionice -c 3 nice -n +19 rsync -a --links /home/${username}/ ${tmp_dir}${username}/home_dir
fi
echo Backing up databases
mkdir -p ${tmp_dir}${username}/mysql/
mysql --defaults-extra-file=/root/.my.cnf -e "show databases LIKE '${username}%';" | grep -v Database | while read databasename
do
     echo Dumping $databasename
     mysqldump --defaults-extra-file=/root/.my.cnf "$databasename" > ${tmp_dir}${username}/mysql/"$databasename.sql" \
               2> ${tmp_dir}${username}/mysql/errors.txt

done
if [ -d /var/vmail/${domain} ]; then
mkdir -p ${tmp_dir}${username}/vmail/
echo Copying email
ionice -c 3 nice -n +19 cp -fR /var/vmail/${domain} ${tmp_dir}${username}/vmail/
fi
echo Consolidating files
ionice -c 3 nice -n +19 tar -cjf ${tmp_dir}${username}.tar.bz2 ${tmp_dir}${username}
mv ${tmp_dir}${username}.tar.bz2 ${backup_dir}${username}-$(date -d "today" +"%Y%m%d%H%M").tar.bz2
echo Cleaning up
/usr/bin/find ${backup_dir} -name "*.bz2" -mtime +${retention} -delete > /dev/null 2>&1
rm -Rf ${tmp_dir}${username}

done
echo Custom Backup Job Finished


[cinique]

@Teo
Good point. The advantage of sharing the script.
I've also made some recent tweaks but want to go a bit further with them.

[cinique]

Note:

Code Select Expand

rsync -a --links
AFAIK, -a (archive) implies -l (links). I'd be extremely careful when messing about with symlinks, though and just be content with -a. Generally speaking, don't use the --links, -l or -L options.

[vtheod]

Hi
When I try to run your backup script I got the following error:

custom-backup.sh: line 40: syntax error: unexpected end of file

Can you please help?

[cinique]

Hi
When I try to run your backup script I got the following error:

custom-backup.sh: line 40: syntax error: unexpected end of file

Can you please help?

Assuming that you mean the first script posted.
A 20 second Google and I think you may be a Windoze luser..

I think file.sh is with CRLF line terminators.
run
dos2unix file.sh
then the problem will be fixed.
You can install dos2unix in ubuntu with this:
sudo apt-get install dos2unix

Substitute file.sh (obviously?) and "sudo apt-get" with yum (maybe less obvious).
Better still, create the file directly on your server using vi (perhaps nano is entering CR_LF, which I doubt - I rarely use it.)

[vtheod]

A 20 second Google and I think you may be a Windoze luser..

You're so right... Unfortunately I'm a windoze luser hahaha
I prefered to create the file with vi editor on my server

Thank you very much!

[cinique]

You're so right... Unfortunately I'm a windoze luser hahaha
I preferred to create the file with vi editor on my server
Thank you very much!

All is not lost if you used vi 
You're welcome.

(dos2unix took me back 30+years !)

[FreshLondon]

I currently use this to supplement the CWP (new) backup function.

Wonderful script, thank you!
Testing atm from CLI, looks great so far. Could really use a limiter on the bzip process. Although its only using one vCPU during the zip process it does bounce up to 100% which would be nice to avoid for a live server 

Oh and just FYI chaps, if you're adding this then write it in VI/VIM/nano from CLI and not from the 'file manager', it messes with encoding.

[cinique]

Whilst running, keep an eye on "top -c -d5"

Code Select Expand

ionice -c 3 nice -n +19 tar -cjf ${tmp_dir}${username}.tar.bz2 ${tmp_dir}${username}
The combination of ionice & nice is supposed to limit the disc & CPU loading, respectively.

Try removing the '+' before 19.

[cinique]

Latest version.

Code Select Expand

#!/usr/bin/bash
# CWP Custom backup
# Version 2.1 ejsolutions/Teo/cynique
# Set the following 3 items to suit
tmp_dir=/home/tmp_bak/
backup_dir=/backup/custom/
retention=1
# Optional though advisable
# Create a backup_exclude.conf file in each user home directory,
# listing directories to exclude, for example
# backupcwp/
# tmp/
# cache/
# -------------------
if [ -d ${tmp_dir} ]; then
mkdir -p ${tmp_dir}
fi
if [ -d ${backup_dir} ]; then
mkdir -p ${backup_dir}
fi
# -------------------
mysql root_cwp -B -N -s -e "SELECT username,domain FROM user WHERE backup='on'" | while read -r username domain
do

echo Custom backup task starting for $username at $domain
mkdir -p ${tmp_dir}${username}/home_dir
echo Copying home directory
if [ -f /home/${username}/backup_exclude.conf ]; then
ionice -c 3 nice rsync -a --exclude-from=/home/${username}/backup_exclude.conf /home/${username}/ ${tmp_dir}${username}/home_dir
else
ionice -c 3 nice rsync -a /home/${username}/ ${tmp_dir}${username}/home_dir
fi
echo Backing up databases
mkdir -p ${tmp_dir}${username}/mysql/
mysql --defaults-extra-file=/root/.my.cnf -e "show databases LIKE '${username}%';" | grep -v Database | while read databasename
do
     echo Dumping $databasename
     nice -n 5 mysqldump --defaults-extra-file=/root/.my.cnf "$databasename" > ${tmp_dir}${username}/mysql/"$databasename.sql" \
               2> ${tmp_dir}${username}/mysql/errors.txt && sync && \
nice gzip ${tmp_dir}${username}/mysql/"$databasename.sql"
done
if [ -d /var/vmail/${domain} ]; then
mkdir -p ${tmp_dir}${username}/vmail/
echo Copying email
ionice -c 3 nice cp -fR /var/vmail/${domain} ${tmp_dir}${username}/vmail/
fi
echo Consolidating files
ionice -c 3 nice -n 15 tar -cjf ${tmp_dir}${username}.tar.bz2 ${tmp_dir}${username} && \
mv ${tmp_dir}${username}.tar.bz2 ${backup_dir}${username}-$(date -d "today" +"%Y%m%d%H%M").tar.bz2
echo Cleaning up
/usr/bin/find ${backup_dir} -name "*.bz2" -mtime +${retention} -delete > /dev/null 2>&1
rm -Rf ${tmp_dir}${username}

done
echo Custom Backup Job Finished



Notes:
Values of nice have been chosen with particular attention, to not coincide with clamd and help the sequence of the script.
As with all backups, this script should be run at quiet times on your server.

[FreshLondon]

Latest version.

Thank you! Testing now

One thing that would be SUPER cool is putting that days backups into its own folder, 20201118 for example.
I do understand that if a backup goes on past midnight it would go to a new folder but the idea is still super cool and useful (I plan on keeping 14+ days retention), would it be possible to add folders? 

or even go further and split the backup?
backups/date/username/mail.zip
backups/date/username/public_html.zip
backups/date/username/databases.zip

[cinique]

Actually, I think I'll do a spot of reconfiguring, to more closely mimic a WHM/cPanel backup, so yes backups grouped by date. This will help with my multiple backups to remote locations - the key reason to have backups consolidated (larger block size).

I'll consider a parameter, to allow keeping the backups split.

Ideally backups should run between the hours of 02:00 and say, 06:30 (accounting for runtime), so date isn't an issue.

[FreshLondon]

If its of interest to you, the result of your most recent snippet:

Works really well and although its very CPU heavy this isn't a clash with ClamAV/Varnish as those are mostly just hogging RAM

Actually, I think I'll do a spot of reconfiguring, to more closely mimic a WHM/cPanel backup, so yes backups grouped by date. This will help with my multiple backups to remote locations - the key reason to have backups consolidated (larger block size).

Amazing, really excited!

I'll consider a parameter, to allow keeping the backups split.

Yes, and if you segment the tasks imagine transferring one segment (say Mail.zip) to the backups location while the next task (say MySQL.zip) could be already preparing.. ooooo the possibilities!