Custom Backup

PMSL at the emails - spammer! ban, ban, ban 

Nah they were legit emails (sending to Amazon SES and then out to the world) hence why I rushed to cancel the backup process.. just in case!

HH:MM remains so that folks can run more than once a day, particularly during testing. Of course, doesn't affect directory sorts etc.

Got it, yea me likely

Yeah, during a test (and later needed) restore I spotted the subdir creation. As you said, ain't major but a bit unpolished. 
Maybe a v3.2 will address this - who knows?

EPIC NEWS!


OK, so one more thing. I have moved the tmp folder to the storage drive. It's not 'ideal' but helps reduce the load on the main drive when large accounts are zipped (main drive storage is much more expensive, and once I purchase allocated space i cant just 'downsize' it again, so i keep the main drive at about 80% capacity.
I'm a bit of an idiot and didn't separate the /home/ folder onto its own drive so that's always going to be at the back of my mind. Something for a rainy day!
I'd also mention that each account goes over its quota while the backup is running. I assume this is because (for example) 8/10GB used quickly becomes 12-16/10GB while the task is running.

On the downside, could this affect a users ability to receive mail or publish content on their site?
Might be a small flaw, albeit only for a short time. Something to ponder

Got the archived directory structure better but the split version is causing weird problems, so shelved for now (or until inspiration happens).
Tried out xz for compression and whilst it reduces backup size by a useful approx. 12% it also consumes more runtime CPU. Additionally, the CWP file manager doesn't recognise it as being able to be "descompressed" (yes another spelling mistake). I'll stay clear of pigz/zstd because they aren't available by default on CWP.

Optimal I/O will be where home, tmp_bak and backup are all on different storage - only likely in larger systems. Certainly, keeping root, home and backup on different partitions allows for better filesystem types (ext2 for backup) and allocation of quotas. With anything above say, 10GB storage /home should never be in the same partition as root and is a basic noob/Windoze-blinkered mistake.

During testing, with partitioned drives of course, I could see no temporary files being created during the compression process. However, this was with archives up to only about 120MB. I should imagine some 'streaming' to disc does happen with large data sets and changing directory to begin at another location might help, though that might negate trimming of the archive file structure. When a tar is created from a current directory files can easily have a relative path.

(I don't trust CWP to run my main/larger client sites, with only one live e-commerce site on CWP. I have 3 Pro licences! I still put up with punitive WHM/cPanel costs for most and the unintuitive DirectAdmin for some sites.)

Got the archived directory structure better but the split version is causing weird problems, so shelved for now (or until inspiration happens).

What problems did you face, could you describe? I understand this is super heavy on CPU which is a bit distressing but..

the CWP file manager doesn't recognise it as being able to be "descompressed"

Gets me every time.. 

After much gnashing of teeth and far too much time spent on it, I present below the latest iteration. I know there's a splattering of probably superfluous code (sync/wait) but this is what I got to work.
That's it for now though as I must concentrate on other things, when it comes to coding etc.
As always, use entirely at your own risk and to supplement the inbuilt CWP backup.

Code: [Select]

#!/usr/bin/bash
# CWP Custom backup
# Version 4.0 ejsolutions/Teo/cynique
# Use split as a parameter to retain split backups
# Set the following 3 items to suit
tmp_dir=/home/tmp_bak/
backup_dir=/backup/custom/
retention=1
# Optional though advisable
# Create a backup_exclude.conf file in each user home directory,
# listing directories to exclude, for example
# backupcwp/
# tmp/
# cache/
# -------------------
if [ -d ${tmp_dir} ]; then
 mkdir -p ${tmp_dir}
fi
if [ -d ${backup_dir} ]; then
 mkdir -p ${backup_dir}
fi
now_dir=${backup_dir}$(date -d "today" +"%Y%m%d%H%M")
echo "Timestamped location: " ${now_dir}
# -------------------
mysql root_cwp -B -N -s -e "SELECT username,domain FROM user WHERE backup='on'" | while read -r username domain
do

 echo Custom backup task starting for $username at $domain
 mkdir -p ${now_dir}/accounts/
 echo Copying home directory
 mkdir -p ${tmp_dir}${username}/home_dir
 sync
 if [ -f /home/${username}/backup_exclude.conf ]; then
ionice -c 3 nice rsync -a --exclude-from=/home/${username}/backup_exclude.conf /home/${username}/ ${tmp_dir}${username}/home_dir \
  && sync
wait $!
 else
ionice -c 3 nice rsync -a /home/${username}/ ${tmp_dir}${username}/home_dir && sync
wait $!
 fi

 echo Backing up databases
 mkdir -p ${tmp_dir}${username}/mysql/
 mysql --defaults-extra-file=/root/.my.cnf -e "show databases LIKE '${username}%';" | grep -v Database | while read databasename
  do
     echo Dumping $databasename
     nice -n 5 mysqldump --defaults-extra-file=/root/.my.cnf "$databasename" > ${tmp_dir}${username}/mysql/"$databasename.sql" \
               && sync && \
nice gzip ${tmp_dir}${username}/mysql/"$databasename.sql"
  done
 if [ -d /var/vmail/${domain} ]; then
  mkdir -p ${tmp_dir}${username}/vmail/
  echo Copying email
  ionice -c 3 nice cp -fR /var/vmail/${domain} ${tmp_dir}${username}/vmail/
 fi
 if [ "split" = "$1" ]; then
  mkdir -p ${now_dir}/accounts/${username}
sync
for i in home_dir mysql vmail
do
  echo "Compressing " $i
  ionice -c 3 nice -n 15 tar -C ${tmp_dir}${username} -cjf ${now_dir}/accounts/${username}/$i.tar.bz2 $i \
&& sync
wait $!
# Remove # from line below if experiencing file/buffer cache problems
# sleep 1m
done
sync
 else
echo Consolidating files
ionice -c 3 nice -n 15 tar -C ${tmp_dir} -cjf ${now_dir}/accounts/${username}.tar.bz2 ${username}  && sync
wait $!
 fi
  sync
  wait $!
  echo Cleaning up ${tmp_dir}${username}
  rm -Rf ${tmp_dir}${username}
  echo /---------------------------------/
done
sync
/usr/bin/find ${backup_dir} -mtime +${retention} -delete > /dev/null 2>&1
echo Custom Backup Job Finished



Cynique, may I ask how you'd go about restoring a mailbox from the vmail files in the backup?

I've tried and it seems that these files are neither importable, nor can they be directly dragged into a vmail folder to recreate the mailbox.
Stuck a bit.. just testing on one mail account to see if this is a viable solution but not yet managing to restore from files and folders.

Thought it might help

..how you'd go about restoring a mailbox from the vmail files in the backup?

I've tried and it seems that these files are neither importable, nor can they be directly dragged into a vmail folder to recreate the mailbox.

IIRC, I created the email account first in admin, then just copied the relevant vmail user directory. IF I get some time/want a diversion, I'll do a test on my failover server.

Didn't manage it in the end but Igor from CWP support sorted it out.
It would be cool if there was a guide on how to do this somewhere, have mentioned it.. you never know!

The way you did it helped me a lot
I could easily back up with this method
Thank you very much for your help 

...
Thank you very much for your help 

Thanks for your feedback; it encourages more development, when I find time.
Now off to build yet another CWP VPS.

Hello,

Anyone still using this script? I am getting strange errors with CWP backups and support team have no update since weeks. I want to use some alternative as CWP team can take ages to fix the issues in backup.

I want to use some alternative as CWP team can take ages to fix the issues in backup.

Try this post, used these backups for a couple of years and restored from them on occasion - worked well enough for me!
https://forum.centos-webpanel.com/index.php?topic=11268.msg38452#msg38452

Works Great...Thank You