Here is how I backup a number of Ubuntu servers with crontab entries, bash scripts and rsync and send backup email.

Read more on useful terminal commands here for as low as $2.5 a month. Read on setting up a Digital Ocean Ubuntu server here for as low as $5 a month here ($10 free credit). Read more on setting up an AWS Ubuntu server here.

I have  6 numbered scripts in my scripts folder that handle backups, I call these scripts at set times via the crontab list.

fyi: Paths below have been changed for the purpose of this post (security).

1 1 * * * /bin/bash /scripts-folder/0.backupfiles.sh >> /backup-folder/0.backupfiles.log
3 1 * * * /bin/bash /scripts-folder/1.backupdbs.sh >> /backup-folder/1.backupdbs.log
5 1 * * * /bin/bash /scripts-folder/2.shrinkmysql.sh >> /backup-folder/2.shrinkmysql.log
10 1 * * * /bin/bash /scripts-folder/3.addtobackuplog.sh >> /backup-folder/3.addtobackuplog.log
11 1 * * * /bin/bash /scripts-folder/4.syncfiles.sh >> /backup-folder/4.syncfiles.log
15 1 * * * /bin/bash /scripts-folder/5.sendbackupemail.sh > /dev/null 2>&1

https://crontab.guru/ is great for specifying times to run jobs on each server (I backup one server at 1 AM,, another at 2 AM etc (never at the same time))

Bring up your crontab list

crontab -e

Check out the Crontab schedule generator here.

Below is the contents of my /scripts/0.backupfiles.sh (sensitive information removed).

I use this script to backup folders and configuration data

cat /scripts-folder/0.backupfiles.sh
#!/bin/bash

echo "Deleting old NGINX config..";
rm /backup-folder/config-nginx.zip

echo "Backing Up NGNIX..";
zip -r -9 /backup-folder/config-nginx.zip /etc/nginx/ -x "*.tmp" -x "*.temp" -x"./backup-folder/*.bak" -x "./backup-folder/*.zip"

echo "Deleting old www backup(s) ..";
#rm /backup-folder/www.zip
echo "Removing old www backup folder";
rm -R /backup-folder/www
echo "Making new backup folder at /backup-folder/www/";
mkdir /backup-folder/www

echo "Copying /www/ to /backup-folder/www/";
cp -rTv /www/ /backup-folder/www/
echo "Done copying /www/ to /backup-folder/www/";

Below is the contents of my /scripts-folder/1.backupdbs.sh (sensitive information removed).

I use this script to dump my MySQL database.

cat /scripts-folder/1.backupdbs.sh
#!/bin/bash

echo "$(date) 1.backupdbs.sh ...." >> /backup-folder/backup.log

echo "Removing old SQL backup..":
rm /backup-folder/mysql/database-dump.sql

echo "Backing up SQL";
/usr/bin/mysqldump --all-databases > /backup-folder/mysql/database-dump.sql -u 'mysqluser' -p'EnterY@urMySQLR00tPa$word'

echo "Done backing up the database";

Below is the contents of my /scripts-folder/2.shrinkmysql.sh (sensitive information removed).

I use this script to tar my SQL dumps as these files can be quite big

cat /scripts-folder/2.shrinkmysql.sh
#!/bin/bash

echo "$(date) 2.shrinkmysql.sh ...." >> /backup-folder/backup.log

echo "Backing up MySQL dump..";
tar -zcf /backup-folder/mysql.tgz /backup-folder/mysql/

echo "Removing old MySQL dump..";
rm /backup-folder/mysql/*.sql

Below is the contents of my /scripts-folder/3.addtobackuplog.sh (sensitive information removed).

This script is handy for dumping extra information.

cat /scripts-folder/3.addtobackuplog.sh
#!/bin/bash

echo "$(date) 3.addtobackuplog.sh ...." >> /backup-folder/backup.log

echo "Server Name.." >> /backup-folder/backup.log
grep "server_name" /etc/nginx/sites-available/default

echo "$(date) Timec" >> /backup-folder/backup.log
sudo hwclock --show  >> /backup-folder/backup.log

echo "$(date) Uptime, Load etc" >> /backup-folder/backup.log
w -i >> /backup-folder/backup.log

echo "$(date) Memory" >> /backup-folder/backup.log
free  >> /backup-folder/backup.log

echo "$(date) Disk Space" >> /backup-folder/backup.log
pydf >> /backup-folder/backup.log

echo "Firewall" >> /backup-folder/backup.log
ufw status >> /backup-folder/backup.log

echo "Adding to Backup Log file..";
echo "$(date) Nightly MySQL Backup Successful....." >> /backup-folder/backup.log

Below is the contents of my /scripts-folder/4.syncfiles.sh (sensitive information removed).

This script is the workhorse routine that rsyncs files to the source to the backup server (a dedicated Vulr server with an A Name record attaching the server to my domain).

I installed sshpass to pass in the ssh user password (after ssh is connected (authorized_keys set), I tried to setup a rsync daemon but had no luck).  I ensured appropriate ports were opened on the source (OUT 22, 873) and backup server (IN 22 873).

cat /scripts-folder/4.syncfiles.sh
#!/bin/bash

echo "$(date) 4.syncfiles.sh ...." >> /backup-folder/backup.log
echo "Syncing Files.";

sudo sshpass -p 'Y0urW0rkingSSHR00tPa$0ord' rsync -a -e  'ssh -p 22 ' --progress -P /backup-folder backup-server.yourdomain.com:/backup-folder/1.www.server01.com/

ufw firewall has great rules for allowing certain IP’s to talk on ports.

Set Outbound firewall rules (to certain IP’s)

sudo ufw allow from 123.123.123.123 to any port 22

Change 123.123.123.123 to your backup server.

Set Inbound firewall rules (to certain IP’s)

sudo ufw allow out from 123.123.123.123 to any port 22

Change 123.123.123.123 to your sending server.

You can and should setup rate limits on IP’s hitting certain ports.

udo ufw limit 22 comment 'Rate limit for this port has been reached'

Install Fail2Ban to automatically ban certain users. Fail2Ban reads log file that contains password failure report
and bans the corresponding IP addresses using firewall rules.  Read more on securing Ubuntu in the cloud here.

Below is the contents of my /scripts-folder/5.sendbackupemail.sh (sensitive information removed).

This script sends an email and attaches a zip file of all log files generated through the backup process.

cat /scripts/5.sendbackupemail.sh
#!/bin/bash

echo "$(date) 5.sendbackupemail.sh ...." >> /backup-folder/backup.log

echo "Zipping up log Files.";

zip -r -9 /backup-folder/backup-log.zip /backup-folder/*.log

echo "Sending Email";
sendemail -f [email protected] -t [email protected] -u "Backup Alert" -m "server01 has been backed up" -s smtp.gmail.com:587 -o tls=yes -xu [email protected] -xp Y0urGSu1tePasswordG0e$Here123 -a /backup-folder/backup-log.zip

Read my guide on setting up sendmail here.

Security Considerations

You should never store passwords in scripts that talk to SSH connections, create MySQL dumps or when talking to email servers, I will update this guide when I solving all of these cases.  Also, create the least access required for user accounts where possible.

Target Server Configuration

Alos you can see in /scripts-folder/4.syncfiles.sh that I am saving to the ‘/backup-folder/1.www.server01.com/’ folder, you can make as many folders as you want to make the most of the backup server.  I would advise you not use the server for anything else like web servers and apps as this server is holding important stuff.

backup-server.yourdomain.com:/backup-folder/1.www.server01.com/

I have a handy script to delete all backups (handy during testing).

#!/bin/bash

echo "Deleting Backup Folders..........................................";

echo " Deleting /backup-folder/1.www.server01.com";
rm -R /backup-folder/1.www.server01.com

echo " Deleting /backup-folder/2.www.server02.com";
rm -R /backup-folder/2.www.server02.com

echo " Deleting /backup-folder/3.www.server03.com";
rm -R /backup-folder/3.www.server03.com

echo " Deleting /backup-folder/4.www.server04.com";
rm -R /backup-folder/4.www.server04.com

echo " Deleting /backup-folder/5.www.server05.com";
rm -R /backup-folder/5.www.server05.com

echo " Deleting /backup-folder/6.www.server06.com";
rm -R /backup-folder/6.www.server06.com

echo " Deleting /backup-folder/7.www.server07.com";
rm -R /backup-folder/7.www.server07.com

echo " Deleting /backup-folder/8.www.server08.com";
rm -R /backup-folder/8.www.server08.com

echo "
";

echo "Creating Backup Folders.........................................";

echo " Making folder /backup-folder/1.www.server01.com";
mkdir /backup-folder/1.www.server01.com

echo " Making folder /backup-folder/2.www.server02.com";
mkdir /backup-folder/2.www.server02.com

echo " Making folder /backup-folder/3.www.server03.com";
mkdir /backup-folder/3.www.server03.com";

echo " Making folder /backup-folder/4.www.server04.com";
mkdir /backup-folder/4.www.server04.com

echo " Making folder /backup-folder/5.www.server04.com";
mkdir /backup-folder/5.www.server04.com

echo " Making folder /backup-folder/6.www.server05.com";
mkdir /backup-folder/6.www.server04.com

echo " Making folder /backup-folder/7.www.server06.com";
mkdir /backup-folder/7.www.server04.com

echo " Making folder /backup-folder/8.www.server07.com";
mkdir /backup-folder/8.www.server08.com

echo "
";

echo "Backup Folder Contents.........................................";
ls /backup-folder -al
echo "
";

echo "Folder Strcuture...............................................";
cd /backup-folder
pwd
tree -a -f -p -h  -l -R

echo "
";

echo "How big is the backup folder...................................";
du -hs /backup-folder

echo "
";

echo "Done...........................................................";

Ensure your backup server is just for backups and only allows traffic from known IP’s

ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       123.123.123.123
22                         ALLOW       123.123.123.124
22                         ALLOW       123.123.123.125
22                         ALLOW       123.123.123.126
22                         ALLOW       123.123.123.127
22                         ALLOW       123.123.123.128
22                         ALLOW       123.123.123.129
22                         ALLOW       123.123.123.130
53                         ALLOW       Anywhere

22                         ALLOW OUT   123.123.123.123
22                         ALLOW OUT   123.123.123.124
22                         ALLOW OUT   123.123.123.125
22                         ALLOW OUT   123.123.123.126
22                         ALLOW OUT   123.123.123.127
22                         ALLOW OUT   123.123.123.128
22                         ALLOW OUT   123.123.123.129
22                         ALLOW OUT   123.123.123.130

Change the 123.x.x.x servers to your servers IP’s

Tip: Keep an eye on the backups with tools like ncdu

sudo ncdu /backup-folder
ncdu 1.11 ~ Use the arrow keys to navigate, press ? for help
--- /backup ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    1.0 GiB [##########] /6.www.server01.com
  462.1 MiB [####      ] /1.www.server02.com
  450.1 MiB [####      ] /5.www.server03.com
   60.1 MiB [          ] /2.www.server04.com
  276.0 KiB [          ] /3.www.server05.com
  276.0 KiB [          ] /4.www.server06.com
e   4.0 KiB [          ] /8.www.server07.com
e   4.0 KiB [          ] /7.www.server08.com

Installing SSH on OSX

If you want to backup to this server with OSyouou will need to install sshpass

curl -O -L http://downloads.sourceforge.net/project/sshpass/sshpass/1.06/sshpass-1.06.tar.gz && tar xvzf sshpass-1.06.tar.gz
cd sshpass-1.06
./configure
sudo make install

sshpass should be installed

sshpass -V
sshpass 1.06
(C) 2006-2011 Lingnu Open Source Consulting Ltd.
(C) 2015-2016 Shachar Shemesh
This program is free software, and can be distributed under the terms of the GPL
See the COPYING file for more information.

Using "assword" as the default password prompt indicator.

I have not got sshp[ass working yet error “Host key verification failed.”  I had to remove the back known host from ~/.ssh/known_hosts” on OSX

But this worked on OSX

rsync -a -e 'ssh -p 22 ' --progress -P ~/Desktop [email protected]:/backup/8.Mac/

Note: Enter the servers sshuser@ before the hostname or rsync will use the logged in OSX username

Don’t forget to check the backup serves disk usage often.

Output from backing up an incremental update (1x new folder)

localhost:~ local-account$ rsync -a -e  'ssh -p 22 ' --progress -P /Users/local-account/folder-to-backup [email protected]:/backup/the-computer/
[email protected]'s password: 
building file list ... 
51354 files to consider
folder-to-backup/
folder-to-backup/TestProject/
folder-to-backup/TestProject/.git/
folder-to-backup/TestProject/.git/COMMIT_EDITMSG
          15 100%    0.00kB/s    0:00:00 (xfer#1, to-check=16600/51354)
folder-to-backup/TestProject/.git/HEAD
          23 100%   22.46kB/s    0:00:00 (xfer#2, to-check=16599/51354)
folder-to-backup/TestProject/.git/config
         137 100%  133.79kB/s    0:00:00 (xfer#3, to-check=16598/51354)
folder-to-backup/TestProject/.git/description
          73 100%   10.18kB/s    0:00:00 (xfer#4, to-check=16597/51354)
folder-to-backup/TestProject/.git/index
        1581 100%  220.56kB/s    0:00:00 (xfer#5, to-check=16596/51354)
folder-to-backup/TestProject/.git/hooks/
folder-to-backup/TestProject/.git/hooks/README.sample
         177 100%   21.61kB/s    0:00:00 (xfer#6, to-check=16594/51354)
folder-to-backup/TestProject/.git/info/
folder-to-backup/TestProject/.git/info/exclude
          40 100%    4.88kB/s    0:00:00 (xfer#7, to-check=16592/51354)
folder-to-backup/TestProject/.git/logs/
folder-to-backup/TestProject/.git/logs/HEAD
         164 100%   20.02kB/s    0:00:00 (xfer#8, to-check=16590/51354)
folder-to-backup/TestProject/.git/logs/refs/
folder-to-backup/TestProject/.git/logs/refs/heads/
folder-to-backup/TestProject/.git/logs/refs/heads/master
         164 100%   20.02kB/s    0:00:00 (xfer#9, to-check=16587/51354)
folder-to-backup/TestProject/.git/objects/
folder-to-backup/TestProject/.git/objects/05/
folder-to-backup/TestProject/.git/objects/05/0853a802dd40cad0e15afa19516e9ad94f5801
        2714 100%  294.49kB/s    0:00:00 (xfer#10, to-check=16584/51354)
folder-to-backup/TestProject/.git/objects/11/
folder-to-backup/TestProject/.git/objects/11/729e81fc116908809fc17d60c8604aa43ec095
         105 100%   11.39kB/s    0:00:00 (xfer#11, to-check=16582/51354)
folder-to-backup/TestProject/.git/objects/23/
folder-to-backup/TestProject/.git/objects/23/768a20baaf8aa0c31b0e485612a5e245bb570d
         131 100%   12.79kB/s    0:00:00 (xfer#12, to-check=16580/51354)
folder-to-backup/TestProject/.git/objects/27/
folder-to-backup/TestProject/.git/objects/27/3375fc70381bd2608e05c03e00ee09c42bdc58
         783 100%   76.46kB/s    0:00:00 (xfer#13, to-check=16578/51354)
folder-to-backup/TestProject/.git/objects/2a/
folder-to-backup/TestProject/.git/objects/2a/507ef5ea3b1d68c2d92bb4aece950ef601543e
         303 100%   26.90kB/s    0:00:00 (xfer#14, to-check=16576/51354)
folder-to-backup/TestProject/.git/objects/2b/
folder-to-backup/TestProject/.git/objects/2b/f8bd93d56787a7548c7f8960a94f05c269b486
         136 100%   12.07kB/s    0:00:00 (xfer#15, to-check=16574/51354)
folder-to-backup/TestProject/.git/objects/2f/
folder-to-backup/TestProject/.git/objects/2f/900764e9d12d8da7e5e01ba34d2b7b2d95ffd4
         209 100%   17.01kB/s    0:00:00 (xfer#16, to-check=16572/51354)
folder-to-backup/TestProject/.git/objects/36/
folder-to-backup/TestProject/.git/objects/36/d2c80d8893178d7e1f2964085b273959bfdc28
         201 100%   16.36kB/s    0:00:00 (xfer#17, to-check=16570/51354)
folder-to-backup/TestProject/.git/objects/3d/
folder-to-backup/TestProject/.git/objects/3d/e5a02083dbe9c23731a38901dca9e913c04dd0
         130 100%   10.58kB/s    0:00:00 (xfer#18, to-check=16568/51354)
folder-to-backup/TestProject/.git/objects/40/
folder-to-backup/TestProject/.git/objects/40/40592d8d4d886a5c81e1369ddcde71dd3b66b5
         841 100%   63.18kB/s    0:00:00 (xfer#19, to-check=16566/51354)
folder-to-backup/TestProject/.git/objects/87/
folder-to-backup/TestProject/.git/objects/87/60f48ddbc9ed0863e3fdcfce5e4536d08f9b8d
          86 100%    6.46kB/s    0:00:00 (xfer#20, to-check=16564/51354)
folder-to-backup/TestProject/.git/objects/a9/
folder-to-backup/TestProject/.git/objects/a9/e6a23fa34a5de4cd36250dc0d797439d85f2ea
         306 100%   22.99kB/s    0:00:00 (xfer#21, to-check=16562/51354)
folder-to-backup/TestProject/.git/objects/b0/
folder-to-backup/TestProject/.git/objects/b0/4364089fdc64fe3b81bcd41462dd55edb7a001
          57 100%    4.28kB/s    0:00:00 (xfer#22, to-check=16560/51354)
folder-to-backup/TestProject/.git/objects/be/
folder-to-backup/TestProject/.git/objects/be/3b93d6d8896d69670f1a8e26d1f51f9743d07e
          60 100%    4.19kB/s    0:00:00 (xfer#23, to-check=16558/51354)
folder-to-backup/TestProject/.git/objects/d0/
folder-to-backup/TestProject/.git/objects/d0/524738680109d9f0ca001dad7c9bbf563e898e
         523 100%   36.48kB/s    0:00:00 (xfer#24, to-check=16556/51354)
folder-to-backup/TestProject/.git/objects/d5/
folder-to-backup/TestProject/.git/objects/d5/4e024fe16b73e5602934ef83e0b32a16243a5e
          69 100%    4.49kB/s    0:00:00 (xfer#25, to-check=16554/51354)
folder-to-backup/TestProject/.git/objects/db/
folder-to-backup/TestProject/.git/objects/db/3f0ce163c8033a175d27de6a4e96aadc115625
          59 100%    3.84kB/s    0:00:00 (xfer#26, to-check=16552/51354)
folder-to-backup/TestProject/.git/objects/df/
folder-to-backup/TestProject/.git/objects/df/cad4828b338206f0a7f18732c086c4ef959a7b
          51 100%    3.32kB/s    0:00:00 (xfer#27, to-check=16550/51354)
folder-to-backup/TestProject/.git/objects/ef/
folder-to-backup/TestProject/.git/objects/ef/e6d036f817624654f77c4a91ae6f20b5ecbe9d
          94 100%    5.74kB/s    0:00:00 (xfer#28, to-check=16548/51354)
folder-to-backup/TestProject/.git/objects/f2/
folder-to-backup/TestProject/.git/objects/f2/b43571ec42bad7ac43f19cf851045b04b6eb29
         936 100%   57.13kB/s    0:00:00 (xfer#29, to-check=16546/51354)
folder-to-backup/TestProject/.git/objects/fd/
folder-to-backup/TestProject/.git/objects/fd/f3f97d1b6e9d8d29bb69a88c4d89ca752bd937
         807 100%   49.26kB/s    0:00:00 (xfer#30, to-check=16544/51354)
folder-to-backup/TestProject/.git/objects/info/
folder-to-backup/TestProject/.git/objects/pack/
folder-to-backup/TestProject/.git/refs/
folder-to-backup/TestProject/.git/refs/heads/
folder-to-backup/TestProject/.git/refs/heads/master
          41 100%    2.50kB/s    0:00:00 (xfer#31, to-check=16539/51354)
folder-to-backup/TestProject/.git/refs/tags/
folder-to-backup/TestProject/TestProject.xcodeproj/
folder-to-backup/TestProject/TestProject.xcodeproj/project.pbxproj
       11476 100%  659.24kB/s    0:00:00 (xfer#32, to-check=16536/51354)
folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/
folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/contents.xcworkspacedata
         156 100%    8.96kB/s    0:00:00 (xfer#33, to-check=16534/51354)
folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/xcuserdata/
folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/xcuserdata/simon.xcuserdatad/
folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/xcuserdata/simon.xcuserdatad/UserInterfaceState.xcuserstate
        8190 100%  470.47kB/s    0:00:00 (xfer#34, to-check=16531/51354)
folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/
folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/
folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/xcschemes/
folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/xcschemes/TestProject.xcscheme
        3351 100%  192.50kB/s    0:00:00 (xfer#35, to-check=16527/51354)
folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/xcschemes/xcschememanagement.plist
         483 100%   27.75kB/s    0:00:00 (xfer#36, to-check=16526/51354)
folder-to-backup/TestProject/TestProject/
folder-to-backup/TestProject/TestProject/AppDelegate.swift
        2172 100%  117.84kB/s    0:00:00 (xfer#37, to-check=16524/51354)
folder-to-backup/TestProject/TestProject/Info.plist
        1442 100%   78.23kB/s    0:00:00 (xfer#38, to-check=16523/51354)
folder-to-backup/TestProject/TestProject/ViewController.swift
         505 100%   27.40kB/s    0:00:00 (xfer#39, to-check=16522/51354)
folder-to-backup/TestProject/TestProject/Assets.xcassets/
folder-to-backup/TestProject/TestProject/Assets.xcassets/AppIcon.appiconset/
folder-to-backup/TestProject/TestProject/Assets.xcassets/AppIcon.appiconset/Contents.json
        1077 100%   58.43kB/s    0:00:00 (xfer#40, to-check=16519/51354)
folder-to-backup/TestProject/TestProject/Base.lproj/
folder-to-backup/TestProject/TestProject/Base.lproj/LaunchScreen.storyboard
        1740 100%   94.40kB/s    0:00:00 (xfer#41, to-check=16517/51354)
folder-to-backup/TestProject/TestProject/Base.lproj/Main.storyboard
        1695 100%   91.96kB/s    0:00:00 (xfer#42, to-check=16516/51354)

sent 1243970 bytes  received 1220 bytes  75466.06 bytes/sec
total size is 10693902652  speedup is 8588.17

Update with no files to upload

localhost:~ local-account$ rsync -a -e  'ssh -p 22 ' --progress -P /Users/local-account/folder-to-backup [email protected]:/backup/the-computer/
[email protected]'s password: 
building file list ... 
51354 files to consider

sent 1198459 bytes  received 20 bytes  82653.72 bytes/sec
total size is 10693902652  speedup is 8922.90

Backup is easy..

rsync -a -e  'ssh -p 22 ' --progress -P /Users/local-account/folder-to-backup [email protected]:/backup/the-computer/

If you want incremental and full backups try Duplicity.

Hope this helps.

Donate and make this blog better

Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.7 Duplicity