Code, Security and Server Stuff
Views are my own and not my employer's.
This guide will show how you can automatically backup files to a Backblaze B2 clouds bucket with the Duplicati software on Windows and Linux.
Read the full article here: https://fearby.com/article/backing-up-files-to-a-backblaze-b2-cloud-bucket-with-duplicati/
This guide will show how you can automatically backup files to a Backblaze B2 clouds bucket with the Duplicati software on Windows and Linux.
I have blogged about how you can use the Backblaze personal backup program ( here, Use my link and get your first month of Backblaze backups free). The personal application from Backblaze is cool but what if you wanted to store 10GB for free (the first 10GB is free) or setup different buckets for different computers or applications.
Welcome to Backblaze B2 Cloud Storage.
In Australia B2 cloud costs (+GST).
Check our my other related posts
This works alongside the private Backblaze backup mentioned here.
How does Backblaze B2 Compare that to other cloud data storage provides.
Comparison chart.
| Provider | Storage ($/GB/Month) | Download ($/GB) |
| Backblaze | $0.005 | $0.01 |
| Amazon S3 | $0.021 | $0.05+ |
| Microsoft Azure | $0.018+ | $0.05+ |
| Google cloud | $0.020 | $0.08 |
Vendor cost breakdown (source).
Read here to view the official page on pricing.
Snip:
Storage:
The first 10 GB of total storage is free, and is $0.005 per GB per month for additional storage beyond 10 GB.
Download:
The first 1 GB of downloads per day are free. The price to download stored data costs $0.01 per GB beyond the free daily 1 GB.
Transactions:
A 'transaction' refers to any time an api call is made with your account, such as listing the contents of your bucket or downloading a file. B2 will charge for some transactions however, for the average user, these charges are largely negligible unless you are making an extremely high volume of api calls.
Class A transactions are free.
The first 2,500 Class B transactions are free each day. Additional Class B API calls are charged at $0.004 per 10,000 calls.
The first 2,500 Class C transactions are free each day. Additional Class C API calls are charged is $0.004 per 1,000 calls.https://downdetector.com is a great site for viewing is a vendor is down or has issues, lets compare, Backblaze with Amazon, Azure and Google Cloud
Backblaze have has far fewer downtime issues during the great outage of May/June 2019 compared to its competitors. What I like is Backblaze tell us how they do it and tell us what hard drives are good or bad.
Login/SignUp to backblaze at https://secure.backblaze.com/user_signin.htm
You can have a sneak peek at the personal Backblaze backup signup steps in this guide (why not take up a free 1 month back free by clicking this link when you create an account).
From your account page click billing and add your account details
Read the B2 Strater Guide: https://www.backblaze.com/b2/docs/
Also review your review the following under ‘My Settings‘
In the Backblaze portal, under ‘Caps and Alerts‘ you should set..
Click ‘Edit Caps’ and set daily limits on all categories
I set alert limits of 10c (I can increase this later)
Daily 10c alerts should see is costs creep above $3 a month.
Go to https://secure.backblaze.com/b2_buckets.htm and click ‘Create Bucket’
You can create upto 100 buckets per account.
I created a descriptive bucket name that reminded me what server, drive and folder this was used to back up.
"#######-drive-backup-folder" (####### was the name of my computer)
I created a number of buckets (all private).
The Backblaze portal lists all of my new storage buckets.
You can click the ‘Lifecycle Settings‘ button and set the desired buckets file retention policy (overwrite or keep all versions of a file etc).
Read the official Lifestyle settings faq here.
The default is to keep all versions of a file. Choose a setting that is required for each bucket(for me ‘Keep only the latest version of the file’ is best and lowers potential cost blowouts. Having all versions of a file may be handy for a Documents folder.
You could opt to keep all versions of a file (forever) but that would be expensive.
You tell Backblaze to delete older versions of a file after xx days (this is a good option)
You can also setup custom lifecycle rules to hide a file with a certain prefix after xx days then delete after xx days.
You will need to create a Master Application Key that will grant you full access (Permissions: listKeys, writeKeys, deleteKeys, listBuckets, writeBuckets, deleteBuckets, listFiles, readFiles, shareFiles, writeFiles, deleteFiles ) to each bucket.
Snip from: https://secure.backblaze.com/app_keys.htm
Application keys are used as a pair: Key ID and Application Key. This allows B2 to communicate securely with different devices or apps. Once you generate your Master Application Key, this key has full capabilities. Create your own Application Keys to limit features like read/write. Learn more.Do write down this “KeyID” and “applicationKey”
Don’t lose the Master Application Key.
Advertisement:
A master key is like an administrator account, its great for complete access but you should create a separate key per bucket for better security.
Back under the App Keys you can click “Add a New Application Key” for each bucket.
Read the official guide on Application Keys here
Do Set..
Optional..
"The file name prefix of files the download authorization token will allow b2_download_file_by_name to access. For example, if you have a private bucket named "photos" and generate a download authorization token for the fileNamePrefix "pets/" you will be able to use the download authorization token to access: https://f345.backblazeb2.com/file/photos/pets/kitten.jpg but not: https://f345.backblazeb2.com/file/photos/vacation.jpg.""The number of seconds before the authorization token will expire. The minimum value is 1 second. The maximum value is 604800 which is one week in seconds."I now has 1x Master Application key and 4x Application Keys for my 4x Buckets
TIP: Make sure you save the keys, if you did not grab the keys delete the keys and buckets (if there is not data and try again). If you don;t have the keys you are locked out of the buckets
Go to https://www.duplicati.com/ and familiarise yourself with the site
Go to https://www.duplicati.com/download and download the latest Windows client. At the time of writing Duplicati 2.0.4.5 beta was the latest version. Thew instal was 1MB (the download serve was not that fast)
Warning: Beta software may have bugs, feel free to check out the Github page for Issues etc.
Open the install and Click ‘Next‘
Agree to the ‘Licence Agreement‘ by clicking ‘Next‘
Select Duplicati components to install and click ‘Next‘
Tick ‘Launch Duplicati‘ an click ‘Finish‘
Upon first open you will be prompted to set a password (do this even if you are the only user of thew computer)
Set a strong and unique password
Click ‘Ok‘
When the page reloads it will report it is not logged in.
Enter your password
The main screen of Duplicati reports that there are no scheduled set.
OK, lets create a backup schedule.
If your screen is small you will need to click the menu at the top right and click ‘Add Backup’, if your screen is large there will be an ‘Add Backup’ menu on the left hand side of the screen.
Select ‘Configure a new backup‘ and click ‘Next‘
You will be able to name and describe the backup schedule name and be able to set up an encryption password.
Then click ‘Next‘
Under ‘Storage Type‘ choose ‘B2 Cloud Storage‘
When you choosing ‘B2 Cloud Storage‘ you will be directed to the screen below where you stecify..
Click ‘Next‘
Advertisement:
I entered my credentials
Click ‘Test Connection‘, if you entered everything OK you should see ‘Connection Worked‘
Click ‘Next‘
Now you can select the folders to backup on your local computer
Click ‘Next‘
Now you can choose a schedule to backup
Click ‘Next‘
Now you can set the remote backup size and backup retention mode.
Read the official guide on setting the volume size here
Choose a desirable backup retention mode. Duplicati only offers full backups (not incremental or differential backups). This will be an issue if your backups are large.
Each storage option has pro’s and cons with potential cost implications. I would not set 300GB to “Keep all backups” and backup daily. That would be expensive.
Click ‘Save‘
TIP: Don”t forget to save your encryption key if you set one
Now you will see the backup schedule on your Duplicati screen.
You can run, edit or delete the backup status here.
I clicked ‘Run‘ and the backup was under way.
A progress bar updates as the backup happens.
A slight delay happens at the end of the backup.
I logged into the Backblaze portal to see how many files and usage credits were used.
Note it seems?
When the files are backed up a verification processes happens.
Now the backup has completed.
I waited 2 minutes and finally Backblaze updated it’s status, Duplicati used 3 files and 87MB
I logged into the view the Duplicati files that were uploaded and I could see the three files were there and they were encrypted.
Awesome
Advertisement:
Having a backup is useless you can restore the files (especially when they are encrypted).
Lets restore a few files and see if it works.
I opened Duplicati and clicked ‘Restore‘
I clicked the bottom option that listed my bucket name/backup name and clicked ‘Next‘
Before I ran the restore I deleted some location files that were just backed up.
Duplicati now asked me to choose files to restore.
I chose the original location to restore to and clicked ‘Restore‘
Duplicati took a few minutes to restore the files.
The restore was a success.
Nice
Beware, make sure you calculate the costs to upload and download files.
Do set alarms on cap’s limits and review them often.
I quickly set a second backup schedule of a different folder to a different bucket.
This backup was 300GB so I will only backup once a month
The backup is listed on the front screen of Duplicati.
Do create multiple buckets and set multiple backup schedules to break up the backups into smaller chunks.
Lets setup B2 Cloud Storage backup on Linux.
I visited the Duplicati forums here. Duplicati informed me that the Linux client needs mono installed
Duplicati depends on other software. For Windows, Microsoft .NET Framework 4.5 or higher needs to be installed. Linux and Mac OS X require Mono to be installed.Mono is a Cross Platform open source .NET that runs on Linux (read here)
On Ubuntu 1.08 I installed mono with these commands
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update
sudo apt install mono-devel gtk-sharp2
sudo apt-get install libmono-2.0-1On Debian 9.9 I installed Mono with these commands (from here)
sudo apt install apt-transport-https dirmngr gnupg ca-certificates
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update
sudo apt install mono-devel gtk-sharp2
sudo apt-get install libmono-2.0-1I rebooted the serves and the CPU and memory went crazy after I installed Mono. My Nixstats Service reports were going off telling me my servers CPU was busy.
If you don’s have reports on Linux server CPU/MEM/Disk reports coming to you via Email and Telegram then read this.
I felt sorry for my web and database server server(s) so I rebooted them, wow, My website came back up in about 10 seconds thanks to UpClouds fast Max IOPS disks.
I visited the download Duplicati page here and copied the Ubuntu and Debian download URL.
At the time of writing it was..
https://updates.duplicati.com/beta/duplicati_2.0.4.5-1_all.debI downloaded the deb package
cd /utils/duplicati
wget https://updates.duplicati.com/beta/duplicati_2.0.4.5-1_all.debI then ran this command to install the package
sudo dpkg -i duplicati_2.0.4.5-1_all.debI received this error even thpogh libmono-2.0-1 was installed
dpkg: dependency problems prevent configuration of duplicati:
duplicati depends on libmono-2.0-1; however:
Package libmono-2.0-1 is not installed.This forum post helped me, I ran
sudo apt install -fOutput
sudo apt install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4
libindicator7 libmono-2.0-1
Suggested packages:
indicator-application
The following NEW packages will be installed:
libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4
libindicator7 libmono-2.0-1
0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.
1 not fully installed or removed.
Need to get 143 kB of archives.
After this operation, 541 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Advertisement:
And I ran the following
sudo apt dist-upgrade --fix-missingI now ran the following to install the duplicati package.
sudo dpkg -i duplicati_2.0.4.5-1_all.debIt worked on Ubuntu 18.08 bt not Debian 9.9
Debian produced the following error
sudo dpkg -i duplicati_2.0.4.5-1_all.deb (Reading database ... 70082 files and directories currently installed.)
Preparing to unpack duplicati_2.0.4.5-1_all.deb ...
Unpacking duplicati (2.0.4.5-1) over (2.0.4.5-1) ...
dpkg: dependency problems prevent configuration of duplicati:
duplicati depends on libappindicator0.1-cil | libappindicator3-0.1-cil; however:
Package libappindicator0.1-cil is not installed.
Package libappindicator3-0.1-cil is not installed.
dpkg: error processing package duplicati (--install):
dependency problems - leaving unconfigured
Processing triggers for mime-support (3.60) ...
Errors were encountered while processing:
duplicatiOn Debian I installed the stretch version of the package
sudo apt-get install libappindicator1Then in Debian I fixed broken packages
sudo apt --fix-broken installThis I was able to install Duplicati on Debian
sudo dpkg -i duplicati_2.0.4.5-1_all.debI can now Run Duplicati CLI (it works)
duplicati-cli
See duplicati.commandline.exe help <topic> for more information.
General: example, changelog
Commands: backup, find, restore, delete, compact, test, compare, purge, vacuum
Repair: repair, affected, list-broken-files, purge-broken-files
Debug: debug, logging, create-report, test-filters, system-info, send-mail
Targets: aftp, amzcd, azure, b2, box, cloudfiles, dropbox, ftp, file,
googledrive, gcs, hubic, jottacloud, mega, msgroup, onedrive, onedrivev2,
sharepoint, openstack, rclone, s3, ssh, od4b, mssp, sia, tahoe, webdav
Modules: aes, gpg, zip, 7z, console-password-input, mssql-options,
hyperv-options, http-options, sendhttp, sendmail, runscript, sendxmpp,
check-mono-ssl
Formats: date, time, size, encryption, compression
Advanced: mail, advanced, returncodes, filter, filter-groups, <option>
http://www.duplicati.com/ Version: - 2.0.4.5_beta_2018-11-28
TIP: Don’t run “duplicati” that’s the desktop verison.
I read this page and this page to get an understanding of how to back up from the command line. I was not sure if I needed to run a service or run the CLI from a bach script so I asked on the forums here.
Within minutes I had this reply
kenkendk
1h
Hi @feartec, welcome to the forum!
The CLI and Web-UI are not the same, and operate independently.
If you use the CLI, you need to use cron or something similar to run the backups periodically. You can use duplicati-cli help to get started. You do not need the server running for this.
For the web-UI we do not currently have any tools for manipulating the backups other than the main WebUI itself. You need to somehow forward/tunnel access to the server, and then access the WebUI from a local browser.I created this folder structure
/test/
/test/testfile.txt < hello
/test/subfolder/
/test/subfolder/file.txt < worldI ran this command to back up the folder structure above to backblaze (official guide). See advanced switches.
/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=**********TIP: read more about “dblock” size here
Output
Checking remote backup ...
Listing remote folder ...
Scanning local files ...
4 files need to be examined (8 bytes)
Checking remote backup ...
Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
Downloading file (1021 bytes) ...
Downloading file (1.06 KB) ...
Downloading file (2.00 KB) ...
0 files need to be examined (0 bytes)
Duration of backup: 00:00:06
Remote files: 3
Remote size: 4.05 KB
Total remote quota: 0 bytes
Available remote quota: 0 bytes
Files added: 0
Files deleted: 0
Files changed: 0
Data uploaded: 0 bytes
Data downloaded: 4.05 KB
Backup completed successfully!
I logged into Backblaze to check the backups
I created a second test file and added 2 test files
mkdir /test2
cd /test2
wget http://ipv4.download.thinkbroadband.com/10MB.zip
wget http://ipv4.download.thinkbroadband.com/5MB.zip
ls -al
total 15368
...
-rw-r--r-- 1 user user 10485760 Jun 3 2008 10MB.zip
-rw-r--r-- 1 user user 5242880 Jun 3 2008 5MB.zip
I ran the backup with the second folder in the command line
/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=********** --retention-policy="1W:1D,4W:1W,12M:1M"
Output
Backup started at 06/07/2019 23:05:58
Checking remote backup ...
Listing remote folder ...
Scanning local files ...
7 files need to be examined (15.00 MB)
Uploading file (15.03 MB) ...
Uploading file (11.43 KB) ...
Uploading file (1.29 KB) ...
Checking remote backup ...
Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
Downloading file (1.29 KB) ...
2 files need to be examined (8 bytes)
Downloading file (11.43 KB) ...
Downloading file (15.03 MB) ...
0 files need to be examined (0 bytes)
Duration of backup: 00:00:10
Remote files: 6
Remote size: 15.05 MB
Total remote quota: 0 bytes
Available remote quota: 0 bytes
Files added: 2
Files deleted: 0
Files changed: 0
Data uploaded: 15.05 MB
Data downloaded: 15.05 MB
Success
O’k lets delete the local “/test2/” folder then restore it
rm -R /test2Now lets restore it (official guide)
/usr/bin/duplicati-cli restore "b2://bucket-name-goes-here/test" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --passphrase=************* --overwrite=true --restore-path="/test2"Output
Restore started at 06/07/2019 23:21:51
Checking remote backup ...
Listing remote folder ...
Checking existing target files ...
2 files need to be restored (15.00 MB)
Scanning local files for needed data ...
Downloading file (15.03 MB) ...
0 files need to be restored (0 bytes)
Verifying restored files ...
Restored 0 (0 bytes) files to /test2
Duration of restore: 00:00:05
Did it work
cd /test2
ls -al
total 15368
...
-rw-r--r-- 1 use user 10485760 Jun 3 2008 10MB.zip
-rw-r--r-- 1 user user 5242880 Jun 3 2008 5MB.zipYes it worked
I use Google GSuite for email and have an app password setup just for duplicati. Here is the official guide for sending an email after a backup operation
I downloaded a 100MB file
cd /test2
wget http://ipv4.download.thinkbroadband.com/100MB.zipI ran this command to backup an emailthe progress
/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=***************** [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"Output
Backup started at 06/08/2019 00:02:28
Checking remote backup …
Listing remote folder …
Scanning local files …
8 files need to be examined (115.00 MB)
7 files need to be examined (15.00 MB)
Uploading file (49.99 MB) …
Uploading file (18.01 KB) …
Uploading file (49.99 MB) …
Uploading file (17.97 KB) …
Uploading file (233.33 KB) …
Uploading file (33.17 KB) …
Uploading file (1.42 KB) …
Compacting remote backup …
Checking remote backup …
Listing remote folder …
Verifying remote backup …
Remote backup verification completed
Downloading file (1.42 KB) …
4 files need to be examined (15.00 MB)
Downloading file (18.01 KB) …
Downloading file (49.99 MB) …
0 files need to be examined (0 bytes)
Duration of backup: 00:00:15
Remote files: 13
Remote size: 115.33 MB
Total remote quota: 0 bytes
Available remote quota: 0 bytes
Files added: 1
Files deleted: 0
Files changed: 0
Data uploaded: 100.28 MB
Data downloaded: 50.01 MB
Backup completed successfully!
Read up here on backup retention policies here
TIP: Use the desktop Duplicati software to generate a backup then use the export to command line option to get the command line parameters to use in the command line version.
After you click Commandline you can view each separate command liene option and parameter.
This was handy for finding this backup retention parameter
--retention-policy="1W:1D,4W:1W,12M:1M“This will give me “one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup. “
Login to your linux machines and edit a bash file (e.g in “/scripts/backupnow.sh” folder).
Make the script executable
chmod +x /scripts/backupnow.shAdd the following
#!/bin/bash
/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=123 [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"Now you can add the script to your crontab by running the following.
crontab -eMake sure you have the following in your crontab
SHELL=/bin/bashAdd a cron entry to run the bash script
#Backup Database
0 22 * * * /bin/bash /scripts/backupnow.shUse https://crontab.guru/ to check the time you want to run thew script
Version
v1.1 Fixed typo (”downalod” thanks to @NicolasWolf for reporting.)
v1.0 Initial Post (added retention polity and Linux crontabs
v0.95 Backup, Restore and Send Email
v0.91 Added Duplicati service info
v0.9 Working Copy
Backblaze ( https://www.backblaze.com/ ) is an awesome company who not only create a mega reliable backup infrastructure but they also tell us how they do it and tell us what hard drives are good or bad.
Read the full article here: https://fearby.com/article/backing-up-your-computer-automatically-with-backblaze-software-no-data-limit/
Backblaze ( https://www.backblaze.com/ ) is an awesome company who not only create a mega reliable backup infrastructure but they also tell us how they do it and tell us what hard drives are good or bad.
Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj
They are so confident and experienced that they offer unlimited backups for $6.60 a month here in Australia. They state they have 750 Petabytes stored on their infrastructure & have restored over 40 billion files to customers.
Check our my other related posts
Main features of Backblaze.
Advertisement:
Go to backblaze and create an account.
Login to backblaze at https://secure.backblaze.com/user_signin.htm
Go to: https://secure.backblaze.com/buy_plan.htm to choose a backup and payment plan. Use my link and get your first month free.
Australian Prices below (add +10% GST).
Choose a payment plan
Advertisement:
Enter payment details
Click “Buy Backblaze”
Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj
Done, now I can download the client.
When the download is down you can start the install.
You will need to login to the install to streamline the setup.
After you login the installation will begin.
Backblaze now reports that the install has complete. Backblaze reports that I have 379GB to backup.
Advertisement:
Pressing OK shows Backblaze is already uploading my files.
Also, I have 15 days of free trial before being billed.
Clicking on the backblaze system tray icon reveals a few options
Backblaze preferences can be opened by clicking Settings from the main screen.
I..
There is a 50 char limit on encryption keys
Warning about not forgetting the encryption key
All other preferences look ok.
Advertisement:
That’s it the backup is uploading automatically and there is nothing else I need to do.
If you login to the backblaze portal ( here ) you can view your files (you will need to enter the private encryption keys if you opted to encrypt your files).
Your initial backup may have a while to upload depending on your backup size and internet connection speed.
I was able to upload 400GB in 3 days on my Internet Plan. Smaller files are uploaded first then larger ones. I have 2 files left to upload totalling 7GB.
Read the official guide on speeding up uploads here.
One way to make the initial backups faster is to see what files are queues to upload in the Backblaze preferences and then excluding unwanted files and folders.
I can see my Internet Explorer cache and Google Chrome Temporary Internet File(s) folders are queued to be back up.
Time to exclude these folders in the Backblaze Settings under exclusions
C:\Users\Simon Fearby\AppData\Local\Microsoft\Windows\INetCache\IE
C:\Users\Simon Fearby\AppData\Local\Google\Chrome\User Data\Profile 1\Code Cache\
C:\ProgramData\MySQL\MySQL Server 8.0\Data\#innodb_temp\
C:\ProgramData\USOShared\Logs\
C:\Users\Simon Fearby\AppData\Roaming\Adobe\SLData\SLCache\Time to exclude these folders.
That knocked off a few thousand files to backup 🙂
If you restore files you have a few options
Choosing to download files you are given a treeview to restore files from.
Advertisement:
I selected a test file to restore (restores are not instant) and was informed I would be emailed when the files are ready to download. Thats cool
I waited 3 minutes and received an email that my files were ready to download.
I hope this helps someone.
Check out my guide on How to back up an iPhone photos and videos blog post here.
Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj
View Official Backblaze guides here.
Version
v1.3 Initial Backup section
v1.1 Added free month link
v1.0 Initial Draft
This guide is for and anyone who needs to back up their iPhone (complete device along with separate backups of photos and videos).
Read the full article here: https://fearby.com/article/how-to-back-up-an-iphone-including-photos-and-videos-multiple-ways//
This guide is for and anyone who needs to back up their iPhone (complete device along with separate backups of photos and videos).
This is not a paid promo, I don’t get a kickback for mentioning the awesome app below.
Check our my other related posts
At the time of writing iTunes v12.9.5.7 was the latest version of iTunes.
iTunes is Apple official software for getting files too and from an iPhone, iPod or iPad. If rumours are right Apple will kill off iTunes software at the end of 2019. If Apple kills off the iTunes software I will update this guide in the future.
Disclaimer: I provide no warranty or support for this app, this is what happened to work for me.
Why write a guide on Backing up a mobile phone?
Golden rules of backing up.
Apple prefers you just pay them to extend your free 5GB iCloud storage and not worry about backups and not to worry yourself with manual steps.
Advertisement:
My wife’s iPhone is always popping up messages saying that her iCloud storage is nearly full and she should upgrade the free space limit.
I know she has 100GB+ images and movies on her iPhone but iCloud has only backed up 4GB of photos and videos. Do not trust any backup statuses unless you can verify all files from a restore.
For the record I use an Android phone (Android backup guide coming soon).
iCloud is a good idea for automatic backup but I prefer to see my photos and backup them up myself. Also, Apple iCloud is not free from troubles. If you want Apple to handle backups then I do suggest you upgrade your iCloud storage from the free 5GB to a more sensible 200GB or more.
Here in Australia Apple charge the following for extra storage.
A rough estimate: 50GB is enough to store about 12,000 files (9,000 photos + 3,000 videos) *
You can upgrade your paid Apple iCloud backup limit from your iCloud Storage Settings from your iPhone’s settings screen.
IMHO: Apple does make it clear what you are getting before you purchase (but they do not push it enough and people assume their data is safe).
TIP: Please review your existing iPhone data usage and iCloud usage before it’s too late
I like you, I am tight too but I do pay Google $2.49 a month for 100GB backup storage on my Android phone. Google Drive (Google storage is a bit cheaper than Apple).
It is nice knowing your phone is auto backed up.
On your Windows 10 computer click the start button. If you don’t have a Microsoft Account go here and create one. Click the start button and type “itunes” (you don’t need to type into a box, just start typing when you click the start button)’.
Then click “iTunes, install app”
Advertisement:
If you are logged into your Windows 10 store with your Microsoft ID click “Get” (if not you will need to login to the store).
After you click ‘Get’, iTunes will start downloading.
When iTunes is downloaded Windows will install it. When it’s installed click ‘Launch’
You will need to agree to Apple’s terms of service. Click ‘Agree’
When iTunes opens click “Agree”
Now login to iTunes with your Apple ID (if you don’t have one create one here)’.
Click the ‘Account’ then ‘Sign In’ menu.
Login to iTunes with your Apple ID
Optional: If you have Two Factor Authorisation (2FA) of Apple ID’s turned on (you should) you will need to enter a 6 digit code.
Apple Two Factor Authorisation (2FA) will send one of your other devices a login code that you will need to use to login (confirm you own the account)
TIP: Check out https://twofactorauth.org/ to see what other sites use Two Factor Authorisation (2FA). I use hardware Yubico YubiKeys to protect logins to WordPress, Linux and websites.
Now back to the article, iTunes should be ready to allow us to backup our iPhone.
Before I continue I will click ‘Edit‘ then ‘Preferences‘ menu to view where iTunes will download media too (different ot backup data), not important but I just want ot see if it is not pointed ot my smaller C drive before I backup my phone
I changed the location to a larger S:\Drive’
I was greeted with a message asking me to confirm that I wanted to sync the iPhone that I just plugged in called ‘EllieRose’, I clicked ‘Continue’
On the iPhone I also clicked ‘Trust’ to allow iTunes to talk to it.
I was then prompted to download a software update to the iPhone, It appears this phone is not running the latest software
I was prompted to sync the iPhone to the computer
A backup of the phone was underway.
If the screen above does not appear click the icon below to view the backup and restore iPhone menu.
The iPhone was backing up to this folder on my computer:
C:\Users\Simon Fearby\Apple\MobileSync\BackupI was not prompted for a location to back up so and I will move this backup folder after the backup completes (so my C: drive does not fill up).
My drives.
The iTunes iPhone backup is now complete (it took about 2 hours).
Now that the iPhone was backed up iTunes the Operating System update started to run.
Update verified.
During the update the iPhone was unavailable.
Done, the iPhone had updated it’s operating system in about 30 minutes.
Now lets see how much space I have available on my C Drive.
It looks like the iPhone backup iTunes made was 60GB in size.
Advertisement:
Using Windows Explorer I moved the backup from..
C:\Users\Simon Fearby\Apple\MobileSync\Backupto..
S:\Backup\AlisonsiPhone\iTunes\31May2019(right click drag an drop action in explorer)
It took me about 20 minutes to move the 60GB folder.
TIP: You can (should) also copy this backup folder to..
Now this backup is available for me to restore in the future if I need it.
In the case of a restore, I just need to move the backup into the iTunes expected location.
Copy from..
S:\Backup\AlisonsiPhone\iTunes\31May2019..to..
C:\Users\Simon Fearby\Apple\MobileSync\BackupThe Catch
The catch with back-ups made with iTunes is they are useless if you wanted to restore individual files (say to find a photograph or video). iTunes backups are usually tens of thousands of files with random folders and filenames
If you want to JUST backup photos and videos read on.
This part of the guide needs a paid version of the iOS App Photo Transfer App. In Australia, the app is free but to turn it into a full version needed to restore everything (via in-app purchase) it cost $10.99.
This app is well worth $10.99 AUD (it may be cheaper in other countries) to have it push photos an videos from your iPhone to a free companion Windows program (a Mac versions exist too).
Buy the iOS version and install it on your iPhone.
Then download the free Windows version: http://phototransferapp.com/win/
Extract the files from the zip file.
When I run PhotoTransferApp.exe I get an error saying I need to install Adobe AIR run times.
Go to https://get.adobe.com/air/ and download and install the Air run time.
Install the Adobe Air run time.
I installed Adobe AIR and again reopened the PhotoTansferApp.exe and was prompted to allow access to the Windows Firewall.
I was on a home network so I enabled firewall access.
The Photo Transfer App prompted me to ‘Discover Devices‘ and to make sure the Transfer app is running on the iPhone.
Before I clicked ‘Discover Devices” on the Photo Transfer App on Windows I opened the Transfer app (mentioned above) on the iPhone.
On the iPhone I clicked ‘Send‘
The transfer app will ask for access to your photos, you will need to press OK.
Advertisement:
On the iPhone again I clicked ”Windows‘
The iOS Transfer app now said I should run the ‘Photo Transfer App‘ on Windows.
TIP: You can transfer over WiFi (if your iPhone and Windows device is on the same WiFi Network) or you can transfer over a USB cable.
Before I started the ‘Detect Device‘ or ‘transfer‘, I set the Backup path location to my S: drive by clicking settings in the bottom right and choosing a folder (as my C Drive is a bit small).
I noticed the port of 57777, I temporarily disabled the whole Windows Firewall just in case it prevents the photo backup.
In hindsight, it was not a good idea to disable the whole firewall, but because I was at home on a safe network I felt safe to do so. Next time I will not disable the firewall and see if this still works.
If you are on an internet cafe, school or university network do not disable your firewall.
I clicked ”Start” then typed ‘firewall‘ and clicked ‘Windows Defender Firewall‘
I turned off my firewall.
I clicked ‘Detect Devices‘ and the iPhone ‘EllieRose‘ appeared on the left. I doubled clicked on the iPhone name and was prompted with an Authorization required message.
I looked at the ‘Transfer‘ app on iOS and clicked ‘Yes, always‘ to allow access.
After 20 seconds I can see photos on my iPhone on Windows.
I selected all camera albums to backup and clicked backup. I prompt to upgrade to the full version will appear, you will need to buy the upgrade.
After the full version if purchased the backup will be allowed.
You can upgrade from the free to paid full version from the main screen of the Transfer app.
The upgrade In App Purchase is $10.99 AUD
When you purchase the In App Purchase you can run the backup again.
Advertisement:
Now I could see photos being copied from the iPhone to my defined backup folder.
I went to bed as I knew there were about 100GB of files on the iPhone and this was going to take a while.
In the morning the backup was done.
26,000 files were backed up (over 100GB).
I now had a full iPhone backup made by iTunes and a copy of all photos and videos.
I turned on the Firewall again.
I now had 160GB of backed up photos, videos and phone backup.
The steps to do this two ways will be added soon.
I will add a section on how you can back up the iTunes and manually synced photo and video backups to the cloud automatically and a more complex but cheaper was for 0.005c per GB.
Watch this post.
Article coming soon.
Read the official iTunes/iCloud backup guide from Apple here: https://support.apple.com/en-au/HT203977
Backing up your computer automatically with BackBlaze software (no data limit)
Versions
v1.3 Added back blaze article link
v1.2 Added more images (from a phone that does not have the iOS app already)
v1.1 Added an Android heading.
v1.0 Initial Post
This is a quick guide demonstrating how you can backup and restore a MySQL database on Windows and Linux using Adminer.
You may need to know how to backup a restore a database for a number of reasons..
e.g
Advertisement:
Having a backup of the VM is good but having a backup of the database too is better. I use UpCloud for hosting my VM’s and setting backups is easy. But I cannot download those backups.
“If anything can go wrong, it will”
The most important reason for taking a backup and knowing how to restore it is for disaster recovery reasons.
Adminer is a free PHP based IDE for MySQL and other databases. Simply install Adminer and save the file on your local computer or remote web server directory.
FYI: The Adminer author Jakub Vrana has a patron page, I am a patron of this awesome software.
Snip from Adminers website. “Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch andMongoDB.”
TIP: The file would be publicly accessible to anyone so don’t save it to a common area, obfuscate the file, protect it of delete the file when you are done using it.
Advertisement:
Once Adminer is installed load it in a web browser, login with your MySQL credentials. Once you login you will see all databases and an Import and Export menu.
tbtest is a simple database with one table and 4 fields (ID, Key, Value and Modified)
.Click Export to open the export screen.
Click Export, a SQL file will be generated (this is the export of the database).
Here is a save of the file:
https://fearby.com/wp-content/uploads/export.txt
Its that simple.
If I add a binary blob file to the table and upload a PNG file lets see how the export looks.
Advertisement:
Let export the database again in Adminer and check out the output. I used Sublime Text editor to view the export file.
OK lets delete the tbtest database and then restore it with Adminer. I used Adminer to delete (DROP) the database.
Database “dbtest” deleted.
Now lets create a blank database to restore to (same name).
Database created.
Advertisement:
Now lets import the database backup using Adminer.
Click Import, select the backup file and un-tick Stop on errors.
TIP: The 2MB next the the choose file button is defined by your web server and PHP configuration. If you are trying to import a larger database (e.g 80MB) first increase the limits in your web server and PHP (via php.ini).
The Import (restore should take seconds)
The database was imported from a backup, all tables and records imported just fine.
On Ubuntu use this guide to backup from the command line. If you use the Oracle MySQL Workbench read this.
I hope this helps someone.
Advertisement:
This is a simple guide that demonstrates how you can log in to a host that offers the CPanel tools to backup all of your website files (and databases). Backing up your website should be done often and especially before you migrate to any another website host. I used to change hosts every few years (they don’t own your site, you do).
Advertisement:
I have a number of guides on moving away from CPanel, setting up VM’s on UpCloud, AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line copying files to a server via command line editing remote files locally etc but how do you manage a website with CPanel?
You can normal login to CPanel tools on a shared host by loading www.yourdomainnam.com/cpanel (failing that login to your domain hosts web GUI and find your CPanel interface there).
Step 1: Login to your Host
Login to your web host
Step 2: Find your CPanel Interface
Hosts are a bit different but in this case, I just click my domain to find the CPanel link.
I found it, I clicked the CPanel login.
Step 3: CPanel Applications
CPanel does offer good tools to manage your websites like web-based File Manager and Database tool called phpMyAdmin.
Aside: CPanel/Hosts Downsides
The thing I don’t like about hosts that offer CPanel is they usually limit delivery of your website to extract more money. Nothing worse than receiving Resource Limit Is Reached errors.
Also shared hosts usually lag way behind in newer software versions like PHP and MySQL (this is a security concern).
TIP: You can scan your site for vulnerabilities using Qualsys Freescan, Zap or Kali Linux.
Here is a security scan of a shared host (with CPanel) that I was using in 1999. Note the high vulnerabilities and old version of Linux.
Also, a shared host will often overcharge you (e.g $150 a year) for a poorly configured SSL certificate.
This was an SSL cert I paid $150 a year for (evaluated with SSL Labs SSL Test) on a shared host with CPanel.
Aside: Self Managed Upsides
After I moved my domain to a self-managed virtual machine I migrated WordPress, set up a free SSL certificate, sped up my site with a CDN, setup Cloudflare, setup better TLS security etc
When you manage your own server you can install a free SSL certificate in under 1 minute.
Below is my SSL certificate. A strong SSL certificate will increase search engine traffic
Aside: Compare Shared host speed v Self Managed
FYI: https://gtmetrix.com/ is a great site for measuring the speed of a website (shared of self-managed). I found great speed improvements after moving away from a host offering CPanel, tweaking the server and setting up cloudflare. A self-managed server will allow you to tweak anything you want.
GTMetrix results:
I like how self-managed servers allow you to scale the server’s resources yourself, move servers or add storage etc.
Aside: SSL Certificate
If you have an SSL cert you should test it often as vulnerabilities pop up from time to time.
FYI: All sites will soon require an SSL certificate to be sent traffic from search engines (no SSL = lower traffic).
SSL Test my site: https://dev.ssllabs.com/ssltest/analyze.html?d=fearby.com&s=104.27.154.69
Now enough with the self-managed serve asides and back to how to backup your website with CPanel tools.
Step 4: Backup your web files in CPanel
Use the File Explorer app in CPanel
Highlight all files that you want to backup (highlight everything but not past backup files).
View the files to compress summary
Click Compress Files(s) and view the backup progress
You can now download the backup zip file in your browser (click the file and click Download).
Download Progress.
Step 5: Backup your database in CPanel
Now we need to backup any MySQL database(s) that may be used by WordPress
Open the phpMyAdmin app in CPanel.
FYI: Alternatively, you can use a free tool called Adminer to backup and restore our database.
Click your WordPress database (on the left). You can identify your current WordPress database by opening the wp-config.php file.
The first step is to perform an online cold backup of the WordPress database.
Now you have an online cold spare that you can use just in case the original database corrupts itself. You can rename the database or configure WordPress to point to this new database if need be.
Now let’s download a copy of the database (Repeat for multiple databases).
You should now have a backup of your website in a zip file and an export of your database in a .sql text file, SQL files can be re-imported to databases later.
TIP: Backup often.
I hope this guide helps someone.
Ask a question or recommend an article
Revision History
v1.0 Initial post
Here is how I backup a number of Ubuntu servers with crontab entries, bash scripts and rsync and send backup email.
Advertisement:
Read more on useful terminal commands here for as low as $2.5 a month. Read on setting up a Digital Ocean Ubuntu server here for as low as $5 a month here ($10 free credit). Read more on setting up an AWS Ubuntu server here.
I have 6 numbered scripts in my scripts folder that handle backups, I call these scripts at set times via the crontab list.
fyi: Paths below have been changed for the purpose of this post (security).
https://crontab.guru/ is great for specifying times to run jobs on each server (I backup one server at 1 AM,, another at 2 AM etc (never at the same time))
Bring up your crontab list
Check out the Crontab schedule generator here.
Below is the contents of my /scripts/0.backupfiles.sh (sensitive information removed).
I use this script to backup folders and configuration data
Below is the contents of my /scripts-folder/1.backupdbs.sh (sensitive information removed).
I use this script to dump my MySQL database.
Below is the contents of my /scripts-folder/2.shrinkmysql.sh (sensitive information removed).
I use this script to tar my SQL dumps as these files can be quite big
Below is the contents of my /scripts-folder/3.addtobackuplog.sh (sensitive information removed).
This script is handy for dumping extra information.
Below is the contents of my /scripts-folder/4.syncfiles.sh (sensitive information removed).
This script is the workhorse routine that rsyncs files to the source to the backup server (a dedicated Vulr server with an A Name record attaching the server to my domain).
I installed sshpass to pass in the ssh user password (after ssh is connected (authorized_keys set), I tried to setup a rsync daemon but had no luck). I ensured appropriate ports were opened on the source (OUT 22, 873) and backup server (IN 22 873).
ufw firewall has great rules for allowing certain IP’s to talk on ports.
Set Outbound firewall rules (to certain IP’s)
sudo ufw allow from 123.123.123.123 to any port 22
Change 123.123.123.123 to your backup server.
Set Inbound firewall rules (to certain IP’s)
sudo ufw allow out from 123.123.123.123 to any port 22
Change 123.123.123.123 to your sending server.
You can and should setup rate limits on IP’s hitting certain ports.
Install Fail2Ban to automatically ban certain users. Fail2Ban reads log file that contains password failure report
and bans the corresponding IP addresses using firewall rules. Read more on securing Ubuntu in the cloud here.
Below is the contents of my /scripts-folder/5.sendbackupemail.sh (sensitive information removed).
This script sends an email and attaches a zip file of all log files generated through the backup process.
Read my guide on setting up sendmail here.
Security Considerations
You should never store passwords in scripts that talk to SSH connections, create MySQL dumps or when talking to email servers, I will update this guide when I solving all of these cases. Also, create the least access required for user accounts where possible.
Target Server Configuration
Alos you can see in /scripts-folder/4.syncfiles.sh that I am saving to the ‘/backup-folder/1.www.server01.com/’ folder, you can make as many folders as you want to make the most of the backup server. I would advise you not use the server for anything else like web servers and apps as this server is holding important stuff.
I have a handy script to delete all backups (handy during testing).
Ensure your backup server is just for backups and only allows traffic from known IP’s
Change the 123.x.x.x servers to your servers IP’s
Tip: Keep an eye on the backups with tools like ncdu
Installing SSH on OSX
If you want to backup to this server with OSyouou will need to install sshpass
sshpass should be installed
I have not got sshp[ass working yet error “Host key verification failed.” I had to remove the back known host from ~/.ssh/known_hosts” on OSX
But this worked on OSX
Note: Enter the servers [email protected] before the hostname or rsync will use the logged in OSX username
Don’t forget to check the backup serves disk usage often.
Output from backing up an incremental update (1x new folder)
localhost:~ local-account$ rsync -a -e 'ssh -p 22 ' --progress -P /Users/local-account/folder-to-backup [email protected]:/backup/the-computer/ [email protected]'s password: building file list ... 51354 files to consider folder-to-backup/ folder-to-backup/TestProject/ folder-to-backup/TestProject/.git/ folder-to-backup/TestProject/.git/COMMIT_EDITMSG 15 100% 0.00kB/s 0:00:00 (xfer#1, to-check=16600/51354) folder-to-backup/TestProject/.git/HEAD 23 100% 22.46kB/s 0:00:00 (xfer#2, to-check=16599/51354) folder-to-backup/TestProject/.git/config 137 100% 133.79kB/s 0:00:00 (xfer#3, to-check=16598/51354) folder-to-backup/TestProject/.git/description 73 100% 10.18kB/s 0:00:00 (xfer#4, to-check=16597/51354) folder-to-backup/TestProject/.git/index 1581 100% 220.56kB/s 0:00:00 (xfer#5, to-check=16596/51354) folder-to-backup/TestProject/.git/hooks/ folder-to-backup/TestProject/.git/hooks/README.sample 177 100% 21.61kB/s 0:00:00 (xfer#6, to-check=16594/51354) folder-to-backup/TestProject/.git/info/ folder-to-backup/TestProject/.git/info/exclude 40 100% 4.88kB/s 0:00:00 (xfer#7, to-check=16592/51354) folder-to-backup/TestProject/.git/logs/ folder-to-backup/TestProject/.git/logs/HEAD 164 100% 20.02kB/s 0:00:00 (xfer#8, to-check=16590/51354) folder-to-backup/TestProject/.git/logs/refs/ folder-to-backup/TestProject/.git/logs/refs/heads/ folder-to-backup/TestProject/.git/logs/refs/heads/master 164 100% 20.02kB/s 0:00:00 (xfer#9, to-check=16587/51354) folder-to-backup/TestProject/.git/objects/ folder-to-backup/TestProject/.git/objects/05/ folder-to-backup/TestProject/.git/objects/05/0853a802dd40cad0e15afa19516e9ad94f5801 2714 100% 294.49kB/s 0:00:00 (xfer#10, to-check=16584/51354) folder-to-backup/TestProject/.git/objects/11/ folder-to-backup/TestProject/.git/objects/11/729e81fc116908809fc17d60c8604aa43ec095 105 100% 11.39kB/s 0:00:00 (xfer#11, to-check=16582/51354) folder-to-backup/TestProject/.git/objects/23/ folder-to-backup/TestProject/.git/objects/23/768a20baaf8aa0c31b0e485612a5e245bb570d 131 100% 12.79kB/s 0:00:00 (xfer#12, to-check=16580/51354) folder-to-backup/TestProject/.git/objects/27/ folder-to-backup/TestProject/.git/objects/27/3375fc70381bd2608e05c03e00ee09c42bdc58 783 100% 76.46kB/s 0:00:00 (xfer#13, to-check=16578/51354) folder-to-backup/TestProject/.git/objects/2a/ folder-to-backup/TestProject/.git/objects/2a/507ef5ea3b1d68c2d92bb4aece950ef601543e 303 100% 26.90kB/s 0:00:00 (xfer#14, to-check=16576/51354) folder-to-backup/TestProject/.git/objects/2b/ folder-to-backup/TestProject/.git/objects/2b/f8bd93d56787a7548c7f8960a94f05c269b486 136 100% 12.07kB/s 0:00:00 (xfer#15, to-check=16574/51354) folder-to-backup/TestProject/.git/objects/2f/ folder-to-backup/TestProject/.git/objects/2f/900764e9d12d8da7e5e01ba34d2b7b2d95ffd4 209 100% 17.01kB/s 0:00:00 (xfer#16, to-check=16572/51354) folder-to-backup/TestProject/.git/objects/36/ folder-to-backup/TestProject/.git/objects/36/d2c80d8893178d7e1f2964085b273959bfdc28 201 100% 16.36kB/s 0:00:00 (xfer#17, to-check=16570/51354) folder-to-backup/TestProject/.git/objects/3d/ folder-to-backup/TestProject/.git/objects/3d/e5a02083dbe9c23731a38901dca9e913c04dd0 130 100% 10.58kB/s 0:00:00 (xfer#18, to-check=16568/51354) folder-to-backup/TestProject/.git/objects/40/ folder-to-backup/TestProject/.git/objects/40/40592d8d4d886a5c81e1369ddcde71dd3b66b5 841 100% 63.18kB/s 0:00:00 (xfer#19, to-check=16566/51354) folder-to-backup/TestProject/.git/objects/87/ folder-to-backup/TestProject/.git/objects/87/60f48ddbc9ed0863e3fdcfce5e4536d08f9b8d 86 100% 6.46kB/s 0:00:00 (xfer#20, to-check=16564/51354) folder-to-backup/TestProject/.git/objects/a9/ folder-to-backup/TestProject/.git/objects/a9/e6a23fa34a5de4cd36250dc0d797439d85f2ea 306 100% 22.99kB/s 0:00:00 (xfer#21, to-check=16562/51354) folder-to-backup/TestProject/.git/objects/b0/ folder-to-backup/TestProject/.git/objects/b0/4364089fdc64fe3b81bcd41462dd55edb7a001 57 100% 4.28kB/s 0:00:00 (xfer#22, to-check=16560/51354) folder-to-backup/TestProject/.git/objects/be/ folder-to-backup/TestProject/.git/objects/be/3b93d6d8896d69670f1a8e26d1f51f9743d07e 60 100% 4.19kB/s 0:00:00 (xfer#23, to-check=16558/51354) folder-to-backup/TestProject/.git/objects/d0/ folder-to-backup/TestProject/.git/objects/d0/524738680109d9f0ca001dad7c9bbf563e898e 523 100% 36.48kB/s 0:00:00 (xfer#24, to-check=16556/51354) folder-to-backup/TestProject/.git/objects/d5/ folder-to-backup/TestProject/.git/objects/d5/4e024fe16b73e5602934ef83e0b32a16243a5e 69 100% 4.49kB/s 0:00:00 (xfer#25, to-check=16554/51354) folder-to-backup/TestProject/.git/objects/db/ folder-to-backup/TestProject/.git/objects/db/3f0ce163c8033a175d27de6a4e96aadc115625 59 100% 3.84kB/s 0:00:00 (xfer#26, to-check=16552/51354) folder-to-backup/TestProject/.git/objects/df/ folder-to-backup/TestProject/.git/objects/df/cad4828b338206f0a7f18732c086c4ef959a7b 51 100% 3.32kB/s 0:00:00 (xfer#27, to-check=16550/51354) folder-to-backup/TestProject/.git/objects/ef/ folder-to-backup/TestProject/.git/objects/ef/e6d036f817624654f77c4a91ae6f20b5ecbe9d 94 100% 5.74kB/s 0:00:00 (xfer#28, to-check=16548/51354) folder-to-backup/TestProject/.git/objects/f2/ folder-to-backup/TestProject/.git/objects/f2/b43571ec42bad7ac43f19cf851045b04b6eb29 936 100% 57.13kB/s 0:00:00 (xfer#29, to-check=16546/51354) folder-to-backup/TestProject/.git/objects/fd/ folder-to-backup/TestProject/.git/objects/fd/f3f97d1b6e9d8d29bb69a88c4d89ca752bd937 807 100% 49.26kB/s 0:00:00 (xfer#30, to-check=16544/51354) folder-to-backup/TestProject/.git/objects/info/ folder-to-backup/TestProject/.git/objects/pack/ folder-to-backup/TestProject/.git/refs/ folder-to-backup/TestProject/.git/refs/heads/ folder-to-backup/TestProject/.git/refs/heads/master 41 100% 2.50kB/s 0:00:00 (xfer#31, to-check=16539/51354) folder-to-backup/TestProject/.git/refs/tags/ folder-to-backup/TestProject/TestProject.xcodeproj/ folder-to-backup/TestProject/TestProject.xcodeproj/project.pbxproj 11476 100% 659.24kB/s 0:00:00 (xfer#32, to-check=16536/51354) folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/ folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/contents.xcworkspacedata 156 100% 8.96kB/s 0:00:00 (xfer#33, to-check=16534/51354) folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/xcuserdata/ folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/xcuserdata/simon.xcuserdatad/ folder-to-backup/TestProject/TestProject.xcodeproj/project.xcworkspace/xcuserdata/simon.xcuserdatad/UserInterfaceState.xcuserstate 8190 100% 470.47kB/s 0:00:00 (xfer#34, to-check=16531/51354) folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/ folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/ folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/xcschemes/ folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/xcschemes/TestProject.xcscheme 3351 100% 192.50kB/s 0:00:00 (xfer#35, to-check=16527/51354) folder-to-backup/TestProject/TestProject.xcodeproj/xcuserdata/simon.xcuserdatad/xcschemes/xcschememanagement.plist 483 100% 27.75kB/s 0:00:00 (xfer#36, to-check=16526/51354) folder-to-backup/TestProject/TestProject/ folder-to-backup/TestProject/TestProject/AppDelegate.swift 2172 100% 117.84kB/s 0:00:00 (xfer#37, to-check=16524/51354) folder-to-backup/TestProject/TestProject/Info.plist 1442 100% 78.23kB/s 0:00:00 (xfer#38, to-check=16523/51354) folder-to-backup/TestProject/TestProject/ViewController.swift 505 100% 27.40kB/s 0:00:00 (xfer#39, to-check=16522/51354) folder-to-backup/TestProject/TestProject/Assets.xcassets/ folder-to-backup/TestProject/TestProject/Assets.xcassets/AppIcon.appiconset/ folder-to-backup/TestProject/TestProject/Assets.xcassets/AppIcon.appiconset/Contents.json 1077 100% 58.43kB/s 0:00:00 (xfer#40, to-check=16519/51354) folder-to-backup/TestProject/TestProject/Base.lproj/ folder-to-backup/TestProject/TestProject/Base.lproj/LaunchScreen.storyboard 1740 100% 94.40kB/s 0:00:00 (xfer#41, to-check=16517/51354) folder-to-backup/TestProject/TestProject/Base.lproj/Main.storyboard 1695 100% 91.96kB/s 0:00:00 (xfer#42, to-check=16516/51354) sent 1243970 bytes received 1220 bytes 75466.06 bytes/sec total size is 10693902652 speedup is 8588.17
Update with no files to upload
localhost:~ local-account$ rsync -a -e 'ssh -p 22 ' --progress -P /Users/local-account/folder-to-backup [email protected]:/backup/the-computer/ [email protected]'s password: building file list ... 51354 files to consider sent 1198459 bytes received 20 bytes 82653.72 bytes/sec total size is 10693902652 speedup is 8922.90
Backup is easy..
If you want incremental and full backups try Duplicity.
Hope this helps.
Donate and make this blog better
Ask a question or recommend an article
v1.7 Duplicity
What would you do right now if the computer or device you were reading this on died? Do you have adequate backups and do you keep your most important files safe?
Advertisement:
I wrote an article back in 2013 about good backup plans and I used to spend 3 hours every month burning backup DVD’s and adding them to an indexed catalog. I did not trust Apple iCloud to backup my photos and have had my fair share of lost photos. I came up with a plan for managing and sorting photos from multiple digital devices that put me in charge of copying photos from devices, backing up high res photos and shrinking photos for local use.
Fast forward 3 years and I am now capturing insane amounts of digital photos and videos and I am no longer backing up every month due to the massive amounts of data to now what seems like the small capacity of DVD’s. I am instead throwing all of my new files into an “Incoming” folder to be processed at a later date. This was fine as I assumed the data would be safe.
Murphy’s Law: Anything that can go wrong will go wrong.
On a recent hot summers day, I was performing maintenance on my primary computer (removing dust from the processor fan) and one of the hard drives dropped 30 centimeters to the ground. I thought that all would be fine and the data would be safe as the hard drive was a good quality one. Nope, the drive is dead, the drive was corrupted and fails to show files. The drive spins up ok and does not sound like anything is broken inside. That drive had my last 3 years of photos on it.
How can I prevent this in future?
This happened because I failed to allocate time to sort and backup files and I assumed that Murphy’s Law would not apply to me.
My traditional golden rules of backup:
Simon’s new rules of backup:
Why so many backups you may ask? Each backup method is best for different scenarios. Dropbox is good for recently changing files, Cloud backups are good for everything and local backups are good for when cloud providers stuff up or to cover major infrastructure damage/failure overseas.
The ultimate local backup solution is a Drobo 5N 5 NAS from https://www.pccasegear.com/ . Simply insert hard drives (up to 5 3.5″/64TB) plug it into a router or switch and you are good to go. Read more here. Even my new NBN router has a slot for a UBS stick to be inserted (see on the right below) allowing me to create a local WiFi file share.
Simple Cloud Backup Options:
Dropbox is the simplest and safest method of syncing and storing files offsite. You can store 2GB of files for free or 1TB (10,000 GB) for $11.58 AUD a month). Dropbox also works on mobile and tablet device too so it is a no-brainer. Encryption options are not baked in so it is not as secure. Install Dropbox at home and work and share files between locations via Dropbox.
Use this link ( https://db.tt/iZmBvErC ) to signup at Dropbox and get a bonus 500MB (2,500MB total) for free.
Backblaze.com was my favorite cloud storage provider for Windows of Mac that offers encrypted backups. Use this link to get a month free https://secure.backblaze.com/r/017nsm. It costs $5 a month and there are virtually no limits on the data you can store. My laptop has 90GB of data stored in Backblaze and it is all encrypted. Back blaze care about backup so much they have open sourced their backup enclosure designs: https://www.backblaze.com/blog/backblaze-storage-pod-4/
Use this link to get a month’s free cloud backup to give it a try: https://secure.backblaze.com/r/017nsm
Advanced Cloud Backup:
If you do not trust cloud providers to store your data and you want to have offsite backups on the internet you can buy your own server online and manage your own files encryption and sync methods.
Digital Ocean allows you to create and store your own files in the cloud from $5 a month (but you seed to set it up). Use this link and get a free server for 1 x month with 20GB storage. You can also create a 50GB cloud repository using a Vultr server. I even back up my servers to another server automatically.
Manual Backup
Performing a manual backup by plugging in a removable hard drive feels good but chances are you will not do it often enough. Most of the time simply highlighting all files and copy paste it to the new drive fails to be effective because files are often in use or access is denied to them by the operating system. Manually copying all files every month talks time and copying many smaller files over and over again will increase the wear on your drive and fragment your drive.
If you are working on important documents then do save as to a new filename frequently (and copy those to offsite destinations), never trust computers.
Manual backups are ok but they are only as good as the data you copy and the frequency you do it.
Network Attached Storage (NAS)
The main downside of cloud backup is it can take weeks if not months to get a full backup into the cloud. That’s where NAS (Network Attached Storage) comes into save the day.
A NAS is a mini computer that sits on your network (wired or wireless) and allows you to store files quickly with or without redundancy.
The ultimate local backup solution is a Drobo 5N 5 NAS from https://www.pccasegear.com/ . Simply insert hard drives (up to 5 3.5″/64TB) plug it into a router or switch and you are good to go. Read more here.
You can choose a NAS with different drive capacities and you can allocate more space to redundancy (referred to as RAID read about raid here https://www.cloudwards.net/what-is-raid/ ). You would want to at least mirror (duplicate) data (RAID 1) so if one drive fails you have a copy of files on the other disk.
Windows Advice:
Windows 7 had a nice Backup feature built in but Microsoft has changed that in Windows 8.1 and 10
https://support.microsoft.com/en-au/help/17143/windows-10-back-up-your-files
I feel Windows approach to automatically create backups of files on the C drive is a bad idea and copies of hundreds of thousands of files only increased the space ( http://www.apple.com/au/shop/product/ME182X/A/airport-time-capsule-3tb?fnode=5f ) used and wears out the drive.
Apple Mac Advice:
Apple OSX comes with a handy feature called Time Machine, if you plug in a large capacity drive OSX will ask if it can use the drive to store backups of your computer. Time Machine does a great job of keeping multiple versions of files and automatically phase out older versions when space on the backup drive gets low. The only catch is you may need to plug in your backup drive every few days/weeks to let Time Capsule do its thing.
Apple sells a wireless Time Capsule for those that do not want to plug in a drive to backup. Time capsules can be a bit tricky to setup if your home or office has a wireless network already but once it is setup it is rock solid.
http://www.apple.com/au/shop/product/ME182X/A/airport-time-capsule-3tb?fnode=5f
Technical Bonus:
Don’t assume new storage devices are more reliable if anything the cut-throat nature of the PC industry is driving down reliability and quality. If someone is telling you otherwise they are lying or are trying to sell you something.
Most larger capacity drives (over 1TB) store data on spinning metal platters and the quality and reliability varies between brand and models. Backblaze has a great write-up on hard drive reliability (based on experience of tens of thousands of drives): https://www.backblaze.com/blog/hard-drive-failure-rates-q3-2017/
Ever increasing “data capacity” of devices is delivering devices that are more fragile and ones that dedicate less physical space to actually store every bit of data on your computer. Most Solid State drives store data in shared cells ( MLC ) and data is more prone to being corrupted due to manufacturers increasing consumers risk to stay profitable.
My recovery tool of choice for any dead or deleted SD card, USB key or physical drive is http://www.diydatarecovery.nl as it ignores the file application table and looks at the sectors and tried to build files back from there. The only catch is recovery scans take days and you need a new/empty drive that is bigger than the drive you are scanning to restore files to. Here is my mini review os DIY DataRecovery.
So by now, you have:
Don’t forget to use good filenames
The World’s Best File Naming System (YouTube)
Good luck and happy backing up.
Restoring Files
If you have backup often up then all will be well when disaster strikes, if not you may need to use Restore Software.
Donate and make this blog better
Ask a question or recommend an article
v1.3 File naming…