This guide will show how you can automatically backup files to a Backblaze B2 clouds bucket with the Duplicati software on Windows and Linux.
I have blogged about how you can use the Backblaze personal backup program ( here, Use my link and get your first month of Backblaze backups free). The personal application from Backblaze is cool but what if you wanted to store 10GB for free (the first 10GB is free) or setup different buckets for different computers or applications.
Welcome to Backblaze B2 Cloud Storage.
In Australia B2 cloud costs (+GST).
- First 10GB is free.
- $0.005c per GB stored per month.
- To download, 1x GB costs $0.01c.
Check our my other related posts
- Backing up your computer automatically with BackBlaze software (no data limit)
- How to back up an iPhone (including photos and videos) multiple ways
This works alongside the private Backblaze backup mentioned here.
Backblaze B2 Pricing
How does Backblaze B2 Compare that to other cloud data storage provides.
|Provider||Storage ($/GB/Month)||Download ($/GB)|
Vendor cost breakdown (source).
Read here to view the official page on pricing.
Storage: The first 10 GB of total storage is free, and is $0.005 per GB per month for additional storage beyond 10 GB. Download: The first 1 GB of downloads per day are free. The price to download stored data costs $0.01 per GB beyond the free daily 1 GB. Transactions: A 'transaction' refers to any time an api call is made with your account, such as listing the contents of your bucket or downloading a file. B2 will charge for some transactions however, for the average user, these charges are largely negligible unless you are making an extremely high volume of api calls. Class A transactions are free. The first 2,500 Class B transactions are free each day. Additional Class B API calls are charged at $0.004 per 10,000 calls. The first 2,500 Class C transactions are free each day. Additional Class C API calls are charged is $0.004 per 1,000 calls.
https://downdetector.com is a great site for viewing is a vendor is down or has issues, lets compare, Backblaze with Amazon, Azure and Google Cloud
- Downdetector.com: Backblaze
- Downdetector.com: Azure
- Downdetector.com: Amazon
- Downdetector.com: Google Cloud
Backblaze have has far fewer downtime issues during the great outage of May/June 2019 compared to its competitors. What I like is Backblaze tell us how they do it and tell us what hard drives are good or bad.
Creating a Backblaze B2 Account
From your account page click billing and add your account details
Read the B2 Strater Guide: https://www.backblaze.com/b2/docs/
Also review your review the following under ‘My Settings‘
- Phone Numer (mobile) for usage alerts.
- Two Factor Authorization for extra security (don’t forget to çopy your backup codes)
Setup Billing Alerts (just in case)
- Alerts to your mobile phone (for Daily storage caps,bandwidth caps and class B/C caps)
- Under ‘Daily Storage Caps’ click ‘edit’ and set monetary alerts (you can increase alert limits later)
Click ‘Edit Caps’ and set daily limits on all categories
I set alert limits of 10c (I can increase this later)
Daily 10c alerts should see is costs creep above $3 a month.
Create a cloud bucket to store files
Go to https://secure.backblaze.com/b2_buckets.htm and click ‘Create Bucket’
You can create upto 100 buckets per account.
I created a descriptive bucket name that reminded me what server, drive and folder this was used to back up.
"#######-drive-backup-folder" (####### was the name of my computer)
I created a number of buckets (all private).
The Backblaze portal lists all of my new storage buckets.
Bucket Storage (Lifestyle Settings)
You can click the ‘Lifecycle Settings‘ button and set the desired buckets file retention policy (overwrite or keep all versions of a file etc).
Read the official Lifestyle settings faq here.
The default is to keep all versions of a file. Choose a setting that is required for each bucket(for me ‘Keep only the latest version of the file’ is best and lowers potential cost blowouts. Having all versions of a file may be handy for a Documents folder.
You could opt to keep all versions of a file (forever) but that would be expensive.
You tell Backblaze to delete older versions of a file after xx days (this is a good option)
You can also setup custom lifecycle rules to hide a file with a certain prefix after xx days then delete after xx days.
Create a Master Application Key for all buckets
You will need to create a Master Application Key that will grant you full access (Permissions: listKeys, writeKeys, deleteKeys, listBuckets, writeBuckets, deleteBuckets, listFiles, readFiles, shareFiles, writeFiles, deleteFiles ) to each bucket.
Snip from: https://secure.backblaze.com/app_keys.htm
Application keys are used as a pair: Key ID and Application Key. This allows B2 to communicate securely with different devices or apps. Once you generate your Master Application Key, this key has full capabilities. Create your own Application Keys to limit features like read/write. Learn more.
Do write down this “KeyID” and “applicationKey”
Don’t lose the Master Application Key.
Create an App Key for each bucket
A master key is like an administrator account, its great for complete access but you should create a separate key per bucket for better security.
Back under the App Keys you can click “Add a New Application Key” for each bucket.
Read the official guide on Application Keys here
- “Name of Key” (similar to your bucket (e.g “my-bucket-name-key”))
- “Allow access to Bucket(s)” (choose a bucket to grant access to).
- “Type of Access” (Read and Write, Read Only, Write Only)
- Set a “File name prefix” (snip from here)
"The file name prefix of files the download authorization token will allow b2_download_file_by_name to access. For example, if you have a private bucket named "photos" and generate a download authorization token for the fileNamePrefix "pets/" you will be able to use the download authorization token to access: https://f345.backblazeb2.com/file/photos/pets/kitten.jpg but not: https://f345.backblazeb2.com/file/photos/vacation.jpg."
- Set a “Duration” (snip from here)
"The number of seconds before the authorization token will expire. The minimum value is 1 second. The maximum value is 604800 which is one week in seconds."
I now has 1x Master Application key and 4x Application Keys for my 4x Buckets
TIP: Make sure you save the keys, if you did not grab the keys delete the keys and buckets (if there is not data and try again). If you don;t have the keys you are locked out of the buckets
OK, Let’s fill those buckets with data with Duplicati
Go to https://www.duplicati.com/ and familiarise yourself with the site
Setup Duplicati on Windows (10)
Go to https://www.duplicati.com/download and download the latest Windows client. At the time of writing Duplicati 184.108.40.206 beta was the latest version. Thew instal was 1MB (the download serve was not that fast)
Warning: Beta software may have bugs, feel free to check out the Github page for Issues etc.
Open the install and Click ‘Next‘
Agree to the ‘Licence Agreement‘ by clicking ‘Next‘
Select Duplicati components to install and click ‘Next‘
Tick ‘Launch Duplicati‘ an click ‘Finish‘
Using Duplicati Software
Upon first open you will be prompted to set a password (do this even if you are the only user of thew computer)
Set a strong and unique password
When the page reloads it will report it is not logged in.
Enter your password
The main screen of Duplicati reports that there are no scheduled set.
OK, lets create a backup schedule.
Creating a Backup on Windows
If your screen is small you will need to click the menu at the top right and click ‘Add Backup’, if your screen is large there will be an ‘Add Backup’ menu on the left hand side of the screen.
Select ‘Configure a new backup‘ and click ‘Next‘
You will be able to name and describe the backup schedule name and be able to set up an encryption password.
Then click ‘Next‘
Under ‘Storage Type‘ choose ‘B2 Cloud Storage‘
When you choosing ‘B2 Cloud Storage‘ you will be directed to the screen below where you stecify..
- B2 Cloud bucket name
- Path in the bucket to store the files (leave blank)
- Enter your Backblaze Account ID (e.g. ##########################)
- Enter the Buckey ApplicationKey (e.g. ###############################)
I entered my credentials
Click ‘Test Connection‘, if you entered everything OK you should see ‘Connection Worked‘
Now you can select the folders to backup on your local computer
Now you can choose a schedule to backup
Now you can set the remote backup size and backup retention mode.
Read the official guide on setting the volume size here
- Increase the ‘Remote Volume Size’ if you are worried about consuming your daily API quota in Backblaze (this may happen if your upload speed is fast and your backup is very large).
- Lower the ‘Remote Volume Size’ size if backups are small or your upload speed is low.
Choose a desirable backup retention mode. Duplicati only offers full backups (not incremental or differential backups). This will be an issue if your backups are large.
- Keep all backups (“Nothing will be deleted. The backup size will grow with each change.“)
- Delete backups that are older than (“If at least one newer backup is found, all backups older than this date are deleted.“)
- Keep a specific number of backups (“Once there are more backups than the specified number, the oldest backups are deleted.“)
- Smart Backup Retention (“Over time backups will be deleted automatically. There will remain one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup.“)
- Custom backup retention (“Enter a retention strategy manually. Placeholders are D/W/Y for days/weeks/years and U for unlimited. The syntax is: 7D:1D,4W:1W,36M:1M. This example keeps one backup for each of the next 7 days, one for each of the next 4 weeks, and one for each of the next 36 months. This can also be written as 1W:1D,1M:1W,3Y:1M.“)
Each storage option has pro’s and cons with potential cost implications. I would not set 300GB to “Keep all backups” and backup daily. That would be expensive.
TIP: Don”t forget to save your encryption key if you set one
Now you will see the backup schedule on your Duplicati screen.
You can run, edit or delete the backup status here.
I clicked ‘Run‘ and the backup was under way.
A progress bar updates as the backup happens.
A slight delay happens at the end of the backup.
I logged into the Backblaze portal to see how many files and usage credits were used.
Note it seems?
When the files are backed up a verification processes happens.
Now the backup has completed.
I waited 2 minutes and finally Backblaze updated it’s status, Duplicati used 3 files and 87MB
I logged into the view the Duplicati files that were uploaded and I could see the three files were there and they were encrypted.
Restoring a Backup on Windows
Having a backup is useless you can restore the files (especially when they are encrypted).
Lets restore a few files and see if it works.
I opened Duplicati and clicked ‘Restore‘
I clicked the bottom option that listed my bucket name/backup name and clicked ‘Next‘
Before I ran the restore I deleted some location files that were just backed up.
Duplicati now asked me to choose files to restore.
I chose the original location to restore to and clicked ‘Restore‘
Duplicati took a few minutes to restore the files.
The restore was a success.
Beware, make sure you calculate the costs to upload and download files.
Do set alarms on cap’s limits and review them often.
Second Backup Schedule on Windows
I quickly set a second backup schedule of a different folder to a different bucket.
This backup was 300GB so I will only backup once a month
The backup is listed on the front screen of Duplicati.
Do create multiple buckets and set multiple backup schedules to break up the backups into smaller chunks.
Setup Duplicati on Linux (Debian or Ubuntu)
Lets setup B2 Cloud Storage backup on Linux.
I visited the Duplicati forums here. Duplicati informed me that the Linux client needs mono installed
Duplicati depends on other software. For Windows, Microsoft .NET Framework 4.5 or higher needs to be installed. Linux and Mac OS X require Mono to be installed.
Mono is a Cross Platform open source .NET that runs on Linux (read here)
On Ubuntu 1.08 I installed mono with these commands
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list sudo apt update sudo apt install mono-devel gtk-sharp2 sudo apt-get install libmono-2.0-1
On Debian 9.9 I installed Mono with these commands (from here)
sudo apt install apt-transport-https dirmngr gnupg ca-certificates sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list sudo apt update sudo apt install mono-devel gtk-sharp2 sudo apt-get install libmono-2.0-1
I rebooted the serves and the CPU and memory went crazy after I installed Mono. My Nixstats Service reports were going off telling me my servers CPU was busy.
If you don’s have reports on Linux server CPU/MEM/Disk reports coming to you via Email and Telegram then read this.
I felt sorry for my web and database server server(s) so I rebooted them, wow, My website came back up in about 10 seconds thanks to UpClouds fast Max IOPS disks.
Installing Duplicati on Linux
I visited the download Duplicati page here and copied the Ubuntu and Debian download URL.
At the time of writing it was..
I downloaded the deb package
cd /utils/duplicati wget https://updates.duplicati.com/beta/duplicati_220.127.116.11-1_all.deb
I then ran this command to install the package
sudo dpkg -i duplicati_18.104.22.168-1_all.deb
I received this error even thpogh libmono-2.0-1 was installed
dpkg: dependency problems prevent configuration of duplicati: duplicati depends on libmono-2.0-1; however: Package libmono-2.0-1 is not installed.
This forum post helped me, I ran
sudo apt install -f
sudo apt install -f Reading package lists... Done Building dependency tree Reading state information... Done Correcting dependencies... Done The following additional packages will be installed: libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4 libindicator7 libmono-2.0-1 Suggested packages: indicator-application The following NEW packages will be installed: libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4 libindicator7 libmono-2.0-1 0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded. 1 not fully installed or removed. Need to get 143 kB of archives. After this operation, 541 kB of additional disk space will be used. Do you want to continue? [Y/n]
And I ran the following
sudo apt dist-upgrade --fix-missing
I now ran the following to install the duplicati package.
sudo dpkg -i duplicati_22.214.171.124-1_all.deb
It worked on Ubuntu 18.08 bt not Debian 9.9
Debian produced the following error
sudo dpkg -i duplicati_126.96.36.199-1_all.deb (Reading database ... 70082 files and directories currently installed.) Preparing to unpack duplicati_188.8.131.52-1_all.deb ... Unpacking duplicati (184.108.40.206-1) over (220.127.116.11-1) ... dpkg: dependency problems prevent configuration of duplicati: duplicati depends on libappindicator0.1-cil | libappindicator3-0.1-cil; however: Package libappindicator0.1-cil is not installed. Package libappindicator3-0.1-cil is not installed. dpkg: error processing package duplicati (--install): dependency problems - leaving unconfigured Processing triggers for mime-support (3.60) ... Errors were encountered while processing: duplicati
On Debian I installed the stretch version of the package
sudo apt-get install libappindicator1
Then in Debian I fixed broken packages
sudo apt --fix-broken install
This I was able to install Duplicati on Debian
sudo dpkg -i duplicati_18.104.22.168-1_all.deb
Running Duplicati CLI
I can now Run Duplicati CLI (it works)
duplicati-cli See duplicati.commandline.exe help <topic> for more information. General: example, changelog Commands: backup, find, restore, delete, compact, test, compare, purge, vacuum Repair: repair, affected, list-broken-files, purge-broken-files Debug: debug, logging, create-report, test-filters, system-info, send-mail Targets: aftp, amzcd, azure, b2, box, cloudfiles, dropbox, ftp, file, googledrive, gcs, hubic, jottacloud, mega, msgroup, onedrive, onedrivev2, sharepoint, openstack, rclone, s3, ssh, od4b, mssp, sia, tahoe, webdav Modules: aes, gpg, zip, 7z, console-password-input, mssql-options, hyperv-options, http-options, sendhttp, sendmail, runscript, sendxmpp, check-mono-ssl Formats: date, time, size, encryption, compression Advanced: mail, advanced, returncodes, filter, filter-groups, <option> http://www.duplicati.com/ Version: - 22.214.171.124_beta_2018-11-28
TIP: Don’t run “duplicati” that’s the desktop verison.
Creating a Duplicati Backup on Linux
I read this page and this page to get an understanding of how to back up from the command line. I was not sure if I needed to run a service or run the CLI from a bach script so I asked on the forums here.
Within minutes I had this reply
kenkendk 1h Hi @feartec, welcome to the forum! The CLI and Web-UI are not the same, and operate independently. If you use the CLI, you need to use cron or something similar to run the backups periodically. You can use duplicati-cli help to get started. You do not need the server running for this. For the web-UI we do not currently have any tools for manipulating the backups other than the main WebUI itself. You need to somehow forward/tunnel access to the server, and then access the WebUI from a local browser.
Backing up a folder with Duplicati to a Backblaze B2 Bucket
I created this folder structure
/test/ /test/testfile.txt < hello /test/subfolder/ /test/subfolder/file.txt < world
/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=**********
TIP: read more about “dblock” size here
Checking remote backup ... Listing remote folder ... Scanning local files ... 4 files need to be examined (8 bytes) Checking remote backup ... Listing remote folder ... Verifying remote backup ... Remote backup verification completed Downloading file (1021 bytes) ... Downloading file (1.06 KB) ... Downloading file (2.00 KB) ... 0 files need to be examined (0 bytes) Duration of backup: 00:00:06 Remote files: 3 Remote size: 4.05 KB Total remote quota: 0 bytes Available remote quota: 0 bytes Files added: 0 Files deleted: 0 Files changed: 0 Data uploaded: 0 bytes Data downloaded: 4.05 KB Backup completed successfully!
I logged into Backblaze to check the backups
I created a second test file and added 2 test files
mkdir /test2 cd /test2 wget http://ipv4.download.thinkbroadband.com/10MB.zip wget http://ipv4.download.thinkbroadband.com/5MB.zip ls -al total 15368 ... -rw-r--r-- 1 user user 10485760 Jun 3 2008 10MB.zip -rw-r--r-- 1 user user 5242880 Jun 3 2008 5MB.zip
I ran the backup with the second folder in the command line
/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=********** --retention-policy="1W:1D,4W:1W,12M:1M"
Backup started at 06/07/2019 23:05:58 Checking remote backup ... Listing remote folder ... Scanning local files ... 7 files need to be examined (15.00 MB) Uploading file (15.03 MB) ... Uploading file (11.43 KB) ... Uploading file (1.29 KB) ... Checking remote backup ... Listing remote folder ... Verifying remote backup ... Remote backup verification completed Downloading file (1.29 KB) ... 2 files need to be examined (8 bytes) Downloading file (11.43 KB) ... Downloading file (15.03 MB) ... 0 files need to be examined (0 bytes) Duration of backup: 00:00:10 Remote files: 6 Remote size: 15.05 MB Total remote quota: 0 bytes Available remote quota: 0 bytes Files added: 2 Files deleted: 0 Files changed: 0 Data uploaded: 15.05 MB Data downloaded: 15.05 MB
Restoring a Duplicati Backup on Linux
O’k lets delete the local “/test2/” folder then restore it
rm -R /test2
Now lets restore it (official guide)
/usr/bin/duplicati-cli restore "b2://bucket-name-goes-here/test" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --passphrase=************* --overwrite=true --restore-path="/test2"
Restore started at 06/07/2019 23:21:51 Checking remote backup ... Listing remote folder ... Checking existing target files ... 2 files need to be restored (15.00 MB) Scanning local files for needed data ... Downloading file (15.03 MB) ... 0 files need to be restored (0 bytes) Verifying restored files ... Restored 0 (0 bytes) files to /test2 Duration of restore: 00:00:05
Did it work
cd /test2 ls -al total 15368 ... -rw-r--r-- 1 use user 10485760 Jun 3 2008 10MB.zip -rw-r--r-- 1 user user 5242880 Jun 3 2008 5MB.zip
Yes it worked
Backup 2 folders an emailing the results
I downloaded a 100MB file
cd /test2 wget http://ipv4.download.thinkbroadband.com/100MB.zip
I ran this command to backup an emailthe progress
/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=***************** [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"
Backup started at 06/08/2019 00:02:28
Checking remote backup …
Listing remote folder …
Scanning local files …
8 files need to be examined (115.00 MB)
7 files need to be examined (15.00 MB)
Uploading file (49.99 MB) …
Uploading file (18.01 KB) …
Uploading file (49.99 MB) …
Uploading file (17.97 KB) …
Uploading file (233.33 KB) …
Uploading file (33.17 KB) …
Uploading file (1.42 KB) …
Compacting remote backup …
Checking remote backup …
Listing remote folder …
Verifying remote backup …
Remote backup verification completed
Downloading file (1.42 KB) …
4 files need to be examined (15.00 MB)
Downloading file (18.01 KB) …
Downloading file (49.99 MB) …
0 files need to be examined (0 bytes)
Duration of backup: 00:00:15
Remote files: 13
Remote size: 115.33 MB
Total remote quota: 0 bytes
Available remote quota: 0 bytes
Files added: 1
Files deleted: 0
Files changed: 0
Data uploaded: 100.28 MB
Data downloaded: 50.01 MB
Backup completed successfully!
Backup Types and Versions
Read up here on backup retention policies here
TIP: Use the desktop Duplicati software to generate a backup then use the export to command line option to get the command line parameters to use in the command line version.
After you click Commandline you can view each separate command liene option and parameter.
This was handy for finding this backup retention parameter
“This will give me “one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup. “
Setup Auto Backup Cron Scripts
Login to your linux machines and edit a bash file (e.g in “/scripts/backupnow.sh” folder).
Make the script executable
chmod +x /scripts/backupnow.sh
Add the following
#!/bin/bash /usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=123 [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"
Now you can add the script to your crontab by running the following.
Make sure you have the following in your crontab
Add a cron entry to run the bash script
#Backup Database 0 22 * * * /bin/bash /scripts/backupnow.sh
Use https://crontab.guru/ to check the time you want to run thew script
v1.1 Fixed typo (”downalod” thanks to @NicolasWolf for reporting.)
v1.0 Initial Post (added retention polity and Linux crontabs
v0.95 Backup, Restore and Send Email
v0.91 Added Duplicati service info
v0.9 Working Copy