• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Create a VM ($25 Credit)
  • Buy a Domain
  • 1 Month free Back Blaze Backup
  • Other Deals
    • Domain Email
    • Nixstats Server Monitoring
    • ewww.io Auto WordPress Image Resizing and Acceleration
  • About
  • Links

IoT, Code, Security, Server Stuff etc

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

  • Cloud
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to buy a new domain and SSL cert from NameCheap, a Server from Digital Ocean and configure it.
    • Setting up a Vultr VM and configuring it
    • All Cloud Articles
  • Dev
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to setup pooled MySQL connections in Node JS that don’t disconnect
    • NodeJS code to handle App logins via API (using MySQL connection pools (1000 connections) and query parameters)
    • Infographic: So you have an idea for an app
    • All Development Articles
  • MySQL
    • Using the free Adminer GUI for MySQL on your website
    • All MySQL Articles
  • Perf
    • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap
    • All Performance Articles
  • Sec
    • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
    • Using OWASP ZAP GUI to scan your Applications for security issues
    • Setting up the Debian Kali Linux distro to perform penetration testing of your systems
    • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
    • PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API
    • Setting strong SSL cryptographic protocols and ciphers on Ubuntu and NGINX
    • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
    • All Security Articles
  • Server
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All Server Articles
  • Ubuntu
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Useful Linux Terminal Commands
    • All Ubuntu Articles
  • VM
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All VM Articles
  • WordPress
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
    • How to backup WordPress on a host that has CPanel
    • Moving WordPress to a new self managed server away from CPanel
    • Moving a CPanel domain with email to a self managed VPS and Gmail
    • All WordPress Articles
  • All

backblaze

Goodbye Dropbox, One Drive, iCloud and Hello Nextcloud private cloud on UpCloud

June 14, 2020 by Simon

I recently came across NextCloud Hub server (free on self-hosted servers) and I wanted to set up my own private cloud server to store my own files.

I wan’t to be able to access my files on Windows, Mac, Android and iOS.

Most of all I want a place in the cloud (that I own) that I can upload my Acronis backup of C Drive as the Backblaze client (read my review of Backblaze here) is a bit slow at uploading a 150GB backup file to the USA.

To create my own Nextcloud server I will need to login to these services.

  • I logged into my Domain Name provider porkbun.com (to ensure I had a domain name)
  • I logged into Cloudflare.com (to manage my DNS for a subdomain (redirected from PorkBun)).
  • I logged into my UpCloud.com account. (to deploy a new virtual machine)

Fyi: If you don’t have a favourite virtual machine provider you can use my referral link to obtain $25 free credit (only if you are new to UpCloud). Every new user who signs up with my referral link will receive a $25 bonus to get started. That’s 5 months free server (1 CPU and 1GB memory Linux server) 

Post Index

  1. NextCloud System Requirements
  2. Creating a new Virtual Machine at UpCloud
  3. Updating Ubuntu
  4. Installing Common Software Packages
  5. Securing SSH with the Google Authentication PAM module
  6. Installing a Firewall
  7. Installing NGINX and DNS
  8. Installing PHP/PHP-FPM
  9. Installing MySQL
  10. Nixstats
  11. CronTab Updates
  12. Misc Security Stuff

1. NextCloud System Requirements

I checked the NextCloud version 18  system requirements and it needs the following to deploy.

  • Ubuntu 18.04 LTS (recommended)
  • MySQL 5.7+ or MariaDB 10.2+ (recommended)
  • Nginx with php-fpm
  • PHP 4 (recommended)

Minimum Memory Requirements 

Nextcloud needs a minimum of 128MB RAM, and they recommend a minimum of 512MB.

I can deploy a server with at least 512MB memory free. The minimum UpCloud server I can deploy comes with 1GB of memory for $5 a month.

Time to create a new server.

2. Creating a new Virtual Machine at UpCloud

I logged into UpCloud and  clicked “Deploy Server“

Deploy Server Button

I selected Singapore as the place to deploy my new server (as it was closest to me here in Australia). UpCloud does not have servers here in Australia yet.

I chose to deploy a server in Singapore

I checked https://wondernetwork.com/pings/ to ensure Singapore is the fastest location near.  My server https://fearby.com is located in Chicago as it’s closer to my average readers and search engines.

Ping Speeds

I would like my NextCloud server to be as fast as possible to me. Singapore is the faster UpCloud datacenter near me.

I selected a server with 1 CPU Core, 1GB of Memory, 25GB of storage and 1TB of network traffic. I will add a 500GB drive to this server for additional storage.

If the server needs more resources I will upgrade it later.

Server tier's $5 month to $640 a month

The only downside of a $5/m server is the 1TB network quota. If I overuse the network (downloads) I will get an extra charge. 

I reached out to the support to verify the costs if I go over my quota.

Long answered the question.

My question to UpCloud chat support.

Q1) With a $5/m server with 1TB quota what is the over charge costs if I go over 1TB
Q2) Is 1TB quota up and down or just down?

Prompt Answer

Hi Simon,

Good to speak to you again.

A1) Only Simple plans include monthly allowance of outgoing network traffic. After the allowance, the cost is $0.01/GB. It was a lot higher, but we reduced it to make it more competitive.

A2) The quota is for outgoing network traffic from your servers, all incoming and private traffic between your UpCloud servers is free of charge.
Regards,
~Long Lam

I hope this is helpful, let us know if you have any further questions. 

Based on this information if I use all of my 1TB Monthly quota downloading files and I download and extra 150GB (e.g A 150GB Acronis backup image) it will cost $1.5 extra. That’s not bad.

UpCloud Chat Support

Before I selected a server type (Simple or Flexible) or storage type  (MAX IOPS or HDD) I jumped onto the UpCloud chat and asked a few questions.

Q1) Hello, When deploying a server is there a cost difference between MAX IOPS and HDD storage? I am looking at a 500GB drive

A1) Storage (MaxIOPS), per GB $0.00031/ hourly $0.22/ monthly, Storage (HDD), per GB $0.000078/hourly  $0.06 / monthly 

Q2) What’s the difference between Simple and Flexible?

A2) Flexible will/turn out more expensive depending on your use case, generally, it is more suited for short term deployments.

> With our flexible plans, you decide yourself how much CPU, memory and block storage your cloud server is allocated. This gives you incredible flexibility and allows you to fully customise your cloud server according to your specific needs.
 
>Do also note when flexible plans are shutdown we only charge you for allocated storages and IPv4. Whereas in simple plans, it will be charged fully even when shutdown.
 
> Our simple plans are billed by the hour, up to a limit of 672 hours per month. Should you decide to use your fewer hours, you will only be billed for the hours you actually used.
Question 1 to UpCloud chat

UpCloud has very responsive and helpful chat staff.  I never had this level of help with Vultr, Digital Ocean or AWS.

Question 2 to UpCloud

After I chatted with UpCloud support I decided to deploy a simple (Ubuntu 18.04) Server with 1 CPU Core, 1TB network traffic, 1GB of memory, 25GB system drive and an extra 500GB storage device.

When you create a server you can add an extra storage device. Nice.

Add a new device to the main storage device.

When adding an extra storage device you can choose faster MaxIOPS storage or slower HDD based storage. 

I will choose HDD storage as it will be cheaper for a 500GB device.

Second storage MaxIOPS or HDD storage

I created a 500GB storage device for a Nextcloud data drive.

You can create up to 2TB storage devices with UpCloud.

Name of the second storage device

I selected Ubuntu 18.04 LTS as the operating system.

I chose Ubuntu as the operating system

I configured a login method as “Only SSH Keys” as I have already added my SSH key with a passphrase.

Login method SSH Keys only

I selected my SSH key.

If you have not previously added an SSH Key to UpCloud then click Add new. Read more here.

I selected an Initialisation script I previously created (that just outputs a “Hello World” to a text file). One day I will create an Ansible or Terraform script to set up a server.

Select SSH Key and choose an init script

I clicked Deploy

Fyi: If you don’t have a favourite virtual machine provider you can use my referral link to obtain $25 free credit (only if you are new to UpCloud). Every new user who signs up with my referral link will receive a $25 bonus to get started. That’s 5 months free server (1 CPU and 1GB memory Linux server).

I entered my desired hostname 

Deploying a server at UpCloud

I had a notification that the UpCloud Deploy is being deployed..

Deploy Underway

I could see in my UpCloud dashboard that the server was being deployed.

List of all my servers at UpCloud

Server deploy is underway

Wow that took a whole minute to deploy a 525GB Server.

Deploy log said it took 1 minute to deploy

Wow UpCloud are fast

Configuring the server with Putty

Now it is time to connect to the Ubuntu Servers CLI and configure the server.  I grabbed the IP address that was listed at UpCloud.

I opened Putty  and added the IP address for the server.

New Putty connection

Under the Auth section in Putty I added the path to my SSH Private Key (the same one that configured in the new server)

Putty add ppk file

I saved the connection and clicked Open. I clicked Yes to the SSH fingerprint when I verified it was correct.

SSH Connect Verity

I now had root access to my new server.

Default login

Time to update Ubuntu.

3. Updating Ubuntu

I ran this command to update Ubuntu.

sudo apt-get update && sudo apt-get upgrade

Confirming the 2x storage disks

I ran this command to verify I had the 2 storage devices I selected at server deploy.

sudo lsblk |grep disk
vda    252:0    0   25G  0 disk
vdb    252:16   0  500G  0 disk

Yes, I have a 25GB disk and a 500GB disk

4. Installing Common Software Packages

I installed these packages

sudo apt-get install htop
sudo apt-get install lshw
sudo apt-get install ufw
sudo apt-get install ncdu
sudo apt-get install nmap
sudo apt-get install iozone3
sudo apt install pydf
sudo apt install mc
sudo apt install nnn

5. Securing SSH with the Google Authentication PAM module

Before I carry on any further I need to enable hardware 2FA login protections to all SSH logins. I will follow the guide I created here (Setup two factor authenticator protection at login on Ubuntu or Debian).

Warning: Take a backup of your server first. If you set this up wrong say bye-bye to your server. If I lose my YubiCo YubiKey and forget my backup codes I will have a hard time getting back in.

I will force all SSH logins to require my Hardware YubiCo YubiKey to be inserted (to generate a temporary One Time Password (OTP)).

You don’t need a YubiCo YubiKey, a generic software authentication app is OK but I prefer hardware devices as they are more secure.

YubiKey In USB Port Photo

I set the timezone to match Australia/Sydney. If I enabled a 2FA (OTP) at login with a different timezone than my connecting machine I would never be able to login to my server as my server and local PC need to be in the same timezone.

I ran this command to set the time in Ubuntu.

pkg-reconfigure tzdata

I then checked the time

sudo hwclock --show
2020-05-31 23:17:02.873751+1000

I installed the Google Authentication PAM Module (read more)

sudo apt install libpam-google-authenticator

I ran this command to configure the Google PAM Module

google-authenticator

I was presented with these questions

Do you want authentication tokens to be time-based (y/n) y

I was presented with a secret key, verification code and backup codes (I saved these somewhere safe)

Do you want me to update your “/root/.google_authenticator” file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server, we allow an extra token before and after the current time. This allows for a time skew of up to 30 seconds between authentication server and client. If you experience problems with poor time synchronization, you can increase the window from its default size of 3 permitted codes (one previous code, the current code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes between client and server.

Do you want to do so? (y/n) y

If the computer that you are logging into isn’t hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.

Do you want to enable rate-limiting? (y/n) y

I can review all config values later with this command

sudo nano ~/.google_authenticator

Now I will enable 2FA at login by editing this file

sudo nano /etc/pam.d/sshd

I searched for “@include common-auth” then added this line after it.

auth required pam_google_authenticator.so

I then comment out the following line (this is the most important step, this forces 2FA)

#@include common-auth

Picture of my /etc/pam.d/sshd changes

pam chnages

I saved the file /etc/pam.d/sshd 

Now I can enable the PAM Module by editing this file

sudo nano /etc/ssh/sshd_config

I searched for

ChallengeResponseAuthentication

And change the value to “yes”

I ensured the following line exists

UsePAM yes

I added this line then saved /etc/ssh/sshd_config

AuthenticationMethods publickey,password publickey,keyboard-interactive

Now I edited /etc/pam.d/common-auth

sudo nano /etc/pam.d/common-auth

I added the following line before the line that says “auth [success=1 default=ignore] pam_unix.so nullok_secure”

auth required pam_google_authenticator.so

Now I can restart the SSH Service and test the 

/etc/init.d/ssh restart
[ ok ] Restarting ssh (via systemctl): ssh.service.

I restarted my putty session and reconnected to my server and I was prompted for the password for my private key and the randomly generated one-time password that was linked to my YubiCo YubiKey. Nice

Now I need to whitelist my SSH port to select IP’s.

6. Installing a Firewall

I installed the UFW firewall by typing this command

sudo apt-get install ufw

I configured UFW to rate limit SSH logins by typing this command

sudo ufw limit ssh comment 'Rate limit hit for openssh server'
Rules updated
Rules updated (v6)

I configured some common ports

sudo ufw allow ssh/tcp
sudo ufw logging on
sudo ufw allow http
sudo ufw allow https
sudo ufw allow 22
sudo ufw allow 53
sudo ufw allow 80
sudo ufw allow 443
sudo ufw allow 873

I added Cloudflare firewall rules (as my domain is behind their firewall and I will remove all direct IP access to my server later)

sudo ufw allow from 173.245.48.0/20
sudo ufw allow from 103.21.244.0/22
sudo ufw allow from 103.22.200.0/22
sudo ufw allow from 103.31.4.0/22
sudo ufw allow from 141.101.64.0/18
sudo ufw allow from 108.162.192.0/18
sudo ufw allow from 190.93.240.0/20
sudo ufw allow from 188.114.96.0/20
sudo ufw allow from 197.234.240.0/22
sudo ufw allow from 198.41.128.0/17
sudo ufw allow from 162.158.0.0/15
sudo ufw allow from 104.16.0.0/12
sudo ufw allow from 172.64.0.0/13
sudo ufw allow from 2400:cb00::/32
sudo ufw allow from 2405:8100::/32
sudo ufw allow from 2405:b500::/32
sudo ufw allow from 2606:4700::/32
sudo ufw allow from 2803:f800::/32
sudo ufw allow from 2c0f:f248::/32
sudo ufw allow from 2a06:98c0::/29

I added appropriate Whitelisted IP’s that can connect to Port 22 (SSH), removed blanket port 22 access and I configured my firewall to allow 91 incoming and outgoing rules (this is a secret)

I reloaded and enabled the firewall.

sudo ufw reload
sudo ufw disable
sudo ufw enable

7. Installing NGINX and DNS

I update Ubuntu again

sudo apt-get update && sudo apt-get upgrade

I installed Nginx

sudo apt-get install nginx

I edited my NGINX config and I change the default www folder location. 

I also configured the log file location, mime types, max body size, gzip, default ports, ssl cert paths, security headers, default page, server name, sensitive file block rules, dns server, cache headers etc.

Read more to here to configure Nginx etc.

Fyi: Nginx config file locations

sudo nano /etc/nginx/nginx.conf
sudo nano /etc/nginx/sites-available/default

I typed my servers IP address into a web browser

Nginx installed

I created an index.html file in the www folder and added “Hello World” to the file.

If I type my server’s IP address into a browser I can see this file.

My DNS is with Cloud flare so I logged in and added 2 DNS entries (IPv4 and IPv6) that direct traffic my new server IP(s) for this subdomain. To obtain the IP addresses I logged into UpCloud and clicked my server then clicked Network and noted my IPv4 and IPv6 addresses.

I then went to Cloudflare and added a DNS record for IPv4 and IPv6 pointing to my servers IP(s). I enabled Cloudflare Proxying to allow Cloud flare to try and hide the IP of the server.  I then configured my firewall to block access to the IP except via Cloudflare and my whitelist.

I then checked for worldwide DNS propagation with https://www.whatsmydns.net/. After 3 minutes my DNS changes were all around the world. Thanks, Cloudflare.

I tried loading my site but CLiudflare said it was down.

Site wont load.

I created a new HTTPS certificate at Cloud flare just to be sure and added it to my sites.

Generated  new SSL cert

After investigating further I found this was because my primary website has a “Strict-Transport-Security header and I had enabled Full (Strict) SSL/TLS Encryption. I changed this to Full at Cloudflare.

Cloudflare HTTPS section

My site was now working.

SIte works

8. Installing PHP/PHP-FPM

To Install PHP 7.4 I ran this command to be able to get the latest version of PHP

sudo apt-get update
sudo apt -y install software-properties-common
sudo add-apt-repository ppa:ondrej/php
sudo apt-get update

I installed PHP 7.4 with this command

sudo apt -y install php7.4

I checked that PHP is installed by running 

php -v
PHP 7.4.6 (cli) (built: May 14 2020 10:02:44) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.6, Copyright (c), by Zend Technologies

I setup some PHP Modules

sudo apt install php7.4-common php7.4-mysql php7.4-xml php7.4-xmlrpc php7.4-curl php7.4-gd php7.4-imagick php7.4-cli php7.4-dev php7.4-imap php7.4-mbstring php7.4-soap php7.4-zip php7.4-bcmath php7.4-tidy 

I noticed apache2 installed (and broke my Nginx)  so I uninstalled it.

 sudo apt-get remove apache2

I also blocked apache from installing again

apt-mark hold apache2
apache2 set on hold.

I checked to make sure Apache was blocked from installing

apt-mark hold apache*

apache2 was already set on hold.
apache2-bin set on hold.
apache2-utils set on hold.
apache2-data set on hold.
apache2-doc set on hold.
apache2-suexec-pristine set on hold.
apache2-suexec-custom set on hold.
apache2-dbg set on hold.
apache2-dev set on hold.
apache2-ssl-dev set on hold.
apachedex set on hold.
apacheds set on hold.
apachetop set on hold.

Now I will install PHP-FPM.

FPM is a process manager to manage FastCGI in PHP

sudo apt-get install php7.4-fpm

I checked the status of the PHP FPM service with

sudo service php7.4-fpm status

Output

php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-06-06 21:34:31 AEST; 1min 54s ago
     Docs: man:php-fpm7.4(8)
  Process: 7767 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
  Process: 7772 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
 Main PID: 7769 (php-fpm7.4)
   Status: "Processes active: 0, idle: 2, Requests: 0, slow: 0, Traffic: 0req/sec"
    Tasks: 3 (limit: 1147)
   CGroup: /system.slice/php7.4-fpm.service
           |-7769 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
           |-7770 php-fpm: pool www
           `-7771 php-fpm: pool www

I might add some PHP child workers if I add more CPU’s to this server later

I edited my php.ini

sudo nano /etc/php/7.4/fpm/php.ini

I made these changes to php.ini

file_uploads = On
allow_url_fopen = On
memory_limit = 512M
post_max_size = 50M
upload_max_filesize = 50M
cgi.fix_pathinfo = 0
max_execution_time = 360
date.timezone = Australia/Sydney

I added read this page (Nginx Configuration) and edited my /etc/nginx/sites-enabled/default

I tested and reloaded the Nginx config and restarted NGINX and PHP

nginx -t
nginx -s reload

sudo systemctl restart nginx.service
sudo systemctl restart php7.4-fpm

sudo systemctl status nginx.service
sudo systemctl status php7.4-fpm

To test PHP FPM I created a php file in my website root and added the following text

<?php phpinfo( ); ?>

I loaded this file in a browser and I confirmed that PHP-FPM was installed.

The test was ok (I deleted this test file), I deleted the index.html and created an index.php file

PHP-FPM test ok

9. Installing MySQL

To install MySQL I ran the following command

fyi: All usernames and database names are for example only.

sudo apt install mysql-server

I configured MySQL With this command

sudo mysql_secure_installation
Securing the MySQL server deployment.

Connecting to MySQL using a blank password.

...
Would you like to setup VALIDATE PASSWORD plugin?
y


There are three levels of password validation policy:
STRONG

Please set the password for root here.
New password:
**************************************************

Re-enter new password:
**************************************************

Estimated strength of the password: 100

Do you wish to continue with the password provided?
y


Remove anonymous users?
y

Disallow root login remotely?
y

Remove test database and access to it?
y

Reload privilege tables now?
y

Now to test MySQL I will login to it

sudo mysql -u root -p
************************************************************

Now I ran the following to create a database for Nextcloud

mysql> CREATE DATABASE databasename CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
Query OK, 1 row affected (0.00 sec)

I verified the database was created

mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| databasename       |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

I created a database user 

mysql> CREATE USER 'username'@'localhost' IDENTIFIED BY '************************************';
Query OK, 0 rows affected (0.00 sec)

I verified the use was created with this command

mysql> SELECT User,Host FROM mysql.user;
+------------------+-----------+
| User             | Host      |
+------------------+-----------+
| **************** | localhost |
| **************** | localhost |
| **************** | localhost |
| username         | localhost |
| **************** | localhost |
+------------------+-----------+
5 rows in set (0.00 sec)

I set permissions to add the user to the database

mysql> GRANT ALL PRIVILEGES ON `databasename`.* TO 'username'@'localhost';
Query OK, 0 rows affected (0.00 sec)

I verified the permissions with this command

mysql> SHOW GRANTS FOR 'username'@'localhost';
+--------------------------------------------------------------------------+
| Grants for [email protected]                                      |
+--------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'username'@'localhost'                       |
| GRANT ALL PRIVILEGES ON `databasename`.* TO 'username'@'localhost' |
+--------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Finally I flushed permissions

mysql> FLUSH PRIVILEGES;

Now the databases is ready for Nextcloud

10. Nixstats

If you do not know what Nixstat’s is check out my post here Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc

I logged into Nixstats and click Add Server. I ran the provided install command.

wget -q -N --no-check-certificate https://nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ################## ##########################

Todo: Configure Nixstats PHP-FPM and NGINX Reporting (work in progress). My firewall rules are too tight for this install.

Handy Links

  • Monitoring Nginx with Nixstats
  • https://help.nixstats.com/en/article/monitoring-php-fpm-1tlyur6/

11. CronTab Updates

I created a update.sh file that I can call from a crontab entry to update Ubuntu and other software every xx hours.

I added this to my crontab.

12. Misc Security Stuff

I made sure my firewall only allowed traffic to my server was from Cloudflare IP’s and Whitelisted IP’s

Cloud flare IP’s can be found here.

https://www.cloudflare.com/ips-v4/
https://www.cloudflare.com/ips-v6/

At the time of writing the IP’s are 

173.245.48.0/20
103.21.244.0/22
103.22.200.0/22
103.31.4.0/22
141.101.64.0/18
108.162.192.0/18
190.93.240.0/20
188.114.96.0/20
197.234.240.0/22
198.41.128.0/17
162.158.0.0/15
104.16.0.0/12
172.64.0.0/13
131.0.72.0/22
2400:cb00::/32
2606:4700::/32
2803:f800::/32
2405:b500::/32
2405:8100::/32
2a06:98c0::/29
2c0f:f248::/32

I blocked access to my webserver (port 80 and 443) to anyone but Cloudflare.

I whitelisted DNS traffic to only Up Cloud. Thanks, Lon.

Up Cloud support is awesome.

UpCloud Support

Installing NextCloud

Finally I can Install Nextcloud, I navigated to https://nextcloud.com/install/ and clicked Download for Server

Download Nextcloud

I will use the Web installer to Install Nextcloud.

Web Installer Tab

Nextcloud web installer instructions

Setup Instructions

Snip about the Nextcloud Installer from the download page

The Web Installer is the easiest way to install Nextcloud on a web space. It checks the dependencies, downloads Nextcloud from the official server, unpacks it with the right permissions and the right user account. Finally, you will be redirected to the Nextcloud installer.

1) Right-click here and save the file to your computer
2) Upload setup-nextcloud.php to your web space
3) Point your web browser to setup-nextcloud.php on your webspace
4) Follow the instructions and configure Nextcloud
5) Login to your newly created Nextcloud instance!

You can find further instructions in the Nextcloud Admin Manual.

Note that the installer uses the same Nextcloud version as available for the built in updater in Nextcloud. After a major release it can take up to a month before it becomes available through the web installer and the updater. This is done to spread the deployment of new major releases out over time.

I used WinSCP to upload the setup-nextcloud.php to my Nginx web root  folder

WinSCP uploading

I loaded the setup-nextcloud.php file from, my web browser.

Loading setup-nextcloud.php

I entered “.” to install Nextcloud to the website root.

Install Next cloud to .

There is no way Nextcloud installed in 2 seconds, I checked the size of the disk usage in my website root.

sudo du -hs /web-root
313M

Nextcloud took about 10 seconds to download 313MB onto my UpCloud Server.

Fyi: I installed the SpeedTest CLI app and ran a benchmark and UpCloud Chicago can download as 937Mbps/sec and UpCloud Singapore can download at 717Mbps/sec. 

Nextcloud is installed.

Now I need to enter the data root folder for Nextcloud . I installed lswh to be able to see my 500GB disk.

sudo apt-get install lshw

I ran the following to see my disks

sudo lshw -class disk -short
H/W path        Device     Class      Description
=================================================
**********      /dev/vda   disk       26GB Virtual I/O device
**********      /dev/vdb   disk       536GB Virtual I/O device

I formatted my disk

sudo mkfs.ext4 /dev/vdb

I created a new folder under mount to connect to the partition. The folder name is a made-up sample

sudo mkdir -p /mnt/foldername

I mounted the partition to the folder

sudo mount /dev/vdb /mnt/foldername

I made sure Nginx can access the folder

sudo chown -R www-data:www-data /mnt/foldername

I changed to the partition mount

cd /mnt/foldername

I created a test 490GB file

fallocate -l 490G test.file

I checked the file

ls -al
-rw-r--r-- 1 username useername     526133493760 Jun  9 19:38 test.file

I deleted this test file and set this mount point as the data file in Nextcloud setup.

I added a new Nextcloud admin username and password,  mount folder for Nextcloud data folder, the SQL database user/password/database name and host and clicked Finish Setup

Nextcloud details

Nextcloud was setup.

Misc Setup

I ran the /settings/admin/overview report to see if I needed to perform andy final setup steps. I have a few missing php modules and a few optimisation tasks that need resolving.

Links to resolve.

  • Path Fixes
  • PHP Memory Limit
  • PHP Server Tuning

Nextcloud External Security Scan

I loaded https://scan.nextcloud.com/ to perform a external security scan.

Security Scan

Scan Results

All good so far.

Adding Two-Factor Authentication (YubiKeys)

I noticed in the Nextcloud security setting page I can setup a YubiKey as a pass-wordless  login device.

Web AuthN device

This would allow me to insert my YubiKey to login automatically

Auto login.

I added my YubiKey and gave it a name.

Name a YubiKey

The password-less login method is a bit insecure as anyone that has my YubiKey can access my site.

I think I will set up a Two-Factor Authentication/OTP login method and link that to my YubiKey.  I visited the /settings/apps/security page and installed the Two-Factor TOTP Provider app.

Install the OTP App
Install; the OTL app

I clicked the checkbox next to TOTP

Enable TOPT

The app generated a QR code that my YubiCo Authentication App can use to link to Nectcloud

I verified the QR scan and entered the 6 number verification code from my YubiCo Authenticator app

Scan the QR Code

Two Factor logins are now enabled.

2FA Enabled/

Now after I log in I have to enter a temporary 6 digit number that is only valid for 30 seconds (and only after entering my YubiCo YubiKey into my PC and entering its password)

2FA enabled at login/

Nice

Nextcloud Overview

I logged into Nextcloud and was greeted with a wizard.

Welcome screen

The sample images in the welcome screen are a bit small.

welcome screen summary

I can add native apps to Windows, Mac, iOS and Android or I can log in via the web page.

App downalod options

Pointers to the manual, community help and forums.

Help options

Main screen is clean.

Main Screen

A user context Menu is linked in the top right.

Drop down menu.

I setup email alerts (I allowed outgoing ports in my firewall)

sudo ufw allow out 465/tcp
sudo ufw allow out 465/udb

I used my GSuite account to send emails.

email settings

Syncing Files from my PC to Nextcloud

I tried uploading my 150GB Acronis Backup image file to Nextcloud by the web interface but this will fail for sure, this will take many hours.

Acronis image uploading.

I decided to configure Acronis True Image to split backups into 100MB chunks.

100GB file sizes

I created another Acronis image of my Windows Drive.

Nextcloud Windows App

I visited https://nextcloud.com/install/ and installed the Nextcloud Windows app to sync files.

Download windows app

I clicked Windows

Windows Download

Click Next

Click Next

Click Next

Click Next

Click Install

Click Install

Nextcloud sync app is now installing

Installing Wizard

Next cloud sync is now installed.

Run Nextcloud

Click Log in

Login Screen

Enter your Nextcloud server https address and click Next

Enter https server

A web browser login screen appeared and I logged in 

Login to the web app.

After I logged in Nextcloud sync was connected

Sync Connected

I was prompted to sync everything online to my local PC or choose folders to Sync .

Sync File dialog.

All files that were in Nextcloud synced down (that I selected)

Nextcloud sync

I set Nextcloud to start at Windows start.

Start at startup.

I reviewed Download and Upload limits

I decided to add my U:\AcronisBackup folder to my Nextcloud server.

U:\AcronisBackup added rto sync

I was asked to add this to a remote Nextcloud folder.

add to destination folder dialog

Files were backing up.

I has 150GB of Acronis backup files backing up.

I could see each 100MB section of the Acronis Backup appearing in the Nextcloud web app.

Nextcloud Web site

I noticed that the raw file system list of files was about 30 seconds ahead of the web list.

ls -al list of the file system

I had an Alert from my Acronis Backup software that new backup files were downloading.

The Acronis backup folder started backing up but I noticed it was redownloading to a new folder.  I don’t want this.

I allowed Nextcloud to access backup files

I paused the Nextloud Sync and my 150GB Backup was re-downloading to a new folder.

pause backup

It looks like U:\AcronisBackup was backing up then downloading to U:\Nextcloud\Simon\AcronisBackup.

File dialog

I moved my Acronis backup from U:\AcronisBackup to U:\Nextcloud\Simon\ZENigma (ZENnigma is the name of my PC)

I moved my 150GB backup files into Nextcloud folder/

I deleted the old sync of U:\AcronisBackup and started the Nextcloud Sync again

Sync restarted

Now my Acronis backup (150GB) was backing up to Nextcloud.

Backup working

It took 24 hours to backup 150GB from my PC to my server in Singapore.

I can see a handy summary of synced files and disk space used/free.

Done

I can control the sync with a System Tray App.

Sys Tray APp

Nextcloud Conclusion

Pros

  • Free
  • Works well.
  • I have an offsite location for backups and an area for file sharing with my family
  • Faster than Backblaze and Dropbox

Cons

  • Needs better Hardware 2FA support
  • Some Nextcloud web pages are not mobile-friendly (e.g add new user)
  • Needs better post install security checks
  • Web view of files could be updated more often, there is as 30-second delay between the web list of files and a CLI list in Putty of /mnt/foldername/username/files/

Troubleshooting

NGINX website is not loading

Check to see if a package has downloaded apache (this will take out Nginx).

Also, make sure you have set permissions on the folder that holds your SSL Certificates and allow your Nginx www-data user read access.

sudo chown -R www-data:www-data /etc/nginx/https-cert/

Deleting a MySQL Database

I had an issue where Nextcloud did not like the database I created so I ran the following to revoke the database users permissions, remove the user and I deleted the database.

Command to revoke the users MySQL permissions

sudo mysql -u root -p
*************************************
mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'databaseusername'@'localhost';

Delete the MySQL user

sudo mysql -u root -p
Enter password: *************************************
mysql> DROP USER 'databaseusername'@'localhost';

I reset flushed permissions

sudo mysql -u root -p
Enter password: *************************************
mysql> 
FLUSH PRIVILEGES;

To delete the database run the following.

mysqladmin -u root -p drop databasename
Enter password: *************************************
Dropping the database is potentially a very bad thing to do.
Any data stored in the database will be destroyed.

Do you really want to drop the 'databasename' database [y/N] y
Database "databasename" dropped

Thanks for Reading

Fyi: If you don’t have a favourite virtual machine provider you can use my referral link to obtain $25 free credit (only if you are new to UpCloud). Every new user who signs up with my referral link will receive a $25 bonus to get started. That’s 5 months free server (1 CPU and 1GB memory Linux server) 

v1.1

Filed Under: 2nd Factor, Backblaze, Backup, Database, Domain, Google, Nextcloud, Putty, SSH, UpCloud Tagged With: backblaze, Dropbox, Google One, Nextcloud

Backing up files to a Backblaze B2 Cloud Bucket with Duplicati

June 6, 2019 by Simon

This guide will show how you can automatically backup files to a Backblaze B2 clouds bucket with the Duplicati software on Windows and Linux.

I have blogged about how you can use the Backblaze personal backup program ( here, Use my link and get your first month of Backblaze backups free). The personal application from Backblaze is cool but what if you wanted to store 10GB for free (the first 10GB is free) or setup different buckets for different computers or applications.

Welcome to Backblaze B2 Cloud Storage.

In Australia B2 cloud costs (+GST).

  • First 10GB is free.
  • $0.005c per GB stored per month.
  • To download, 1x GB costs $0.01c.

Check our my other related posts

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways

This works alongside the private Backblaze backup mentioned here.

Screenshot: Duplicati Downalods page https://www.duplicati.com/download

Backblaze B2 Pricing

How does Backblaze B2 Compare that to other cloud data storage provides.

Screenshot of https://www.backblaze.com/b2/cloud-storage.html

Comparison chart.

ProviderStorage ($/GB/Month)Download ($/GB)
Backblaze$0.005$0.01
Amazon S3$0.021$0.05+
Microsoft Azure$0.018+$0.05+
Google cloud$0.020$0.08

Vendor cost breakdown (source).

Read here to view the official page on pricing.

Snip:

Storage:

The first 10 GB of total storage is free, and is $0.005 per GB per month for additional storage beyond 10 GB.   

Download:

The first 1 GB of downloads per day are free.  The price to download stored data costs $0.01 per GB beyond the free daily 1 GB. 

Transactions:

A 'transaction' refers to any time an api call is made with your account, such as listing the contents of your bucket or downloading a file.   B2 will charge for some transactions however, for the average user, these charges are largely negligible unless you are making an extremely high volume of api calls.

Class A transactions are free.

The first 2,500 Class B transactions are free each day. Additional Class B API calls are charged at $0.004 per 10,000 calls.  

The first 2,500 Class C transactions are free each day. Additional Class C API calls are charged is $0.004 per 1,000 calls.

Vendor Relability

https://downdetector.com is  a great site for viewing is  a vendor is down or has issues, lets compare, Backblaze with Amazon, Azure and Google Cloud

  • Downdetector.com: Backblaze
  • Downdetector.com: Azure
  • Downdetector.com: Amazon
  • Downdetector.com: Google Cloud

Backblaze have has far fewer downtime issues during the great outage of May/June 2019 compared to its competitors. What I like is Backblaze tell us how they do it and tell us what hard drives are good or bad.

Creating a Backblaze B2 Account

Login/SignUp to backblaze at https://secure.backblaze.com/user_signin.htm

You can have a sneak peek at the personal Backblaze backup signup steps in this guide (why not take up a free 1 month back free by clicking this link when you create an account).

From your account page click billing and add your account details

Screenshot of https://secure.backblaze.com/billing.htm?billing_page=b2

Read the B2 Strater Guide: https://www.backblaze.com/b2/docs/

Also review your review the following under ‘My Settings‘

  • Phone Numer (mobile) for usage alerts.
  • Two Factor Authorization for extra security (don’t forget to çopy your backup codes)

Setup Billing Alerts (just in case)

In the Backblaze portal, under ‘Caps and Alerts‘ you should set..

  • Alerts to your mobile phone (for Daily storage caps,bandwidth caps and class B/C caps)
  • Under ‘Daily Storage Caps’ click ‘edit’ and set monetary alerts (you can increase alert limits later)
Added $5 and $10 daily storage alerts

Click ‘Edit Caps’ and set daily limits on all categories

Edit Caps button

I set alert limits of 10c (I can increase this later)

Daily 10c alerts should see is costs creep above $3 a month.

Screenshot of cap alerts, I set 10c warnings on everyitng

Create a cloud bucket to store files

Go to https://secure.backblaze.com/b2_buckets.htm and click ‘Create Bucket’

Screenshot of https://secure.backblaze.com/b2_buckets.htm (create bucket button)

You can create upto 100 buckets per account.

I created a descriptive bucket name that reminded me what server, drive and folder this was used to back up.

"#######-drive-backup-folder" (####### was the name of my computer)
Screenshot of bucket being created an named.

I created a number of buckets (all private).

The Backblaze portal lists all of my new storage buckets.

Screenshot of my 4 new storage buckets

Bucket Storage (Lifestyle Settings)

You can click the ‘Lifecycle Settings‘ button and set the desired buckets file retention policy (overwrite or keep all versions of a file etc).

Read the official Lifestyle settings faq here.

The default is to keep all versions of a file. Choose a setting that is required for each bucket(for me ‘Keep only the latest version of the file’ is best and lowers potential cost blowouts. Having all versions of a file may be handy for a Documents folder.

Screenshot of  Lifestyle settings at https://secure.backblaze.com/b2_buckets.htm

You could opt to keep all versions of a file (forever) but that would be expensive.

You tell Backblaze to delete older versions of a file after xx days (this is a good option)

You can also setup custom lifecycle rules to hide a file with a certain prefix after xx days then delete after xx days.

Create a Master Application Key for all buckets

You will need to create a Master Application Key that will grant you full access (Permissions: listKeys, writeKeys, deleteKeys, listBuckets, writeBuckets, deleteBuckets, listFiles, readFiles, shareFiles, writeFiles, deleteFiles ) to each bucket.

Snip from: https://secure.backblaze.com/app_keys.htm

Application keys are used as a pair: Key ID and Application Key. This allows B2 to communicate securely with different devices or apps. Once you generate your Master Application Key, this key has full capabilities. Create your own Application Keys to limit features like read/write. Learn more.

Do write down this “KeyID” and “applicationKey”

Screenshot of https://secure.backblaze.com/app_keys.htm (generated "KeyID" and "applicationKey")

Don’t lose the Master Application Key.

Create an App Key for each bucket

A master key is like an administrator account, its great for complete access but you should create a separate key per bucket for better security.

Back under the App Keys you can click “Add a New Application Key” for each bucket.

Screenshot of Add a new application key button

Read the official guide on Application Keys here

Screenshot of add application key (add name, option to choose a bucket and assign read/write acess

Do Set..

  • “Name of Key” (similar to your bucket (e.g “my-bucket-name-key”))
  • “Allow access to Bucket(s)” (choose a bucket to grant access to).
  • “Type of Access” (Read and Write, Read Only, Write Only)

Optional..

  • Set a “File name prefix” (snip from here)
"The file name prefix of files the download authorization token will allow b2_download_file_by_name to access. For example, if you have a private bucket named "photos" and generate a download authorization token for the fileNamePrefix "pets/" you will be able to use the download authorization token to access: https://f345.backblazeb2.com/file/photos/pets/kitten.jpg but not: https://f345.backblazeb2.com/file/photos/vacation.jpg."
  • Set a “Duration” (snip from here)
"The number of seconds before the authorization token will expire. The minimum value is 1 second. The maximum value is 604800 which is one week in seconds."

I now has 1x Master Application key and 4x Application Keys for my 4x Buckets

Screenshot of https://secure.backblaze.com/app_keys.htm (1 master key and 4x app keys)

TIP: Make sure you save the keys, if you did not grab the keys delete the keys and buckets (if there is not data and try again). If you don;t have the keys you are locked out of the buckets

OK, Let’s fill those buckets with data with Duplicati

Go to https://www.duplicati.com/ and familiarise yourself with the site

Setup Duplicati on Windows (10)

Go to https://www.duplicati.com/download and download the latest Windows client. At the time of writing Duplicati 2.0.4.5 beta was the latest version. Thew instal was 1MB (the download serve was not that fast)

Warning: Beta software may have bugs, feel free to check out the Github page for Issues etc.

Duplicati 2.0.4.5 download button

Open the install and Click ‘Next‘

Duplicati install wizard start, click Next

Agree to the ‘Licence Agreement‘ by clicking ‘Next‘

Duplicati licence agreement screenshot

Select Duplicati components to install and click ‘Next‘

Install an components screenshot.

Tick ‘Launch Duplicati‘ an click ‘Finish‘

Duplicati has finished installing screenshot.

Using Duplicati Software

Upon first open you will be prompted to set a password (do this even if you are the only user of thew computer)

First run screenshot

Set a strong and unique password

Screenshot of setting a duplicati password

Click ‘Ok‘

Ok Button

When the page reloads it will report it is not logged in.

Screenshot of user is not logged in error.

Enter your password

Enter password screenshot

The main screen of Duplicati reports that there are no scheduled set.

Screenshot no scheduled tasks screenshot.

OK, lets create a backup schedule.

Creating a Backup on Windows

If your screen is small you will need to click the menu at the top right and click ‘Add Backup’, if your screen is large there will be an ‘Add Backup’ menu on the left hand side of the screen.

Duplicati menu screenshot

Select ‘Configure a new backup‘ and click ‘Next‘

Configure a new backup screenshot

You will be able to name and describe the backup schedule name and be able to set up an encryption password.

Then click ‘Next‘

Screenshot of create backup schedule with name, description and encryption password

Under ‘Storage Type‘ choose ‘B2 Cloud Storage‘

When you choosing ‘B2 Cloud Storage‘ you will be directed to the screen below where you stecify..

  • B2 Cloud bucket name
  • Path in the bucket to store the files (leave blank)
  • Enter your Backblaze Account ID (e.g. ##########################)
  • Enter the Buckey ApplicationKey (e.g. ###############################)

Click ‘Next‘

Screenshot of enter B2 credentials

I entered my credentials

I entered my B2 bucket credentials

Click ‘Test Connection‘, if you entered everything OK you should see ‘Connection Worked‘

Connection worked Screenshot

Click ‘Next‘

Next button screenshot

Now you can select the folders to backup on your local computer

Duplicati, select folders to backup.

Click ‘Next‘

Next button

Now you can choose a schedule to backup

Screenshot choose time and days to run the backup.

Click ‘Next‘

Click Next button screenshot

Now you can set the remote backup size and backup retention mode.

Read the official guide on setting the volume size here

  • Increase the ‘Remote Volume Size’ if you are worried about consuming your daily API quota in Backblaze (this may happen if your upload speed is fast and your backup is very large).
  • Lower the ‘Remote Volume Size’ size if backups are small or your upload speed is low.
Screenshot of remote volume size and backup retention

Choose a desirable backup retention mode. Duplicati only offers full backups (not incremental or differential backups). This will be an issue if your backups are large.

  • Keep all backups (“Nothing will be deleted. The backup size will grow with each change.“)
  • Delete backups that are older than (“If at least one newer backup is found, all backups older than this date are deleted.“)
  • Keep a specific number of backups (“Once there are more backups than the specified number, the oldest backups are deleted.“)
  • Smart Backup Retention (“Over time backups will be deleted automatically. There will remain one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup.“)
  • Custom backup retention (“Enter a retention strategy manually. Placeholders are D/W/Y for days/weeks/years and U for unlimited. The syntax is: 7D:1D,4W:1W,36M:1M. This example keeps one backup for each of the next 7 days, one for each of the next 4 weeks, and one for each of the next 36 months. This can also be written as 1W:1D,1M:1W,3Y:1M.“)

Each storage option has pro’s and cons with potential cost implications. I would not set 300GB to “Keep all backups” and backup daily. That would be expensive.

Screenshot of the 5 backup retention modes

Click ‘Save‘

Save button screenshot

TIP: Don”t forget to save your encryption key if you set one

Screenshot of a duplicati warning saying have we saved our encryption ley

Now you will see the backup schedule on your Duplicati screen.

You can run, edit or delete the backup status here.

Screenshot of the backup schedule on the home screen.

I clicked ‘Run‘ and the backup was under way.

A progress bar updates as the backup happens.

Screenshot of backup underway

A slight delay happens at the end of the backup.

Screenshot of backup waiting to finish screen.

I logged into the Backblaze portal to see how many files and usage credits were used.

Note it seems?

Screenshot of backblaze portal

When the files are backed up a verification processes happens.

screenshot of verify files screen

Now the backup has completed.

Screenshot of the backup completed screen

I waited 2 minutes and finally Backblaze updated it’s status, Duplicati used 3 files and 87MB

Screenshot of B2 Cloud uploads stats

I logged into the view the Duplicati files that were uploaded and I could see the three files were there and they were encrypted.

Screenshot of B2 cloud encrypted Duplicati files.

Awesome

Restoring a Backup on Windows

Having a backup is useless you can restore the files (especially when they are encrypted).

Lets restore a few files and see if it works.

I opened Duplicati and clicked ‘Restore‘

Screenshot: Duplicati restore menu item

I clicked the bottom option that listed my bucket name/backup name and clicked ‘Next‘

Screenshot restore from the backup

Before I ran the restore I deleted some location files that were just backed up.

Screenshot of me deleting 2 files.

Duplicati now asked me to choose files to restore.

Screenshot restore files treeview with selections

I chose the original location to restore to and clicked ‘Restore‘

Screenshot restore to.

Duplicati took a few minutes to restore the files.

Screenshot: Duplicati downloading files

The restore was a success.

Screenshot restore success

Nice

Backblaze Costs?

Beware, make sure you calculate the costs to upload and download files.

Do set alarms on cap’s limits and review them often.

Screenshot of data usage and caps.

Second Backup Schedule on Windows

I quickly set a second backup schedule of a different folder to a different bucket.

This backup was 300GB so I will only backup once a month

Schedule once a month backup schedule.

The backup is listed on the front screen of Duplicati.

Do create multiple buckets and set multiple backup schedules to break up the backups into smaller chunks.

Screenshot of multiple backup schedules.

Setup Duplicati on Linux (Debian or Ubuntu)

Lets setup B2 Cloud Storage backup on Linux.

I visited the Duplicati forums here. Duplicati informed me that the Linux client needs mono installed

Duplicati depends on other software. For Windows, Microsoft .NET Framework 4.5 or higher needs to be installed. Linux and Mac OS X require Mono to be installed.

Mono is a Cross Platform open source .NET that runs on Linux (read here)

On Ubuntu 1.08 I installed mono with these commands

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update
sudo apt install mono-devel gtk-sharp2
sudo apt-get install libmono-2.0-1

On Debian 9.9 I installed Mono with these commands (from here)

sudo apt install apt-transport-https dirmngr gnupg ca-certificates
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update
sudo apt install mono-devel gtk-sharp2
sudo apt-get install libmono-2.0-1

I rebooted the serves and the CPU and memory went crazy after I installed Mono. My Nixstats Service reports were going off telling me my servers CPU was busy.

If you don’s have reports on Linux server CPU/MEM/Disk reports coming to you via Email and Telegram then read this.

Screensot of Nixstats Reports

I felt sorry for my web and database server server(s) so I rebooted them, wow, My website came back up in about 10 seconds thanks to UpClouds fast Max IOPS disks.

Installing Duplicati on Linux

I visited the download Duplicati page here and copied the Ubuntu and Debian download URL.

At the time of writing it was..

https://updates.duplicati.com/beta/duplicati_2.0.4.5-1_all.deb

I downloaded the deb package

cd /utils/duplicati
wget https://updates.duplicati.com/beta/duplicati_2.0.4.5-1_all.deb

I then ran this command to install the package

sudo dpkg -i duplicati_2.0.4.5-1_all.deb

I received this error even thpogh libmono-2.0-1 was installed

dpkg: dependency problems prevent configuration of duplicati:
duplicati depends on libmono-2.0-1; however:
Package libmono-2.0-1 is not installed.

This forum post helped me, I ran

sudo apt install -f

Output

sudo apt install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4
  libindicator7 libmono-2.0-1
Suggested packages:
  indicator-application
The following NEW packages will be installed:
  libappindicator0.1-cil libappindicator1 libdbusmenu-glib4 libdbusmenu-gtk4
  libindicator7 libmono-2.0-1
0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.
1 not fully installed or removed.
Need to get 143 kB of archives.
After this operation, 541 kB of additional disk space will be used.
Do you want to continue? [Y/n]

And I ran the following

sudo apt dist-upgrade --fix-missing

I now ran the following to install the duplicati package.

sudo dpkg -i duplicati_2.0.4.5-1_all.deb

It worked on Ubuntu 18.08 bt not Debian 9.9

Debian produced the following error

sudo dpkg -i duplicati_2.0.4.5-1_all.deb                (Reading database ... 70082 files and directories currently installed.)
Preparing to unpack duplicati_2.0.4.5-1_all.deb ...
Unpacking duplicati (2.0.4.5-1) over (2.0.4.5-1) ...
dpkg: dependency problems prevent configuration of duplicati:
 duplicati depends on libappindicator0.1-cil | libappindicator3-0.1-cil; however:
  Package libappindicator0.1-cil is not installed.
  Package libappindicator3-0.1-cil is not installed.

dpkg: error processing package duplicati (--install):
 dependency problems - leaving unconfigured
Processing triggers for mime-support (3.60) ...
Errors were encountered while processing:
 duplicati

On Debian I installed the stretch version of the package

sudo apt-get install libappindicator1

Then in Debian I fixed broken packages

sudo apt --fix-broken install

This I was able to install Duplicati on Debian

sudo dpkg -i duplicati_2.0.4.5-1_all.deb

Running Duplicati CLI

I can now Run Duplicati CLI (it works)

duplicati-cli

See duplicati.commandline.exe help <topic> for more information.
  General: example, changelog
  Commands: backup, find, restore, delete, compact, test, compare, purge, vacuum
  Repair: repair, affected, list-broken-files, purge-broken-files
  Debug: debug, logging, create-report, test-filters, system-info, send-mail
  Targets: aftp, amzcd, azure, b2, box, cloudfiles, dropbox, ftp, file,
  googledrive, gcs, hubic, jottacloud, mega, msgroup, onedrive, onedrivev2,
  sharepoint, openstack, rclone, s3, ssh, od4b, mssp, sia, tahoe, webdav
  Modules: aes, gpg, zip, 7z, console-password-input, mssql-options,
  hyperv-options, http-options, sendhttp, sendmail, runscript, sendxmpp,
  check-mono-ssl
  Formats: date, time, size, encryption, compression
  Advanced: mail, advanced, returncodes, filter, filter-groups, <option>

http://www.duplicati.com/              Version:  - 2.0.4.5_beta_2018-11-28

TIP: Don’t run “duplicati” that’s the desktop verison.

Creating a Duplicati Backup on Linux

I read this page and this page to get an understanding of how to back up from the command line. I was not sure if I needed to run a service or run the CLI from a bach script so I asked on the forums here.

Within minutes I had this reply

kenkendk
1h
Hi @feartec, welcome to the forum!

The CLI and Web-UI are not the same, and operate independently.

If you use the CLI, you need to use cron or something similar to run the backups periodically. You can use duplicati-cli help to get started. You do not need the server running for this.

For the web-UI we do not currently have any tools for manipulating the backups other than the main WebUI itself. You need to somehow forward/tunnel access to the server, and then access the WebUI from a local browser.

Backing up a folder with Duplicati to a Backblaze B2 Bucket

I created this folder structure

/test/
/test/testfile.txt < hello
/test/subfolder/
/test/subfolder/file.txt < world

I ran this command to back up the folder structure above to backblaze (official guide). See advanced switches.

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=**********

TIP: read more about “dblock” size here

Output

Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  4 files need to be examined (8 bytes)
Checking remote backup ...
  Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
  Downloading file (1021 bytes) ...
  Downloading file (1.06 KB) ...
  Downloading file (2.00 KB) ...
  0 files need to be examined (0 bytes)
  Duration of backup: 00:00:06
  Remote files: 3
  Remote size: 4.05 KB
  Total remote quota: 0 bytes
  Available remote quota: 0 bytes
  Files added: 0
  Files deleted: 0
  Files changed: 0
  Data uploaded: 0 bytes
  Data downloaded: 4.05 KB
Backup completed successfully!

I logged into Backblaze to check the backups

Duplicati backup success

I created a second test file and added 2 test files

mkdir /test2
cd /test2
wget http://ipv4.download.thinkbroadband.com/10MB.zip
wget http://ipv4.download.thinkbroadband.com/5MB.zip

ls -al
total 15368
...
-rw-r--r--  1 user user 10485760 Jun  3  2008 10MB.zip
-rw-r--r--  1 user user 5242880 Jun  3  2008 5MB.zip

I ran the backup with the second folder in the command line

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=********** --retention-policy="1W:1D,4W:1W,12M:1M"

Output

Backup started at 06/07/2019 23:05:58
Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  7 files need to be examined (15.00 MB)
  Uploading file (15.03 MB) ...
  Uploading file (11.43 KB) ...
  Uploading file (1.29 KB) ...
Checking remote backup ...
  Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
  Downloading file (1.29 KB) ...
  2 files need to be examined (8 bytes)
  Downloading file (11.43 KB) ...
  Downloading file (15.03 MB) ...
  0 files need to be examined (0 bytes)
  Duration of backup: 00:00:10
  Remote files: 6
  Remote size: 15.05 MB
  Total remote quota: 0 bytes
  Available remote quota: 0 bytes
  Files added: 2
  Files deleted: 0
  Files changed: 0
  Data uploaded: 15.05 MB
  Data downloaded: 15.05 MB

Success

Restoring a Duplicati Backup on Linux

O’k lets delete the local “/test2/” folder then restore it

rm -R /test2

Now lets restore it (official guide)

/usr/bin/duplicati-cli restore "b2://bucket-name-goes-here/test" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --passphrase=************* --overwrite=true --restore-path="/test2"

Output

Restore started at 06/07/2019 23:21:51
Checking remote backup ...
  Listing remote folder ...
Checking existing target files ...
  2 files need to be restored (15.00 MB)
Scanning local files for needed data ...
  Downloading file (15.03 MB) ...
  0 files need to be restored (0 bytes)
Verifying restored files ...
Restored 0 (0 bytes) files to /test2
Duration of restore: 00:00:05

Did it work

cd /test2
ls -al
total 15368
...
-rw-r--r--  1 use user 10485760 Jun  3  2008 10MB.zip
-rw-r--r--  1 user user 5242880 Jun  3  2008 5MB.zip

Yes it worked

Backup 2 folders an emailing the results

I use Google GSuite for email and have an app password setup just for duplicati. Here is the official guide for sending an email after a backup operation

I downloaded a 100MB file

cd /test2
wget http://ipv4.download.thinkbroadband.com/100MB.zip

I ran this command to backup an emailthe progress

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=***************** [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"

Output

Backup started at 06/08/2019 00:02:28
Checking remote backup …
Listing remote folder …
Scanning local files …
8 files need to be examined (115.00 MB)
7 files need to be examined (15.00 MB)
Uploading file (49.99 MB) …
Uploading file (18.01 KB) …
Uploading file (49.99 MB) …
Uploading file (17.97 KB) …
Uploading file (233.33 KB) …
Uploading file (33.17 KB) …
Uploading file (1.42 KB) …
Compacting remote backup …
Checking remote backup …
Listing remote folder …
Verifying remote backup …
Remote backup verification completed
Downloading file (1.42 KB) …
4 files need to be examined (15.00 MB)
Downloading file (18.01 KB) …
Downloading file (49.99 MB) …
0 files need to be examined (0 bytes)
Duration of backup: 00:00:15
Remote files: 13
Remote size: 115.33 MB
Total remote quota: 0 bytes
Available remote quota: 0 bytes
Files added: 1
Files deleted: 0
Files changed: 0
Data uploaded: 100.28 MB
Data downloaded: 50.01 MB
Backup completed successfully!

Email

Screenshot of and email from Duplicati software

Backup Types and Versions

Read up here on backup retention policies here

TIP: Use the desktop Duplicati software to generate a backup then use the export to command line option to get the command line parameters to use in the command line version.

Screenshot of the desktop version of Duplicati and the export to command line link

After you click Commandline you can view each separate command liene option and parameter.

This was handy for finding this backup retention parameter

--retention-policy="1W:1D,4W:1W,12M:1M

“This will give me “one backup for each of the last 7 days, each of the last 4 weeks, each of the last 12 months. There will always be at least one remaining backup. “

Setup Auto Backup Cron Scripts

Login to your linux machines and edit a bash file (e.g in “/scripts/backupnow.sh” folder).

Make the script executable

chmod +x /scripts/backupnow.sh

Add the following

#!/bin/bash

/usr/bin/duplicati-cli backup "b2://bucket-name-goes-here/test" "/test/" "/test2/" --auth-username="#########keyID#########" --auth-password="###########applicationKey#########" --compression-module="zip" --dblock-size="50mb" --passphrase=123 [email protected] --send-mail-subject="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-body="Duplicati Backup on yourserver.com - Src: %LOCALPATH% , Dest: %REMOTEURL% (%RESULT%)" --send-mail-url="tls://smtp.gmail.com:587" [email protected] --send-mail-password=***************** --retention-policy="1W:1D,4W:1W,12M:1M"

Now you can add the script to your crontab by running the following.

crontab -e

Make sure you have the following in your crontab

SHELL=/bin/bash

Add a cron entry to run the bash script

#Backup Database
0 22 * * * /bin/bash /scripts/backupnow.sh

Use https://crontab.guru/ to check the time you want to run thew script

https://crontab.guru/ screenshot

Links

  • B2 Starter Guides
  • Duplicati Manual

 

 

 

Version

v1.1 Fixed typo (”downalod” thanks to @NicolasWolf for reporting.)

v1.0 Initial Post (added retention polity and Linux crontabs

v0.95 Backup, Restore and Send Email

v0.91 Added Duplicati service info

v0.9 Working Copy

Filed Under: Uncategorized Tagged With: B2, backblaze, Backup, cli, cloud, duplicati, Linux, windows

Backing up your computer automatically with BackBlaze software (no data limit)

June 2, 2019 by Simon

Backblaze ( https://www.backblaze.com/ ) is an awesome company who not only create a mega reliable backup infrastructure but they also tell us how they do it and tell us what hard drives are good or bad.

Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj

They are so confident and experienced that they offer unlimited backups for $6.60 a month here in Australia. They state they have 750 Petabytes stored on their infrastructure & have restored over 40 billion files to customers.

Check our my other related posts

  • Backing up files to a Backblaze B2 Cloud Bucket with Duplicati
  • How to back up an iPhone (including photos and videos) multiple ways

Main features of Backblaze.

  • No limit to files that you back up (number or size or files)
  • Web-based file recovery (if needed)
  • They will mail you a hard drive if you need to restore large amounts of files.
  • Ability to locate lost or stolen computers.
  • Mobile app access your backups.
  • Business options for large computer fleets.

Creating a Back Blaze Account

Go to backblaze and create an account.

Login to backblaze at https://secure.backblaze.com/user_signin.htm

Screenshot Login.

Choose a Backblaze Plan

Go to: https://secure.backblaze.com/buy_plan.htm to choose a backup and payment plan. Use my link and get your first month free.

Australian Prices below (add +10% GST).

  • $6 AUD a month
  • or $60 for 12 Months
  • or $110 for 24 Months
Screenshot $6 AUD a month or $60 for 12 Months or $110 for 24 Months

Choose a payment plan

Pay by the month, year or bi year

Enter payment details

Screenshot, add CC or paypal

Click “Buy Backblaze”

Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj

Done, now I can download the client.

Installing the Backblaze Software

  • Windows Client
  • Mac Client

When the download is down you can start the install.

You will need to login to the install to streamline the setup.

Backblaze install screen asking me to login

After you login the installation will begin.

Backblaze is installing and checking what files need backing up.

Backblaze now reports that the install has complete. Backblaze reports that I have 379GB to backup.

Backblaze reports that I have 379GB to backup.

Pressing OK shows Backblaze is already uploading my files.

Also, I have 15 days of free trial before being billed.

Clicking on the backblaze system tray icon reveals a few options

Backblaze system tray icon menu

Backblaze Preferences

Backblaze preferences can be opened by clicking Settings from the main screen.

I..

  • Renamed the computer.
  • Disabled backup on battery power
  • Set upload threads to 5 (then lowering if the network gets busy or when the backup is almost complete)
  • Disabled Automatic throttling and set faster backups
  • I set a private encryption key
I entered and encryption key

There is a 50 char limit on encryption keys

50 char limit on encryption keys message

Warning about not forgetting the encryption key

Message don;t loose the key

All other preferences look ok.

Backblaze preferences

That’s it the backup is uploading automatically and there is nothing else I need to do.

Backup is happily uploading.

Backblaze Portal

If you login to the backblaze portal ( here ) you can view your files (you will need to enter the private encryption keys if you opted to encrypt your files).

Backblaze portal

Initial Backup time

Your initial backup may have a while to upload depending on your backup size and internet connection speed.

I was able to upload 400GB in 3 days on my Internet Plan. Smaller files are uploaded first then larger ones. I have 2 files left to upload totalling 7GB.

Read the official guide on speeding up uploads here.

Backblaze files uploading

One way to make the initial backups faster is to see what files are queues to upload in the Backblaze preferences and then excluding unwanted files and folders.

Backblaze files ro backup queue

I can see my Internet Explorer cache and Google Chrome Temporary Internet File(s) folders are queued to be back up.

Time to exclude these folders in the Backblaze Settings under exclusions

C:\Users\Simon Fearby\AppData\Local\Microsoft\Windows\INetCache\IE
C:\Users\Simon Fearby\AppData\Local\Google\Chrome\User Data\Profile 1\Code Cache\
C:\ProgramData\MySQL\MySQL Server 8.0\Data\#innodb_temp\
C:\ProgramData\USOShared\Logs\
C:\Users\Simon Fearby\AppData\Roaming\Adobe\SLData\SLCache\

Time to exclude these folders.

Backblaze exclude folders

That knocked off a few thousand files to backup 🙂

Restore Files from Backblaze

If you restore files you have a few options

  • Download the files (free)
  • Save files to B2 Cloud (Backblaze)
  • USB Flash Drive
  • USB Hard Drive
restore options. 1) download, 2) save to BS cloud, 3) USB Key, 4) USB Hard Drive

Choosing to download files you are given a treeview to restore files from.

Treeview folder to restore files

I selected a test file to restore (restores are not instant) and was informed I would be emailed when the files are ready to download. Thats cool

Screenshot, restore pending.

I waited 3 minutes and received an email that my files were ready to download.

Email: download ready

I hope this helps someone.

Other Links

Check out my guide on How to back up an iPhone photos and videos blog post here.

Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj

View Official Backblaze guides here.

https://help.backblaze.com/hc/en-us/sections/203997408-B2-Guides

Version

v1.3 Initial Backup section

v1.1 Added free month link

v1.0 Initial Draft

Filed Under: Backblaze, Backup Tagged With: Automatic, backblaze, Backup

Backing up OSX or an Ubuntu server with Backblaze B2 Cloud Storage from the Command Line

March 14, 2018 by Simon

This computer will show you can back up computer or server with Backblaze B2 Cloud Storage from the Command Line n OSX and Ubuntu.

This post is still being written. I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Also, I have blogged about how you can add block storage to a Vultr server, backup and restore snapshots , syncing files with rsync along with using GitHub and Bitbucket but what do you do if you need to backup large amounts of data?

Backblaze has a Cloud storage solution that costs as low as $0.005c a GB (a month), The first 10G is free. Backblaze say “From bytes to petabytes Backblaze B2 is the lowest cost high-performance cloud storage in the world. ”

Back Blaze have open sourced internal drive enclosure designs and drive failure stats and it’s time I gave them a try.

Goto https://www.backblaze.com

Backblaze

Create or sign in.

Backblaze Login

After you login got the dashboard.

B2 Cloud

Click Backblaze B2 Cloud Storage

Activate B2 Cloud

Signup

Create a Bucket

Create Bucket

Name the bucket (long names with a GUID are good).

Name the Bucket

You can rename the bucket here and change public/private and or upload/download files manually.

Manage Bucket

The first thing I did was limit the versions of files under the lifecycle settings for the bucket.

Version Settings

Now I created a series of subfolders to store files from different servers (I could have used many buckets but one bucket will do).

Folders

I can upload files via the Backblaze bucket GUI if I needed to.

Upload and Download

Back Blaze has a command line tool for uploading: https://www.backblaze.com/b2/docs/quick_command_line.html

Install Steps

Backblaze state “The B2 command-line tool is available from the Python Package Index (PyPI) using the standard pip installation tool. Your first step is to make sure that you have either Python 2 (2.6 or later) or Python 3 (3.2 or later) installed.”

I have Python 2.7 installed

python --version
Python 2.7.10

Install PIP

sudo easy_install pip
Password:
Searching for pip
Best match: pip 1.5.6
Processing pip-1.5.6-py2.7.egg
pip 1.5.6 is already the active version in easy-install.pth
Installing pip script to /usr/local/bin
Installing pip2.7 script to /usr/local/bin
Installing pip2 script to /usr/local/bin

Using /Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

I ran into issues updating b2 CLI

sudo pip install --upgrade b2
Requirement already up-to-date: b2 in /Library/Python/2.7/site-packages
Requirement already up-to-date: arrow>=0.8.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: logfury>=0.1.2 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: requests>=2.9.1 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: six>=1.10 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: tqdm>=4.5.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: futures>=3.0.5 in /Library/Python/2.7/site-packages (from b2)
Downloading/unpacking python-dateutil from https://pypi.python.org/packages/bc/c5/3449988d33baca4e9619f49a14e28026399b0a8c32817e28b503923a04ab/python_dateutil-2.7.0-py2.py3-none-any.whl#md5=5a86a548fe776cc079bf4a835473e3f8 (from arrow>=0.8.0->b2)
  Downloading python_dateutil-2.7.0-py2.py3-none-any.whl (207kB): 207kB downloaded
Installing collected packages: python-dateutil
  Found existing installation: python-dateutil 1.5
    Uninstalling python-dateutil:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

I tried installing via the alternative method (with no luck)

git clone https://github.com/Backblaze/B2_Command_Line_Tool.git
Cloning into 'B2_Command_Line_Tool'...
remote: Counting objects: 5084, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 5084 (delta 1), reused 1 (delta 0), pack-reused 5076
Receiving objects: 100% (5084/5084), 1.25 MiB | 689.00 KiB/s, done.
Resolving deltas: 100% (3622/3622), done.
cd B2_Command_Line_Tool/

I tried running the setup script (with no luck)

sudo python setup.py install
setuptools 20.2 or later is required. To fix, try running: pip install "setuptools>=20.2"

Upgrading setup tools also failed

sudo pip install "setuptools>=20.2"
Downloading/unpacking setuptools>=20.2
  Downloading setuptools-38.5.2-py2.py3-none-any.whl (490kB): 490kB downloaded
Installing collected packages: setuptools
  Found existing installation: setuptools 18.5
    Uninstalling setuptools:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

Backing up a Mac via command line with B2

More to come when I can get B2 CLI Installed.

Backing up an Ubuntu machine via command line with B2

More to come when I can get B2 CLI Installed.

Update

My ticket with Backblaze was automatically closed with this note “If the issue is persisting, it may be easiest to map the installation to the user folder, rather than the system level.”

No ideas how but something to research.

Ask a question or the recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

V1.1 ticket closed

v1.0 Initial post

Filed Under: Backup Tagged With: an, B2, backblaze, Backing, cloud, command, from, line, or, OSX, server, storage, the, ubuntu, up, with

Primary Sidebar

Poll

What would you like to see more posts about?
Results

Support this Blog

Create your own server today (support me by using these links

Create your own server on UpCloud here ($25 free credit).

Create your own server on Vultr here.

Create your own server on Digital Ocean here ($10 free credit).

Remember you can install the Runcloud server management dashboard here if you need DevOps help.

Advertisement:

Tags

2FA (9) Advice (17) Analytics (9) App (9) Apple (10) AWS (9) Backup (21) Business (8) CDN (8) Cloud (49) Cloudflare (8) Code (8) Development (26) Digital Ocean (13) DNS (11) Domain (27) Firewall (12) Git (7) Hosting (18) IoT (9) LetsEncrypt (7) Linux (21) Marketing (11) MySQL (24) NGINX (11) NodeJS (11) OS (10) Performance (6) PHP (13) Scalability (12) Scalable (14) Security (45) SEO (7) Server (26) Software (7) SSH (7) ssl (17) Tech Advice (9) Ubuntu (39) Uncategorized (23) UpCloud (12) VM (45) Vultr (24) Website (14) Wordpress (25)

Disclaimer

Terms And Conditions Of Use All content provided on this "www.fearby.com" blog is for informational purposes only. Views are his own and not his employers. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. Never make changes to a live site without backing it up first.

Advertisement:

Footer

Popular

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Add Google AdWords to your WordPress blog

Security

  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • Setting up DNSSEC on a Namecheap domain hosted on UpCloud using CloudFlare
  • Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx
  • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
  • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
  • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
  • Beyond SSL with Content Security Policy, Public Key Pinning etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Run an Ubuntu VM system audit with Lynis
  • Securing Ubuntu in the cloud
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Code

  • How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains
  • Useful Java FX Code I use in a project using IntelliJ IDEA and jdk1.8.0_161.jdk
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider
  • How to setup PHP FPM on demand child workers in PHP 7.x to increase website traffic
  • Installing Android Studio 3 and creating your first Kotlin Android App
  • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
  • How to use Sublime Text editor locally to edit code files on a remote server via SSH
  • Creating your first Java FX app and using the Gluon Scene Builder in the IntelliJ IDEA IDE
  • Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

Tech

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Is OSX Mojave on a 2014 MacBook Pro slower or faster than High Sierra
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • The case of the overheating Mac Book Pro and Occam’s Razor
  • Useful Linux Terminal Commands
  • Useful OSX Terminal Commands
  • Useful Linux Terminal Commands
  • What is the difference between 2D, 3D, 360 Video, AR, AR2D, AR3D, MR, VR and HR?
  • Application scalability on a budget (my journey)
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

Wordpress

  • Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution
  • Setting web push notifications in WordPress with OneSignal
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Wordfence Security Plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
  • Moving WordPress to a new self managed server away from CPanel
  • Moving WordPress to a new self managed server away from CPanel

General

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Using the WinSCP Client on Windows to transfer files to and from a Linux server over SFTP
  • Connecting to a server via SSH with Putty
  • Setting web push notifications in WordPress with OneSignal
  • Infographic: So you have an idea for an app
  • Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl
  • Building faster web apps with google tools and exceed user expectations
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in

Some ads on this site use cookies. You can opt-out if of local analytics tracking by scrolling to the bottom of the front page or any article and clicking "You are not opted out. Click here to opt out.". Accept Reject Read More
GDPR, Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT