Backup

Goodbye Dropbox, One Drive, iCloud and Hello Nextcloud private cloud on UpCloud

June 14, 2020 by Simon

I recently came across NextCloud Hub server (free on self-hosted servers) and I wanted to set up my own private cloud server to store my own files.

I wan’t to be able to access my files on Windows, Mac, Android and iOS.

Most of all I want a place in the cloud (that I own) that I can upload my Acronis backup of C Drive as the Backblaze client (read my review of Backblaze here) is a bit slow at uploading a 150GB backup file to the USA.

To create my own Nextcloud server I will need to login to these services.

I logged into my Domain Name provider porkbun.com (to ensure I had a domain name)
I logged into Cloudflare.com (to manage my DNS for a subdomain (redirected from PorkBun)).
I logged into my UpCloud.com account. (to deploy a new virtual machine)

Fyi: If you don’t have a favourite virtual machine provider you can use my referral link to obtain $25 free credit (only if you are new to UpCloud). Every new user who signs up with my referral link will receive a $25 bonus to get started. That’s 5 months free server (1 CPU and 1GB memory Linux server)

Post Index

NextCloud System Requirements
Creating a new Virtual Machine at UpCloud
Updating Ubuntu
Installing Common Software Packages
Securing SSH with the Google Authentication PAM module
Installing a Firewall
Installing NGINX and DNS
Installing PHP/PHP-FPM
Installing MySQL
Nixstats
CronTab Updates
Misc Security Stuff

1. NextCloud System Requirements

I checked the NextCloud version 18 system requirements and it needs the following to deploy.

Ubuntu 18.04 LTS (recommended)
MySQL 5.7+ or MariaDB 10.2+ (recommended)
Nginx with php-fpm
PHP 4 (recommended)

Minimum Memory Requirements

Nextcloud needs a minimum of 128MB RAM, and they recommend a minimum of 512MB.

I can deploy a server with at least 512MB memory free. The minimum UpCloud server I can deploy comes with 1GB of memory for $5 a month.

Time to create a new server.

2. Creating a new Virtual Machine at UpCloud

I logged into UpCloud and clicked “Deploy Server“

I selected Singapore as the place to deploy my new server (as it was closest to me here in Australia). UpCloud does not have servers here in Australia yet.

I checked https://wondernetwork.com/pings/ to ensure Singapore is the fastest location near. My server https://fearby.com is located in Chicago as it’s closer to my average readers and search engines.

I would like my NextCloud server to be as fast as possible to me. Singapore is the faster UpCloud datacenter near me.

I selected a server with 1 CPU Core, 1GB of Memory, 25GB of storage and 1TB of network traffic. I will add a 500GB drive to this server for additional storage.

If the server needs more resources I will upgrade it later.

The only downside of a $5/m server is the 1TB network quota. If I overuse the network (downloads) I will get an extra charge.

I reached out to the support to verify the costs if I go over my quota.

My question to UpCloud chat support.

Q1) With a $5/m server with 1TB quota what is the over charge costs if I go over 1TB
Q2) Is 1TB quota up and down or just down?

Prompt Answer

Hi Simon,

Good to speak to you again.

A1) Only Simple plans include monthly allowance of outgoing network traffic. After the allowance, the cost is $0.01/GB. It was a lot higher, but we reduced it to make it more competitive.

A2) The quota is for outgoing network traffic from your servers, all incoming and private traffic between your UpCloud servers is free of charge.
Regards,
~Long Lam

I hope this is helpful, let us know if you have any further questions.

Based on this information if I use all of my 1TB Monthly quota downloading files and I download and extra 150GB (e.g A 150GB Acronis backup image) it will cost $1.5 extra. That’s not bad.

UpCloud Chat Support

Before I selected a server type (Simple or Flexible) or storage type (MAX IOPS or HDD) I jumped onto the UpCloud chat and asked a few questions.

Q1) Hello, When deploying a server is there a cost difference between MAX IOPS and HDD storage? I am looking at a 500GB drive

A1) Storage (MaxIOPS), per GB $0.00031/ hourly $0.22/ monthly, Storage (HDD), per GB $0.000078/hourly $0.06 / monthly

Q2) What’s the difference between Simple and Flexible?

A2) Flexible will/turn out more expensive depending on your use case, generally, it is more suited for short term deployments.

> With our flexible plans, you decide yourself how much CPU, memory and block storage your cloud server is allocated. This gives you incredible flexibility and allows you to fully customise your cloud server according to your specific needs.

>Do also note when flexible plans are shutdown we only charge you for allocated storages and IPv4. Whereas in simple plans, it will be charged fully even when shutdown.

> Our simple plans are billed by the hour, up to a limit of 672 hours per month. Should you decide to use your fewer hours, you will only be billed for the hours you actually used.

UpCloud has very responsive and helpful chat staff. I never had this level of help with Vultr, Digital Ocean or AWS.

After I chatted with UpCloud support I decided to deploy a simple (Ubuntu 18.04) Server with 1 CPU Core, 1TB network traffic, 1GB of memory, 25GB system drive and an extra 500GB storage device.

When you create a server you can add an extra storage device. Nice.

Add a new device to the main storage device.

When adding an extra storage device you can choose faster MaxIOPS storage or slower HDD based storage.

I will choose HDD storage as it will be cheaper for a 500GB device.

I created a 500GB storage device for a Nextcloud data drive.

You can create up to 2TB storage devices with UpCloud.

I selected Ubuntu 18.04 LTS as the operating system.

I configured a login method as “Only SSH Keys” as I have already added my SSH key with a passphrase.

I selected my SSH key.

If you have not previously added an SSH Key to UpCloud then click Add new. Read more here.

I selected an Initialisation script I previously created (that just outputs a “Hello World” to a text file). One day I will create an Ansible or Terraform script to set up a server.

Select SSH Key and choose an init script

I clicked Deploy

I entered my desired hostname

I had a notification that the UpCloud Deploy is being deployed..

I could see in my UpCloud dashboard that the server was being deployed.

Server deploy is underway

Wow that took a whole minute to deploy a 525GB Server.

Deploy log said it took 1 minute to deploy

Wow UpCloud are fast

Configuring the server with Putty

Now it is time to connect to the Ubuntu Servers CLI and configure the server. I grabbed the IP address that was listed at UpCloud.

I opened Putty and added the IP address for the server.

Under the Auth section in Putty I added the path to my SSH Private Key (the same one that configured in the new server)

I saved the connection and clicked Open. I clicked Yes to the SSH fingerprint when I verified it was correct.

I now had root access to my new server.

Time to update Ubuntu.

3. Updating Ubuntu

I ran this command to update Ubuntu.

sudo apt-get update && sudo apt-get upgrade

Confirming the 2x storage disks

I ran this command to verify I had the 2 storage devices I selected at server deploy.

sudo lsblk |grep disk
vda    252:0    0   25G  0 disk
vdb    252:16   0  500G  0 disk

Yes, I have a 25GB disk and a 500GB disk

4. Installing Common Software Packages

I installed these packages

sudo apt-get install htop
sudo apt-get install lshw
sudo apt-get install ufw
sudo apt-get install ncdu
sudo apt-get install nmap
sudo apt-get install iozone3
sudo apt install pydf
sudo apt install mc
sudo apt install nnn

5. Securing SSH with the Google Authentication PAM module

Before I carry on any further I need to enable hardware 2FA login protections to all SSH logins. I will follow the guide I created here (Setup two factor authenticator protection at login on Ubuntu or Debian).

Warning: Take a backup of your server first. If you set this up wrong say bye-bye to your server. If I lose my YubiCo YubiKey and forget my backup codes I will have a hard time getting back in.

I will force all SSH logins to require my Hardware YubiCo YubiKey to be inserted (to generate a temporary One Time Password (OTP)).

You don’t need a YubiCo YubiKey, a generic software authentication app is OK but I prefer hardware devices as they are more secure.

I set the timezone to match Australia/Sydney. If I enabled a 2FA (OTP) at login with a different timezone than my connecting machine I would never be able to login to my server as my server and local PC need to be in the same timezone.

I ran this command to set the time in Ubuntu.

pkg-reconfigure tzdata

I then checked the time

sudo hwclock --show
2020-05-31 23:17:02.873751+1000

I installed the Google Authentication PAM Module (read more)

sudo apt install libpam-google-authenticator

I ran this command to configure the Google PAM Module

google-authenticator

I was presented with these questions

Do you want authentication tokens to be time-based (y/n) y

I was presented with a secret key, verification code and backup codes (I saved these somewhere safe)

Do you want me to update your “/root/.google_authenticator” file? (y/n) y
Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y
By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server, we allow an extra token before and after the current time. This allows for a time skew of up to 30 seconds between authentication server and client. If you experience problems with poor time synchronization, you can increase the window from its default size of 3 permitted codes (one previous code, the current code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes between client and server.
Do you want to do so? (y/n) y

If the computer that you are logging into isn’t hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

I can review all config values later with this command

sudo nano ~/.google_authenticator

Now I will enable 2FA at login by editing this file

sudo nano /etc/pam.d/sshd

I searched for “@include common-auth” then added this line after it.

auth required pam_google_authenticator.so

I then comment out the following line (this is the most important step, this forces 2FA)

#@include common-auth

Picture of my /etc/pam.d/sshd changes

I saved the file /etc/pam.d/sshd

Now I can enable the PAM Module by editing this file

sudo nano /etc/ssh/sshd_config

I searched for

ChallengeResponseAuthentication

And change the value to “yes”

I ensured the following line exists

UsePAM yes

I added this line then saved /etc/ssh/sshd_config

AuthenticationMethods publickey,password publickey,keyboard-interactive

Now I edited /etc/pam.d/common-auth

sudo nano /etc/pam.d/common-auth

I added the following line before the line that says “auth [success=1 default=ignore] pam_unix.so nullok_secure”

auth required pam_google_authenticator.so

Now I can restart the SSH Service and test the

/etc/init.d/ssh restart
[ ok ] Restarting ssh (via systemctl): ssh.service.

I restarted my putty session and reconnected to my server and I was prompted for the password for my private key and the randomly generated one-time password that was linked to my YubiCo YubiKey. Nice

Now I need to whitelist my SSH port to select IP’s.

6. Installing a Firewall

I installed the UFW firewall by typing this command

sudo apt-get install ufw

I configured UFW to rate limit SSH logins by typing this command

sudo ufw limit ssh comment 'Rate limit hit for openssh server'
Rules updated
Rules updated (v6)

I configured some common ports

sudo ufw allow ssh/tcp
sudo ufw logging on
sudo ufw allow http
sudo ufw allow https
sudo ufw allow 22
sudo ufw allow 53
sudo ufw allow 80
sudo ufw allow 443
sudo ufw allow 873

I added Cloudflare firewall rules (as my domain is behind their firewall and I will remove all direct IP access to my server later)

sudo ufw allow from 173.245.48.0/20
sudo ufw allow from 103.21.244.0/22
sudo ufw allow from 103.22.200.0/22
sudo ufw allow from 103.31.4.0/22
sudo ufw allow from 141.101.64.0/18
sudo ufw allow from 108.162.192.0/18
sudo ufw allow from 190.93.240.0/20
sudo ufw allow from 188.114.96.0/20
sudo ufw allow from 197.234.240.0/22
sudo ufw allow from 198.41.128.0/17
sudo ufw allow from 162.158.0.0/15
sudo ufw allow from 104.16.0.0/12
sudo ufw allow from 172.64.0.0/13
sudo ufw allow from 2400:cb00::/32
sudo ufw allow from 2405:8100::/32
sudo ufw allow from 2405:b500::/32
sudo ufw allow from 2606:4700::/32
sudo ufw allow from 2803:f800::/32
sudo ufw allow from 2c0f:f248::/32
sudo ufw allow from 2a06:98c0::/29

I added appropriate Whitelisted IP’s that can connect to Port 22 (SSH), removed blanket port 22 access and I configured my firewall to allow 91 incoming and outgoing rules (this is a secret)

I reloaded and enabled the firewall.

sudo ufw reload
sudo ufw disable
sudo ufw enable

7. Installing NGINX and DNS

I update Ubuntu again

sudo apt-get update && sudo apt-get upgrade

I installed Nginx

sudo apt-get install nginx

I edited my NGINX config and I change the default www folder location.

I also configured the log file location, mime types, max body size, gzip, default ports, ssl cert paths, security headers, default page, server name, sensitive file block rules, dns server, cache headers etc.

Read more to here to configure Nginx etc.

Fyi: Nginx config file locations

sudo nano /etc/nginx/nginx.conf
sudo nano /etc/nginx/sites-available/default

I typed my servers IP address into a web browser

I created an index.html file in the www folder and added “Hello World” to the file.

If I type my server’s IP address into a browser I can see this file.

My DNS is with Cloud flare so I logged in and added 2 DNS entries (IPv4 and IPv6) that direct traffic my new server IP(s) for this subdomain. To obtain the IP addresses I logged into UpCloud and clicked my server then clicked Network and noted my IPv4 and IPv6 addresses.

I then went to Cloudflare and added a DNS record for IPv4 and IPv6 pointing to my servers IP(s). I enabled Cloudflare Proxying to allow Cloud flare to try and hide the IP of the server. I then configured my firewall to block access to the IP except via Cloudflare and my whitelist.

I then checked for worldwide DNS propagation with https://www.whatsmydns.net/. After 3 minutes my DNS changes were all around the world. Thanks, Cloudflare.

I tried loading my site but CLiudflare said it was down.

I created a new HTTPS certificate at Cloud flare just to be sure and added it to my sites.

After investigating further I found this was because my primary website has a “Strict-Transport-Security header and I had enabled Full (Strict) SSL/TLS Encryption. I changed this to Full at Cloudflare.

My site was now working.

8. Installing PHP/PHP-FPM

To Install PHP 7.4 I ran this command to be able to get the latest version of PHP

sudo apt-get update
sudo apt -y install software-properties-common
sudo add-apt-repository ppa:ondrej/php
sudo apt-get update

I installed PHP 7.4 with this command

sudo apt -y install php7.4

I checked that PHP is installed by running

php -v
PHP 7.4.6 (cli) (built: May 14 2020 10:02:44) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.6, Copyright (c), by Zend Technologies

I setup some PHP Modules

sudo apt install php7.4-common php7.4-mysql php7.4-xml php7.4-xmlrpc php7.4-curl php7.4-gd php7.4-imagick php7.4-cli php7.4-dev php7.4-imap php7.4-mbstring php7.4-soap php7.4-zip php7.4-bcmath php7.4-tidy

I noticed apache2 installed (and broke my Nginx) so I uninstalled it.

 sudo apt-get remove apache2

I also blocked apache from installing again

apt-mark hold apache2
apache2 set on hold.

I checked to make sure Apache was blocked from installing

apt-mark hold apache*

apache2 was already set on hold.
apache2-bin set on hold.
apache2-utils set on hold.
apache2-data set on hold.
apache2-doc set on hold.
apache2-suexec-pristine set on hold.
apache2-suexec-custom set on hold.
apache2-dbg set on hold.
apache2-dev set on hold.
apache2-ssl-dev set on hold.
apachedex set on hold.
apacheds set on hold.
apachetop set on hold.

Now I will install PHP-FPM.

FPM is a process manager to manage FastCGI in PHP

sudo apt-get install php7.4-fpm

I checked the status of the PHP FPM service with

sudo service php7.4-fpm status

Output

php7.4-fpm.service - The PHP 7.4 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.4-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-06-06 21:34:31 AEST; 1min 54s ago
     Docs: man:php-fpm7.4(8)
  Process: 7767 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
  Process: 7772 ExecStartPost=/usr/lib/php/php-fpm-socket-helper install /run/php/php-fpm.sock /etc/php/7.4/fpm/pool.d/www.conf 74 (code=exited, status=0/SUCCESS)
 Main PID: 7769 (php-fpm7.4)
   Status: "Processes active: 0, idle: 2, Requests: 0, slow: 0, Traffic: 0req/sec"
    Tasks: 3 (limit: 1147)
   CGroup: /system.slice/php7.4-fpm.service
           |-7769 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf)
           |-7770 php-fpm: pool www
           `-7771 php-fpm: pool www

I might add some PHP child workers if I add more CPU’s to this server later

I edited my php.ini

sudo nano /etc/php/7.4/fpm/php.ini

I made these changes to php.ini

file_uploads = On
allow_url_fopen = On
memory_limit = 512M
post_max_size = 50M
upload_max_filesize = 50M
cgi.fix_pathinfo = 0
max_execution_time = 360
date.timezone = Australia/Sydney

I added read this page (Nginx Configuration) and edited my /etc/nginx/sites-enabled/default

I tested and reloaded the Nginx config and restarted NGINX and PHP

nginx -t
nginx -s reload

sudo systemctl restart nginx.service
sudo systemctl restart php7.4-fpm

sudo systemctl status nginx.service
sudo systemctl status php7.4-fpm

To test PHP FPM I created a php file in my website root and added the following text

<?php phpinfo( ); ?>

I loaded this file in a browser and I confirmed that PHP-FPM was installed.

The test was ok (I deleted this test file), I deleted the index.html and created an index.php file

9. Installing MySQL

To install MySQL I ran the following command

fyi: All usernames and database names are for example only.

sudo apt install mysql-server

I configured MySQL With this command

sudo mysql_secure_installation
Securing the MySQL server deployment.

Connecting to MySQL using a blank password.

...
Would you like to setup VALIDATE PASSWORD plugin?
y


There are three levels of password validation policy:
STRONG

Please set the password for root here.
New password:
**************************************************

Re-enter new password:
**************************************************

Estimated strength of the password: 100

Do you wish to continue with the password provided?
y


Remove anonymous users?
y

Disallow root login remotely?
y

Remove test database and access to it?
y

Reload privilege tables now?
y

Now to test MySQL I will login to it

sudo mysql -u root -p
************************************************************

Now I ran the following to create a database for Nextcloud

mysql> CREATE DATABASE databasename CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
Query OK, 1 row affected (0.00 sec)

I verified the database was created

mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| databasename       |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

I created a database user

mysql> CREATE USER 'username'@'localhost' IDENTIFIED BY '************************************';
Query OK, 0 rows affected (0.00 sec)

I verified the use was created with this command

mysql> SELECT User,Host FROM mysql.user;
+------------------+-----------+
| User             | Host      |
+------------------+-----------+
| **************** | localhost |
| **************** | localhost |
| **************** | localhost |
| username         | localhost |
| **************** | localhost |
+------------------+-----------+
5 rows in set (0.00 sec)

I set permissions to add the user to the database

mysql> GRANT ALL PRIVILEGES ON `databasename`.* TO 'username'@'localhost';
Query OK, 0 rows affected (0.00 sec)

I verified the permissions with this command

mysql> SHOW GRANTS FOR 'username'@'localhost';
+--------------------------------------------------------------------------+
| Grants for [email protected]                                      |
+--------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'username'@'localhost'                       |
| GRANT ALL PRIVILEGES ON `databasename`.* TO 'username'@'localhost' |
+--------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Finally I flushed permissions

mysql> FLUSH PRIVILEGES;

Now the databases is ready for Nextcloud

10. Nixstats

If you do not know what Nixstat’s is check out my post here Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc

I logged into Nixstats and click Add Server. I ran the provided install command.

wget -q -N --no-check-certificate https://nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ################## ##########################

Todo: Configure Nixstats PHP-FPM and NGINX Reporting (work in progress). My firewall rules are too tight for this install.

Handy Links

11. CronTab Updates

I created a update.sh file that I can call from a crontab entry to update Ubuntu and other software every xx hours.

I added this to my crontab.

12. Misc Security Stuff

I made sure my firewall only allowed traffic to my server was from Cloudflare IP’s and Whitelisted IP’s

Cloud flare IP’s can be found here.

https://www.cloudflare.com/ips-v4/
https://www.cloudflare.com/ips-v6/

At the time of writing the IP’s are

173.245.48.0/20
103.21.244.0/22
103.22.200.0/22
103.31.4.0/22
141.101.64.0/18
108.162.192.0/18
190.93.240.0/20
188.114.96.0/20
197.234.240.0/22
198.41.128.0/17
162.158.0.0/15
104.16.0.0/12
172.64.0.0/13
131.0.72.0/22
2400:cb00::/32
2606:4700::/32
2803:f800::/32
2405:b500::/32
2405:8100::/32
2a06:98c0::/29
2c0f:f248::/32

I blocked access to my webserver (port 80 and 443) to anyone but Cloudflare.

I whitelisted DNS traffic to only Up Cloud. Thanks, Lon.

Up Cloud support is awesome.

Installing NextCloud

Finally I can Install Nextcloud, I navigated to https://nextcloud.com/install/ and clicked Download for Server

I will use the Web installer to Install Nextcloud.

Nextcloud web installer instructions

Snip about the Nextcloud Installer from the download page

The Web Installer is the easiest way to install Nextcloud on a web space. It checks the dependencies, downloads Nextcloud from the official server, unpacks it with the right permissions and the right user account. Finally, you will be redirected to the Nextcloud installer.

1) Right-click here and save the file to your computer
2) Upload setup-nextcloud.php to your web space
3) Point your web browser to setup-nextcloud.php on your webspace
4) Follow the instructions and configure Nextcloud
5) Login to your newly created Nextcloud instance!

You can find further instructions in the Nextcloud Admin Manual.

Note that the installer uses the same Nextcloud version as available for the built in updater in Nextcloud. After a major release it can take up to a month before it becomes available through the web installer and the updater. This is done to spread the deployment of new major releases out over time.

I used WinSCP to upload the setup-nextcloud.php to my Nginx web root folder

I loaded the setup-nextcloud.php file from, my web browser.

I entered “.” to install Nextcloud to the website root.

There is no way Nextcloud installed in 2 seconds, I checked the size of the disk usage in my website root.

sudo du -hs /web-root
313M

Nextcloud took about 10 seconds to download 313MB onto my UpCloud Server.

Fyi: I installed the SpeedTest CLI app and ran a benchmark and UpCloud Chicago can download as 937Mbps/sec and UpCloud Singapore can download at 717Mbps/sec.

Now I need to enter the data root folder for Nextcloud . I installed lswh to be able to see my 500GB disk.

sudo apt-get install lshw

I ran the following to see my disks

sudo lshw -class disk -short
H/W path        Device     Class      Description
=================================================
**********      /dev/vda   disk       26GB Virtual I/O device
**********      /dev/vdb   disk       536GB Virtual I/O device

I formatted my disk

sudo mkfs.ext4 /dev/vdb

I created a new folder under mount to connect to the partition. The folder name is a made-up sample

sudo mkdir -p /mnt/foldername

I mounted the partition to the folder

sudo mount /dev/vdb /mnt/foldername

I made sure Nginx can access the folder

sudo chown -R www-data:www-data /mnt/foldername

I changed to the partition mount

cd /mnt/foldername

I created a test 490GB file

fallocate -l 490G test.file

I checked the file

ls -al
-rw-r--r-- 1 username useername     526133493760 Jun  9 19:38 test.file

I deleted this test file and set this mount point as the data file in Nextcloud setup.

I added a new Nextcloud admin username and password, mount folder for Nextcloud data folder, the SQL database user/password/database name and host and clicked Finish Setup

Nextcloud was setup.

Misc Setup

I ran the /settings/admin/overview report to see if I needed to perform andy final setup steps. I have a few missing php modules and a few optimisation tasks that need resolving.

Links to resolve.

Nextcloud External Security Scan

I loaded https://scan.nextcloud.com/ to perform a external security scan.

Scan Results

Adding Two-Factor Authentication (YubiKeys)

I noticed in the Nextcloud security setting page I can setup a YubiKey as a pass-wordless login device.

This would allow me to insert my YubiKey to login automatically

I added my YubiKey and gave it a name.

The password-less login method is a bit insecure as anyone that has my YubiKey can access my site.

I think I will set up a Two-Factor Authentication/OTP login method and link that to my YubiKey. I visited the /settings/apps/security page and installed the Two-Factor TOTP Provider app.

Install the OTP App — Install; the OTL app

I clicked the checkbox next to TOTP

The app generated a QR code that my YubiCo Authentication App can use to link to Nectcloud

I verified the QR scan and entered the 6 number verification code from my YubiCo Authenticator app

Two Factor logins are now enabled.

Now after I log in I have to enter a temporary 6 digit number that is only valid for 30 seconds (and only after entering my YubiCo YubiKey into my PC and entering its password)

Nice

Nextcloud Overview

I logged into Nextcloud and was greeted with a wizard.

The sample images in the welcome screen are a bit small.

I can add native apps to Windows, Mac, iOS and Android or I can log in via the web page.

Pointers to the manual, community help and forums.

Main screen is clean.

A user context Menu is linked in the top right.

I setup email alerts (I allowed outgoing ports in my firewall)

sudo ufw allow out 465/tcp
sudo ufw allow out 465/udb

I used my GSuite account to send emails.

Syncing Files from my PC to Nextcloud

I tried uploading my 150GB Acronis Backup image file to Nextcloud by the web interface but this will fail for sure, this will take many hours.

I decided to configure Acronis True Image to split backups into 100MB chunks.

I created another Acronis image of my Windows Drive.

Nextcloud Windows App

I visited https://nextcloud.com/install/ and installed the Nextcloud Windows app to sync files.

I clicked Windows

Click Next

Click Install

Nextcloud sync app is now installing

Next cloud sync is now installed.

Click Log in

Enter your Nextcloud server https address and click Next

A web browser login screen appeared and I logged in

After I logged in Nextcloud sync was connected

I was prompted to sync everything online to my local PC or choose folders to Sync .

All files that were in Nextcloud synced down (that I selected)

I set Nextcloud to start at Windows start.

I reviewed Download and Upload limits

I decided to add my U:\AcronisBackup folder to my Nextcloud server.

I was asked to add this to a remote Nextcloud folder.

Files were backing up.

I has 150GB of Acronis backup files backing up.

I could see each 100MB section of the Acronis Backup appearing in the Nextcloud web app.

I noticed that the raw file system list of files was about 30 seconds ahead of the web list.

I had an Alert from my Acronis Backup software that new backup files were downloading.

The Acronis backup folder started backing up but I noticed it was redownloading to a new folder. I don’t want this.

I allowed Nextcloud to access backup files

I paused the Nextloud Sync and my 150GB Backup was re-downloading to a new folder.

It looks like U:\AcronisBackup was backing up then downloading to U:\Nextcloud\Simon\AcronisBackup.

I moved my Acronis backup from U:\AcronisBackup to U:\Nextcloud\Simon\ZENigma (ZENnigma is the name of my PC)

I moved my 150GB backup files into Nextcloud folder/

I deleted the old sync of U:\AcronisBackup and started the Nextcloud Sync again

Now my Acronis backup (150GB) was backing up to Nextcloud.

It took 24 hours to backup 150GB from my PC to my server in Singapore.

I can see a handy summary of synced files and disk space used/free.

I can control the sync with a System Tray App.

Nextcloud Conclusion

Pros

Free
Works well.
I have an offsite location for backups and an area for file sharing with my family
Faster than Backblaze and Dropbox

Cons

Needs better Hardware 2FA support
Some Nextcloud web pages are not mobile-friendly (e.g add new user)
Needs better post install security checks
Web view of files could be updated more often, there is as 30-second delay between the web list of files and a CLI list in Putty of /mnt/foldername/username/files/

Troubleshooting

NGINX website is not loading

Check to see if a package has downloaded apache (this will take out Nginx).

Also, make sure you have set permissions on the folder that holds your SSL Certificates and allow your Nginx www-data user read access.

sudo chown -R www-data:www-data /etc/nginx/https-cert/

Deleting a MySQL Database

I had an issue where Nextcloud did not like the database I created so I ran the following to revoke the database users permissions, remove the user and I deleted the database.

Command to revoke the users MySQL permissions

sudo mysql -u root -p
*************************************
mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'databaseusername'@'localhost';

Delete the MySQL user

sudo mysql -u root -p
Enter password: *************************************
mysql> DROP USER 'databaseusername'@'localhost';

I reset flushed permissions

sudo mysql -u root -p
Enter password: *************************************
mysql> 
FLUSH PRIVILEGES;

To delete the database run the following.

mysqladmin -u root -p drop databasename
Enter password: *************************************
Dropping the database is potentially a very bad thing to do.
Any data stored in the database will be destroyed.

Do you really want to drop the 'databasename' database [y/N] y
Database "databasename" dropped

Thanks for Reading

v1.1

Finding software that needs updating on Windows using SUMo (Review)

May 30, 2020 by Simon

Recently I blogged about using Acronis True Image 2020 to backup and restore your Windows computer after a friend had to manually reinstall her Windows PC.

Her Windows PC died and she could not recover Windows with a System Restore snapshot or via the retailers systems image recovery process.

I use Back Blaze software to automatically backup my computer but I do not have an up to date collection of needed installation files in case I need to perform a Clean Installation of Windows 10.

I found SUMo in the process of assisting my friend.

SUMo to the Rescue

I found SUMo (Software Update Monitor) from KC Softwares that allows me to analyse my existing installed software and their versions, find and download new versions (installers or setup files) for my software.

I am using the Pro version of SUMo (buy here).

Tip: The trial version will inform you of an update but it will not show you links to obtain the updates easily.

SUMo Trial Limitations

The best feature of SUMo is I can find apps on my PC that need updating.

Also in one click (Right click) you will be directed to the applications official site (official site (or App page on SUMo)) and be able to download the app installer to the new version of the app.

Rigth click on an outdated app in SUMo to get the new aopo.

The Trial version is limited to informing you that an update is available.

SUMo telling your there is an update available

Also updating SUMo to the PRO version streamlines it’s own updates too. You can update SUMo in the PRO version with one click. 🙂

SUMo Pro Scan

After I opening SUMo PRO (I waited for it to find installed apps) then I clicked “Check” to scan all my apps for updates.

SUMo happily went through the list of my installed apps and gathered the applications versions (and checked online if newer versions existed).

SUMo found 239 installed applications on my PC, 188 were up to date, 53 needed minor (recent updates) and 2 needed major (old) updates.

Its awesome that in one click and I can see what software needs updating. In the process I was able to see what applications I don’t use (they were old versions) and I uninstalled them

SUMo Settings

I will update my software and to ensure my software does not get out of date I enabled “Load at startup” and “Check for new software at startup” (SUMo).

I did not enable “Scan for new software at startup” but seeing SUMo Start and update will remind me to check for updates if I have time.

I also enabled “Show file path” in the SUMo Application list so I can see where the programs are. I found multiple instances of Python installed all over my PC.

Getting Application Updates

I right clicked on each of my applications (that needed updating) and clicked “Get Update from…” then “.. from developer’s Site (Direct)“.

If you don’t want to right click you can click the “Get Update” button in the tool bar to get selected applications update.

This will open a web browser page to the software vendors website.

SUMo also has a page available with stats on the application’s known statistics (nice table of versions, beta or stable status and numbers of users).

Updating Software

I was able to right click “Get Update from…” – “.. from developer’s Site (Direct)” for each application in the list that needed an update. I was directed to the software vendors page for each software where I could download the app updates.

Within minutes I had dozens of vendor pages open with links to new versions of my outdated software.

Dozens of site pages with fresh app installs.

Within a short period I had amasses gigabytes of new application installs.

4.5GB of data of new application installs

I installed each of the new updates and then moved the new installs to my B:\Installs folder (in case I have to manually reinstall Windows in the future).

Checking Application Update Status

After I updated my out of date applications I then right clicked on the apps in SUMo and clicked Check to re check the status of the applications

SUMo Right Click Actions

Fyi: SUMo has a number of actions in the right click menu (on an installed Application).

Delete: Removes item from list. Will appear again on next scan or manually addition
Ignore: Blacklist : this Software/company pair will never appear again, whatever the version, file path, …
Skip: Items remains listed but a SPECIFIC version gets no longer notified as update for a given time (1 day / week / month) or forever.

(snip from here)

I had a few items I Deleted from the list or Ignored.

My Visual Studio (development) software was reporting an older version of Python (2.7). I updated Visual Studio to the latest version but Python 2.7 remained. I deleted Visual Studio from the list and was prompted (this is not deleting the application from the hard disk just from the generated list.

After a few hours updating apps

I whittled down the list of my applications that needed updating

I could sort applications in the list by OK status or apps that needed Minor or Major updates applied.

All Apps Updated

I can now say that all of my Applications are all up to date.

Export

I used the export feature in SUMo to build an excel file listing all my applications.

Fyi: SUMo exported this list as a “;” separated file, I opened the file in Sublime Text Editor and replaced “;” with TAB and then pasted this list back into Excel.

Conclusion

SUMo allowed me to update my installed applications and update my aged cache of application installs I was keeping in case I needed to manually rebuild Windows.

SUMo made this an absolute pleasure to do.

Pros

Price: 19.99 Euros (about $33.27 AUD) is a bargain.
Simplicity (it just works so well, why does Windows not do this)
Saves me time finding out of date apps and update files for an emergency and inevitable manual Windows reinstall.
My apps are more secure after updating them

Cons

I needed this in my life 10 years ago
Would be nice to have an inbuilt help file (instead of email support)

Try the free version here

v 1.0 Initial Post

Using Acronis True Image 2020 to backup and restore your Windows computer

May 24, 2020 by Simon

This is a lengthy (but simple) guide on how I get files ready for a clean install of Windows, backup and restore Windows (C Drive and associated hidden Windows partitions) using Acronis True Image 2020).

Backup and Restore Windows Prerequisites

You will need to buy a licence of Acronis True Image 2020 to..
- Backup Windows
- Restore Windows
You will need a second hard disk larger than your Windows drive (to backup to).

Backup Rules

Backup to Three Places.
Two of the media need to be different
One of the backups need to be offsite

Even though I use Back blaze software to automate backup of all files on my PC for $6.60 (AUD) a month. I pay an extra $2 (AUD) a month Backblaze will retain all of my files for up to 1 year. I also back up data to Backblaze Bucket(s) with Duplicati.

With all the horror stories of Windows updates killing machine’s it’s time to return to using Acronis True Image (not a paid endorsement) to start backing up the drive my Windows is on (all partitions)

I remember using Acronis True Image 8 in 2004 and loved it. Time to buy the new version. I loaded https://acronis.com/ (Australia Region) and noticed a cloud backup in the marketing information. I hope they still do local backups line in 2004.

I loaded the Acronis True Image 2020 link from the downtown menu (Australian link).

Acronis True Image 2020 Download Menu at acronis dot com

$69.99 seems fair,

Acronis True Image 2020 buy now button at acronis dot com

Darn, $69.99 is a yearly subscription. I don’t like subscriptions if I can get away with it.

Acronis True Image 2020 subscription and standalone options at acronis dot com

I clicked Buy under standard licence.

Acronis True Image 2020 was $83.99 AUD at acronis dot com

$83.99 is a bit high

Saving a dollar

TIP: Make sure you install the Honey Extension (from joinhoney.com) and use the Google Chrome Web Browser before purchasing to get a discount on Acronis True Image 2020.

The Honey extension applied the coupon so fast I could not get a screenshot it(so here is a screenshot with American Currency from a different web browser)

Now back to my purchase (and country), $33.60 is a nice saving.

I am paying for this because 16+ years ago I used an older version of Acronis True Image and the free version backed up OK but when I tried to restore from a boot CD I needed to pay?

Honey saved me $33.80

I am happy to pay $50.33 for Acronis True Image 2020.

After I purchased Acronis True Image 2020 I was presented with links to download the related files to use the software and to restore.

I downloaded the main Windows Acronis True Image Application for Windows and the Universal Restore program.

The files were about 900MB in total.

While the downloads were downloading I created an Acronis account

I added my purchased licence to my Account.

I was shown a welcome to Acronis page after creating an account.

The downloads finished and I started the AcronisTrueImage2020.exe setup file

I clicked Install

After the setup completed I was directed to a Quick Start Guide

Starting Acronis True Image 2020

When I loaded Acronis True Image 2020 for the fits time I had to agree to the Licence Agreement

A pasted in my just purchased Licence key

Start a trial or activate with a serial number

I already had an account (created minutes earlier in a web browser) so I clicked Sign In.

I was presented with a help wizard on startup.

Now I am ready to create my first backup.

Creating a Backup

Acronis True Image 2020 wanted me to back up all of my PC but it is 8TB, I don’t have an 8TB drive handy to back up to. For the life of Me I could not see how much 8TB would cost per month to backup.

By default Acronis wants to backup your whole PC

I clicked on my computer name (“ZENIGMA”) and selected “Disks and partitions“

I un-ticked all system drives and partitions that were not my C Drive disk. I have installed Windows 10 onto a Corsair MP600 M.2 PCIE 1TB SSD and I want to back it up. The M.2 SSD connects to my main board.

I selected this drive to backup.

View of all disks and partitions, I ticked the MP600 drive.

Oh dang, I don’t have enough space free on any drive to backup the C Drive to., Time to move some files.

Backup Destination

I have 8 partitions (drive letters) on 5 SATA hard rives and 2 M.2 SSD’s. I think I can move some data and free some space for this backup.

I had to shuffle data across drives to create enough space to backup my c drive

File Musical Chair’s

I used WinDirStat from https://windirstat.net/ to scan my U drive (my newest hard drive) for files to move.

I used WInDIRStat to find files I can move

WinDirStat shows a nice graph of files on my U Drive. WinDirStat colour codes file types by colour and also sizes squared by actual file size.

Windirstats scan results, Loads o Music I can ,move.

I could see 240GB of music that could be moved. My wife and I have converted all CD’s that we have purchased since the 1990’s to live alongside out Apple iTunes collection.

I used the Windows feature to right click on the Music folder and move it the folder to another drive.

Windows confirmed hat I wanted to move this folder.

It took 2 hours to move my Music folder to a smaller slower drive.

Now that I have 800GB free I can backup up my C:\ Disk

I re opened Acronis True image and resumed from Backup point where I left off and made sure I was backing up my C Drive (Corsair MP600 SSD)

My C drive is 500gb and the destination has 888GB free

Acronis showed the Windows drive to web backed up.

I clicked “Select destination“.

I clicked “Browse“

I selected “U:\AcronisBackup” as the target to backup C drive to

I was prompted for a encryption password (nice)

Now the backup is ready to start.

I clicked Back up now

Backup in Progress

The Backup was going to take about 40 minutes. I could see My C and U drive were busy in Task Manager

After the backup is complete Acronis verifies the Backup

After the backup finished I could see that Acronis managed to backup C drive (570GB) to a 329GB file on a different drive

Now I need to restore the backup to confirm it works.

Only a working backup is of value.

Creating Rescue (Recovery) DVD Media?

Now that I have a backup of Windows I need to create a boot media that will allow me to restore the backup image in case Windows dies.

I clicked in the 4th icon from the top in Acronis True Image 2020 then Rescue Media Builder

I clicked Simple

First I created a DVD boot disc.

I selected my DVD Dive (it had a blank DVD in it)

The Rescue Media needs a 800MB CD or DVD.

I clicked Proceed to burn the rescue media to the blank DVD

The Rescue Media DVD was burnt to a blank DVD, a few tips on using the Media were presented

I labelled the DVD Rescue Media and put it in a safe place.

Creating Rescue (Recovery) USB Media?

Now I will create a USB rescue Media Key.

I opened Acronis, Click the Tools Icon (same as the DVD method above), then Rescue Media Builder, click Simple and select your Empty USB key.

TIP: Make sure it is the correct Key as they key will be deleted first.

As with the DVD method above the Rescue Media files to be copied were about 800MB.

When the UBS key is prepared you will receive the same success screen as the DVD method above.

Restoring a backup from the DVD Rescue Media

I inserted the Rescue DVD into my DVD drive, I rebooted my PC and pressed F2 to enter my BIOS (your key may be different).

I navigated to the section in my BIOS where I could choose my Boot device and made the DVD drive boot first, this will allow my Rescue DVD to boot before Windows.

I rebooted my PC and I pressed a key when prompted to boot from the Rescue DVD

The DVD buzzed away for a minute and then I was prompted with Back Up or Recover menu. I clicked U:\Blog\Acronis True Image\DVD Restore.

I navigated to the U Drive (SFEARABY) Drive and selected the Backup I had taken earlier.

FYI: The Drive letters did not match in Windows and the Rescue Media. Take note of tour drive names. My U:\ Was listed as E:\.

I entered the password that I used to encrypt the backup.

I selected the backup of my C Drive to restore. I assume you can choose multiple, drives if you had backed up multiple drives.

I selected Recover whole disk and partitions.

I was asked to select the date to backup from (I assume this is for when you have multiple backup events over many days).

I was again asked to select partitions to recover.

A summary of the restore was presented and I clicked Proceed.

The Recovery was underway, It took about 8 minutes to restore over 500GB.

It worked.

I restarted my PC and removed the Rescue DVD.

Windows was all restored to it’s earlier state.

Restoring a backup from the USB Rescue Media

Restoring is exactly the same as the DVD method.

Reboot and Enter your BIOS
Set the Boos Order to allow the USB key to boot first
Reboot (no need to press a key to boot from USB)
Backup and restore (as needed)

I was able to restore the backup no problems,

FYI: I had the same slowness in the UI in the same spots as the DVD, nothing that breaks the process.

The USB and DVD Recovery offers a backup and restore options.

The Recovery media also allows you to clone a disk and setup a secure hidden partition for backups.

Backup Protection

Acronis True Image 2020 also protects backup locations and prevents access to them. I had to Allow Windows to be able to delete a backup.

Once I allowed Windows to be able to access the backup I was able to delete it.

Acronis allowing windows to manage backups files on Windows

Backup Schedules

Acronis True Image 2020 allows you to set Backup Schedules.

I do not have enough disk space to setup anything other than one full backup at a time

Backblaze

I still have Backblaze automatically backing up my files from within windows and this is my 2nd later of defence.

Backblaze will backup the 300+ GB backup.

Other Features

Active Protection

Acronis True Image 2020 has built in Active Protection to watch processes for ransomware or cryptographic. Read more here: https://www.acronis.com/en-au/active-protection/

I have had one false positive upgrading Java but ill leave it enabled.

A nice log is available of all events and whitelisted apps.

I have had to exclude apps but this has settles down.

A notification appears on blocked or whitelisted events 🙂

I can simply block or allow to deal with each exception.

Thus is a nice feature.

Cloud Backup

I am now using any Cloud backup features in Acronis, ill let Back blaze handle that.

Archive large or old files

The Archive large files or old files is a bit basic for me to use.

I will keep using CCCleaner and WinDirStats for this job.

Even the Windows default Disk Cleanup is good.

Cloud Sync

I do not use Cloud Sync, I would but I can’t find a price on the Acronis site at all.

I noticed that on the DVD and USB recovery boot I can restore a windows from a Acronis cloud backup (nice).

Other Tools

Clone disk (like EASE US?)
Universal restore?
Try and Decide
System Clean Up (backup First)
Acronis Secure Zone
Drive Cleanser
Third Party Tools

Acronis True Image (Conclusion)

I am being picky here as I loved using Acronis 10 years ago and I have rose coloured glasses.

Pros

It works
You can backup Windows while Windows is open
DVD and USB Rescue Media creation is easy
Backup Schedules
Notifications of disk space and operational state
Exclusions of non essential files
Backup Validation (at time of creation and schedule)
Ability to call pre and post backup commands
Choice of Incremental, Full or differential backups schemes
Cleanup of older incremental versions (and full version frequency)
Simple single version backup (single file, overwrites each time)
It works.

Cons

Some Buttons in the GUI are slow to respond
The DVD an USB Rescue Media is little unresponsive at times during the restore (but it works)
The DVD and USB Rescue Media GUI can use a bit of design love (I am sure it was better 10 years ago (basic and clean))
When I restored I had no Idea what C:\ Drive was listed as H:\ (loads of confusion on the internet)

Disclaimer

I am not paid by Acronis to say this, this is just my guide for my friends. I paid with my own cash. I take no responsibility if this does not work for you. Good luck though.

Troubleshooting

DVD Rescue Media

The DVD/USB Rescue Media is a bit laggy. At one point in the recovery I has a white screen for over a minute. I waited and the screen came good

Also the Menu in the DVD/USB toolbar does seem a bit squashed.

Bonus: Windows System Restore

There may be a time when you have created a full Operating system backup but the restore does not work (e.g hardware has failed (and been changed)). Acronis True Image 2020 will allow you to restore files from a backup image (and not the whole partition or disk).

Make sure System Protection is enabled for your C Drive

Create a System Restore Image

If you have Windows 10 it is a good idea to create a System Restore point ever few months just in-case Windows goes bad.

Click Your Start Button then type “Create a restore point“

Click your C Drive and click “Create“

The restore point will be created

Restoring a restore point

At any time you feel that Windows is sick (and still working) you can restore and old system restore.

Click the the desired restore point then click Next

You will be able to restore the old system snapshot.

Bonus: Clean Windows 10 Install

TBA: Blog post coming soon

Bonus: How I prepare all files needed for a clean install

Before you do a clean install of Windows 10 you may need to document what software you have installed so you have a change to reinstall them after you reinstall Windows.

This is possibly the most boring job but the one with the most reward. The hardest party is knowing what software you have an need.

Finding Installed Software with SUMo

I use a free program called SUMo from KC Software (not a plug) to list all installed programs and to see what software is out of date.

SUMo informed me that I have 256 software products installed.

7 were really out of date
62 products had recent updates

I took this list and downloaded and installed new copies of the apps I had installed (as newer version’s of apps can be more secure) and I also saved the installer files to a folder away from C Drive (e.g B:\Installs).

I am a bit OSD and I have created 10 folders under the B:\Installs folder that contain installs from Essential Drivers to non essential apps that I can use in the event of a clean install of Windows

In the Installs folder I have these sub folders

\100 Drivers\
\200 Essential Apps\
\300 Productivity Apps\
\400 Development Apps\
\500 Utils\
\600 Games\
\700 Virtual Machine OS Installs\
\800 Virtual Machine Apps\
\900 Learning\

When reinstalling Windows I start by installing all apps in each group (starting with “100 Drivers”), The larger the parent folders’ number is the less important the files are int hat folder.

I have a Legend.txt with a summary of the contents of each folder

Install Folders Legend

100+ ~ 199 Drivers
     - Mainboard ASUS RUF X570 (WIFI)
     - AMD Chipset Drivers
     - Nvidia Video Card
     - Etc
201+ ~ 299 Essential Apps
     - Printer Drives
     - 1Pasword
     - Antivrus
     - Backblaze
     - VPN
     - Acronis
     - Etc
300+ ~ 399  Productivity Apps
     - Microsoft Office 
     - Microsoft Visio
     - Etc
400+ ~ 499  Development Apps
     - Visual Studio
     - Arduino IDE
     - Fritrzing
     - Microsoft SQL Developer
     - MySQL
     - Python
     - PHP Storm
     - Etc
500+ ~ 599 Utils
     - CPU-Z
     - Core Temp
     - Acrbat Reader
     - Etc
600+ ~ 699 Games
     - Steam Installer
     - Origin Installer
     - Etc
700+ ~ 799 Virtual Machine OS Installs
     - Windows 3.11
     - Windows 95
	 - Etc
800+ ~ 899 Virtual Machine Apps
     - Office 95
     - Office XP
     - Etc
900+ ~ 909 Learning
     - Python Books
     - Etc

You can create whatever numbering scheme you want.

I also have a cached folder of my Development Software (Visual Studio) Install cache at “B:\Installs\vscache“

Under my drivers folder I have all the drivers I need to reinstall windows

An under my “500 Utils” folder is all the utilities I install (many).

Hundreds of sub folders with utilities in them

The main thing is I have all the files and installers needed to do a clean setup of Windows 10 if need be.

Links

Version History

v1.1 Added “Backup and Restore Windows Prerequisites”

v1.0 Initial

Backing up your computer automatically with BackBlaze software (no data limit)

June 2, 2019 by Simon

Backblaze ( https://www.backblaze.com/ ) is an awesome company who not only create a mega reliable backup infrastructure but they also tell us how they do it and tell us what hard drives are good or bad.

Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj

They are so confident and experienced that they offer unlimited backups for $6.60 a month here in Australia. They state they have 750 Petabytes stored on their infrastructure & have restored over 40 billion files to customers.

Check our my other related posts

Main features of Backblaze.

No limit to files that you back up (number or size or files)
Web-based file recovery (if needed)
They will mail you a hard drive if you need to restore large amounts of files.
Ability to locate lost or stolen computers.
Mobile app access your backups.
Business options for large computer fleets.

Creating a Back Blaze Account

Go to backblaze and create an account.

Choose a Backblaze Plan

Go to: https://secure.backblaze.com/buy_plan.htm to choose a backup and payment plan. Use my link and get your first month free.

Australian Prices below (add +10% GST).

$6 AUD a month
or $60 for 12 Months
or $110 for 24 Months

Screenshot $6 AUD a month or $60 for 12 Months or $110 for 24 Months

Choose a payment plan

Enter payment details

Click “Buy Backblaze”

Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj

Done, now I can download the client.

Installing the Backblaze Software

When the download is down you can start the install.

You will need to login to the install to streamline the setup.

Backblaze install screen asking me to login

After you login the installation will begin.

Backblaze is installing and checking what files need backing up.

Backblaze now reports that the install has complete. Backblaze reports that I have 379GB to backup.

Pressing OK shows Backblaze is already uploading my files.

Also, I have 15 days of free trial before being billed.

Clicking on the backblaze system tray icon reveals a few options

Backblaze Preferences

Backblaze preferences can be opened by clicking Settings from the main screen.

I..

Renamed the computer.
Disabled backup on battery power
Set upload threads to 5 (then lowering if the network gets busy or when the backup is almost complete)
Disabled Automatic throttling and set faster backups

I set a private encryption key

There is a 50 char limit on encryption keys

50 char limit on encryption keys message

Warning about not forgetting the encryption key

All other preferences look ok.

That’s it the backup is uploading automatically and there is nothing else I need to do.

Backblaze Portal

If you login to the backblaze portal ( here ) you can view your files (you will need to enter the private encryption keys if you opted to encrypt your files).

Initial Backup time

Your initial backup may have a while to upload depending on your backup size and internet connection speed.

I was able to upload 400GB in 3 days on my Internet Plan. Smaller files are uploaded first then larger ones. I have 2 files left to upload totalling 7GB.

Read the official guide on speeding up uploads here.

One way to make the initial backups faster is to see what files are queues to upload in the Backblaze preferences and then excluding unwanted files and folders.

I can see my Internet Explorer cache and Google Chrome Temporary Internet File(s) folders are queued to be back up.

Time to exclude these folders in the Backblaze Settings under exclusions

C:\Users\Simon Fearby\AppData\Local\Microsoft\Windows\INetCache\IE
C:\Users\Simon Fearby\AppData\Local\Google\Chrome\User Data\Profile 1\Code Cache\
C:\ProgramData\MySQL\MySQL Server 8.0\Data\#innodb_temp\
C:\ProgramData\USOShared\Logs\
C:\Users\Simon Fearby\AppData\Roaming\Adobe\SLData\SLCache\

Time to exclude these folders.

That knocked off a few thousand files to backup 🙂

Restore Files from Backblaze

If you restore files you have a few options

Download the files (free)
Save files to B2 Cloud (Backblaze)
USB Flash Drive
USB Hard Drive

restore options. 1) download, 2) save to BS cloud, 3) USB Key, 4) USB Hard Drive

Choosing to download files you are given a treeview to restore files from.

I selected a test file to restore (restores are not instant) and was informed I would be emailed when the files are ready to download. Thats cool

I waited 3 minutes and received an email that my files were ready to download.

I hope this helps someone.

How to back up an iPhone (including photos and videos) multiple ways

June 2, 2019 by Simon

This guide is for and anyone who needs to back up their iPhone (complete device along with separate backups of photos and videos).

This is not a paid promo, I don’t get a kickback for mentioning the awesome app below.

Check our my other related posts

At the time of writing iTunes v12.9.5.7 was the latest version of iTunes.

iTunes is Apple official software for getting files too and from an iPhone, iPod or iPad. If rumours are right Apple will kill off iTunes software at the end of 2019. If Apple kills off the iTunes software I will update this guide in the future.

Disclaimer: I provide no warranty or support for this app, this is what happened to work for me.

Why write a guide on Backing up a mobile phone?

It’s not that simple an I get asked how to do this almost every week by someone.
No one thinks to backup up their photos until they fear their phone is lost or stolen.
I am about to publish a few posts on backing up (development machines, servers etc) with automated software and with free open source software so this post will be handy to link in what data you can back up in that post.
The last time I wrote an article on backups was in March 2016 and its not that great these days.

Golden rules of backing up.

Backup to Three different locations.
Two of the media need to be different.
One of the locations needs to be offsite
Test your backups (a backup is only good if you can restore). (from @Daniel15)
If the backup is encrypted make sure you have the decryption keys. (from @Daniel15)

Backing up the official way (with iTunes)

Apple prefers you just pay them to extend your free 5GB iCloud storage and not worry about backups and not to worry yourself with manual steps.

My wife’s iPhone is always popping up messages saying that her iCloud storage is nearly full and she should upgrade the free space limit.

Screenshot of iCloud setting on iPhone saying your iCloud is nearly full but you have used 5GB or 5GB?

I know she has 100GB+ images and movies on her iPhone but iCloud has only backed up 4GB of photos and videos. Do not trust any backup statuses unless you can verify all files from a restore.

For the record I use an Android phone (Android backup guide coming soon).

iCloud is a good idea for automatic backup but I prefer to see my photos and backup them up myself. Also, Apple iCloud is not free from troubles. If you want Apple to handle backups then I do suggest you upgrade your iCloud storage from the free 5GB to a more sensible 200GB or more.

Screenshot of https://support.apple.com/en-au/HT201318

Here in Australia Apple charge the following for extra storage.

50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

A rough estimate: 50GB is enough to store about 12,000 files (9,000 photos + 3,000 videos) *

* = based on stats from the backup below (depending on the size an length of your videos).

Screenshot of apple iCloud pricing 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

You can upgrade your paid Apple iCloud backup limit from your iCloud Storage Settings from your iPhone’s settings screen.

Screenshot of iPhone iCloud upgrade button.

IMHO: Apple does make it clear what you are getting before you purchase (but they do not push it enough and people assume their data is safe).

iPhone picture of 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

TIP: Please review your existing iPhone data usage and iCloud usage before it’s too late

I don’t want ot pay Apple for more iCloud storage

I like you, I am tight too but I do pay Google $2.49 a month for 100GB backup storage on my Android phone. Google Drive (Google storage is a bit cheaper than Apple).

Screenshot of Google Drive https://www.google.com/drive/

It is nice knowing your phone is auto backed up.

Install iTunes on Windows (10)

On your Windows 10 computer click the start button. If you don’t have a Microsoft Account go here and create one. Click the start button and type “itunes” (you don’t need to type into a box, just start typing when you click the start button)’.

Then click “iTunes, install app”

Screenshot of a Windows start menu and Get iTunes icon

If you are logged into your Windows 10 store with your Microsoft ID click “Get” (if not you will need to login to the store).

Screenshot of me logged into the Windows Microsoft Store.

After you click ‘Get’, iTunes will start downloading.

Screenshot of the Windows Store downloading iTunes

When iTunes is downloaded Windows will install it. When it’s installed click ‘Launch’

iTunes is installed, Click launch screenshot

You will need to agree to Apple’s terms of service. Click ‘Agree’

When iTunes opens click “Agree”

Screenshot of Apple iTunes asking if we can agree to share analytics data

Now login to iTunes with your Apple ID (if you don’t have one create one here)’.

Screenshot of iTunes open with an arrow pointing to the Account menu

Click the ‘Account’ then ‘Sign In’ menu.

Screenshot of the account then sign in menu

Optional: If you have Two Factor Authorisation (2FA) of Apple ID’s turned on (you should) you will need to enter a 6 digit code.

Apple Two Factor Authorisation (2FA) will send one of your other devices a login code that you will need to use to login (confirm you own the account)

Screenshot of apples 2FA web page at https://support.apple.com/en-au/HT204915

TIP: Check out https://twofactorauth.org/ to see what other sites use Two Factor Authorisation (2FA). I use hardware Yubico YubiKeys to protect logins to WordPress, Linux and websites.

Now back to the article, iTunes should be ready to allow us to backup our iPhone.

Before I continue I will click ‘Edit‘ then ‘Preferences‘ menu to view where iTunes will download media too (different ot backup data), not important but I just want ot see if it is not pointed ot my smaller C drive before I backup my phone

I changed the location to a larger S:\Drive’

Screenshot of Apple setting screen (tab Advanced)

I was greeted with a message asking me to confirm that I wanted to sync the iPhone that I just plugged in called ‘EllieRose’, I clicked ‘Continue’

On the iPhone I also clicked ‘Trust’ to allow iTunes to talk to it.

iPhone screenshot to allow iTunes to talk to it.

I was then prompted to download a software update to the iPhone, It appears this phone is not running the latest software

Screenshot of a question to update to iOS v 12.3.1

I was prompted to sync the iPhone to the computer

screenshot of a prompt to sync purchases from the iPhone to the computer

A backup of the phone was underway.

Screenshot of iTunes backup progress with a progress bar

If the screen above does not appear click the icon below to view the backup and restore iPhone menu.

Screenshot showing the iPhone Device button in iTunes

The iPhone was backing up to this folder on my computer:

C:\Users\Simon Fearby\Apple\MobileSync\Backup

I was not prompted for a location to back up so and I will move this backup folder after the backup completes (so my C: drive does not fill up).

My drives.

C:\ = 500GB drive (faster SSD)
S:\ = 2,000GB Drive (slower Magnetic)

Screenshot showing iTunes backup and the C:\Users\Simon Fearby\Apple\MobileSync\Backup folder

The iTunes iPhone backup is now complete (it took about 2 hours).

Now that the iPhone was backed up iTunes the Operating System update started to run.

Screenshot of iTunes verifying the iPhone update

Update verified.

Screenshot of iTunes updating the iPhone OS.

During the update the iPhone was unavailable.

Screenshot of iPhone installing firmware

Done, the iPhone had updated it’s operating system in about 30 minutes.

Screenshot of iTunes showing the newer OS version.

Now lets see how much space I have available on my C Drive.

Screenshot of disk usage on my c and S drive.

It looks like the iPhone backup iTunes made was 60GB in size.

Screenshot of Windows reporting the backup was 60GB in size.

Using Windows Explorer I moved the backup from..

C:\Users\Simon Fearby\Apple\MobileSync\Backup

to..

S:\Backup\AlisonsiPhone\iTunes\31May2019

(right click drag an drop action in explorer)

Screenshot of right click drag and drop move folder.

It took me about 20 minutes to move the 60GB folder.

Screenshot of thew Windows copy dialog progress.

TIP: You can (should) also copy this backup folder to..

Now this backup is available for me to restore in the future if I need it.

In the case of a restore, I just need to move the backup into the iTunes expected location.

Copy from..

S:\Backup\AlisonsiPhone\iTunes\31May2019

..to..

C:\Users\Simon Fearby\Apple\MobileSync\Backup

The Catch

The catch with back-ups made with iTunes is they are useless if you wanted to restore individual files (say to find a photograph or video). iTunes backups are usually tens of thousands of files with random folders and filenames

Screenshot of iTunes nasty backup, i obfuscated all files to random guid filenames

If you want to JUST backup photos and videos read on.

How to backup just photos and videos from your iPhone.

This part of the guide needs a paid version of the iOS App Photo Transfer App. In Australia, the app is free but to turn it into a full version needed to restore everything (via in-app purchase) it cost $10.99.

This app is well worth $10.99 AUD (it may be cheaper in other countries) to have it push photos an videos from your iPhone to a free companion Windows program (a Mac versions exist too).

Screenshot of http://phototransferapp.com/

Buy the iOS version and install it on your iPhone.

Then download the free Windows version: http://phototransferapp.com/win/

Extract the files from the zip file.

I extracted PhotoTransferapp.exe from PhotoTransferapp.zip

When I run PhotoTransferApp.exe I get an error saying I need to install Adobe AIR run times.

Screenshot of me installing Adobe Air run times from https://get.adobe.com/air/

Go to https://get.adobe.com/air/ and download and install the Air run time.

Screenshot of https://get.adobe.com/air/

Install the Adobe Air run time.

Install Adobe Air From https://get.adobe.com/air/

I installed Adobe AIR and again reopened the PhotoTansferApp.exe and was prompted to allow access to the Windows Firewall.

I was on a home network so I enabled firewall access.

Screenshot of Photo Transfer App asking fore firewall access.

The Photo Transfer App prompted me to ‘Discover Devices‘ and to make sure the Transfer app is running on the iPhone.

Before I clicked ‘Discover Devices” on the Photo Transfer App on Windows I opened the Transfer app (mentioned above) on the iPhone.

On the iPhone I clicked ‘Send‘

The transfer app will ask for access to your photos, you will need to press OK.

Screenshot of the transfer app asking for permissions to photos

On the iPhone again I clicked ”Windows‘

The iOS Transfer app now said I should run the ‘Photo Transfer App‘ on Windows.

TIP: You can transfer over WiFi (if your iPhone and Windows device is on the same WiFi Network) or you can transfer over a USB cable.

Before I started the ‘Detect Device‘ or ‘transfer‘, I set the Backup path location to my S: drive by clicking settings in the bottom right and choosing a folder (as my C Drive is a bit small).

Screenshot of Photo Transfer App choosing a folder to backup to.

I noticed the port of 57777, I temporarily disabled the whole Windows Firewall just in case it prevents the photo backup.

In hindsight, it was not a good idea to disable the whole firewall, but because I was at home on a safe network I felt safe to do so. Next time I will not disable the firewall and see if this still works.

If you are on an internet cafe, school or university network do not disable your firewall.

I clicked ”Start” then typed ‘firewall‘ and clicked ‘Windows Defender Firewall‘

Screenshot Windows Defender firewall icon

I turned off my firewall.

I clicked ‘Detect Devices‘ and the iPhone ‘EllieRose‘ appeared on the left. I doubled clicked on the iPhone name and was prompted with an Authorization required message.

Screenshot: Authorisation required message.

I looked at the ‘Transfer‘ app on iOS and clicked ‘Yes, always‘ to allow access.

Screenshot: Authorisation yes/no on the iPhone

After 20 seconds I can see photos on my iPhone on Windows.

Screenshot, all of my iPhone images were appearing on Window.

I selected all camera albums to backup and clicked backup. I prompt to upgrade to the full version will appear, you will need to buy the upgrade.

Full backup happens only if you upgrade to a full version via an in app purchase

After the full version if purchased the backup will be allowed.

You can upgrade from the free to paid full version from the main screen of the Transfer app.

Screenshot get full access by upgrading the free app to a paid version

The upgrade In App Purchase is $10.99 AUD

Screenshot of a $10.99 In App Purchase on the iPhone

When you purchase the In App Purchase you can run the backup again.

Screenshot of all camera categories on the iPhone on windows

Now I could see photos being copied from the iPhone to my defined backup folder.

I went to bed as I knew there were about 100GB of files on the iPhone and this was going to take a while.

In the morning the backup was done.

26,000 files were backed up (over 100GB).

I now had a full iPhone backup made by iTunes and a copy of all photos and videos.

Screenshot showing the 2 backups (a) iTunes made and b) Photo Transfer App made)

I turned on the Firewall again.

I now had 160GB of backed up photos, videos and phone backup.

Backup your iPhone backups to the cloud

The steps to do this two ways will be added soon.

I will add a section on how you can back up the iTunes and manually synced photo and video backups to the cloud automatically and a more complex but cheaper was for 0.005c per GB.

Watch this post.

How to backup and Android Phone

Article coming soon.

How to backup and restore a MySQL database on Windows and Linux

April 21, 2019 by Simon

Why backup and restore

This is a quick guide demonstrating how you can backup and restore a MySQL database on Windows and Linux using Adminer.

You may need to know how to backup a restore a database for a number of reasons..

e.g

Send the database to someone to debug or give feedback while learning.
Move the database from a local machine to the cloud
Move the database from cloud vendor A to cloud vendor B
etc.

Having a backup of the VM is good but having a backup of the database too is better. I use UpCloud for hosting my VM’s and setting backups is easy. But I cannot download those backups.

Murphy’s Law

“If anything can go wrong, it will”

The most important reason for taking a backup and knowing how to restore it is for disaster recovery reasons.

Backup (the easiest way) with Adminer

Adminer is a free PHP based IDE for MySQL and other databases. Simply install Adminer and save the file on your local computer or remote web server directory.

FYI: The Adminer author Jakub Vrana has a patron page, I am a patron of this awesome software.

Snip from Adminers website. “Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch andMongoDB.”

TIP: The file would be publicly accessible to anyone so don’t save it to a common area, obfuscate the file, protect it of delete the file when you are done using it.

Once Adminer is installed load it in a web browser, login with your MySQL credentials. Once you login you will see all databases and an Import and Export menu.

Adminer main screen, all databases and import and export menu.

tbtest is a simple database with one table and 4 fields (ID, Key, Value and Modified)

.Click Export to open the export screen.

Export screen showing a list of databases and export options

Click Export, a SQL file will be generated (this is the export of the database).

Here is a save of the file:
https://fearby.com/wp-content/uploads/export.txt

Exported view of https://dev.mysql.com/doc/workbench/en/wb-admin-export-import-management.html

Its that simple.

If I add a binary blob file to the table and upload a PNG file lets see how the export looks.

Screenshot o the new table with a blog field in Adminer UI

Let export the database again in Adminer and check out the output. I used Sublime Text editor to view the export file.

New Export shows the binary file in the Backup SQL file

Restore (the easiest way) with Adminer

OK lets delete the tbtest database and then restore it with Adminer. I used Adminer to delete (DROP) the database.

Database “dbtest” deleted.

Now lets create a blank database to restore to (same name).

Database created.

Now lets import the database backup using Adminer.

Click Import, select the backup file and un-tick Stop on errors.

Import screenshot, dxtest selectded, Restore file selected, stop on errors disabled

TIP: The 2MB next the the choose file button is defined by your web server and PHP configuration. If you are trying to import a larger database (e.g 80MB) first increase the limits in your web server and PHP (via php.ini).

The Import (restore should take seconds)

The database was imported from a backup, all tables and records imported just fine.

Bonus methods.

On Ubuntu use this guide to backup from the command line. If you use the Oracle MySQL Workbench read this.

I hope this helps someone.

How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains

March 31, 2019 by Simon

This is a quick guide that will show you how you can connect to a cloud server via SFTP with the PHPStorm IDE from Jet Brians and deploy files from your localhost to the cloud. This is my opinion, I am not paid to promote PHPStorm or UpCloud.

Pre-Requisites/Assumptions

This guide will assume you already have (or know how to)..

Buy a domain name and point it’s DNS to a server (I use Namecheap.com for buying domains)
Buy and deploy a server in the cloud. I have used AWS, Digital Ocean, Vultr but now use UpCloud for deploying fast self-managed servers. Read this guide here to see how I create a server from scratch on Up Cloud.
Setup SSH access to your server and configure a firewall.
You have or know how to set up PHP and Web Servers and configure them on your localhost and remote server (guides here, here, here, here, here and here).
etc ( check out all my guides here https://fearby.com/all )

I am using Windows 10 Home (with IIS Web Server (document root redirected to S:\Code\) and a pre built Ubuntu Cloud servers.

Why no FTP? I do not create FTP servers on my serves to increase security and I only access servers via SSH via white-listed IP’s and then authenticate with hardware 2FA keys from YubiCo (read me 2FA guide here and also how to secure *nix servers and WordPress with 2FA).

Background

2 years ago I used to use the Cloud 9 IDE to connect to, and code files on cloud servers and life was good. I could configure and connect to servers, drag and drop files, run bash scripts from a web page and close the Cloud 9 browsers tabs, travel hundreds of kilometres and log back into C9 and all code and bash scripts would reappear.

Here is the Cloud 9 IDE showing code on the left and a Browser on the right.

C9.IDE showing code on the left and web page on the right

With Cloud 9 code could be easily accessed, edited and run.

See screenshot of code running from a Cloud 9 hosted server with properties windows on the right.

C9 IDE showing a bash terminal windows and code

Regrettably, I cancelled my $9/m subscription to Cloud 9 after a minor stroke and since then I have gone back to a terminal screen to code and transfer files. In the last 2 years.

Screenshot of the putty program connected to a Ubuntu box editing a file with nano

Uploading and downloading files on mass is painful via pure SSH.

AWS has since purchased Cloud 9 and I am not sure if it will ditch support of non-AWS servers in the future. AWS is good but servers are very expensive for what you get IMHO. I have found Disk IO on UpCloud is awesome (also UpCloud support is great (I am not paid to say that)).

PHPStorm IDE?

A quick Google of IDE’s like Cloud 9 mentioned PHPStorm from Jet Brains. I have used IntelliJ IDEA from Jet Brains before and PHPStorm seems to be very popular.

Go to
https://www.jetbrains.com/phpstorm/ and see the features of PHPStorm

Watch PHPStorm in action

Whats new in PHPStorm 2019.1

Install PHPStorm

Visit https://www.jetbrains.com/phpstorm/download/ and download and install PHPStorm (free trial 30 days)

System requirements

Microsoft 10/8/7/Vista/2003/XP (incl. 64-bit)
2 GB RAM minimum
4 GB RAM recommended
1024×768 minimum screen resolution

Pricing

PHPStorm is $8.90/m for individual use (or $19.90/m commercial). For the first 12 months of uninterrupted subscription payments qualify you for receiving a perpetual fallback license (20% discount for an uninterrupted subscription for a 2nd year, 40% discount for an uninterrupted subscription for 3rd year onwards).

PHPStorm work on Windows, OSX or Linux. This great an I use Windows locally and Linux remotely but I’m keen to use Linux locally to match local and remote dev environments.

Official PHPStorm Pricing Page:
https://www.jetbrains.com/phpstorm/buy/#edition=commercial

fyi: Jet Brains has free licencing for individual use for Students and faculty members.

Creating a Project

Open PHPStorm and select “Create New Project”.

Choose a project type on the left (e.g “PHP Empty Project“) and choose a location to save too (I chose “S:\code\php001) on my local machine. I chose “S:\code\php001” on my local machine.

New Project screenshot asking for a name and save location

Choose a folder to save to.

Choose a project and location to save to locally

Click “OK” to create the project.

PHPStorm will have created a project for you. You will notice a “.idea“folder under the location you saved with these files.

misc.xml
modules.xml
php001.xml
wordspace.xml

Do not delete these files.

Creating your first PHP file

You can right click on the project root and select New then PHP File

Right click on the root in the tree view then new PHP File

Or clicking the File then New menu choosing PHP File.

Name the file (e.g index.php)

The file has been created and its available in my localhost web server.

Screenshot showing PHPStorm with index.php, S:\Code showing index.php and http://localhpst/php001/index.php loading

Creating a Deploy Target

Now we need to specify a deploy target in PHPStorm to push the file changes to the cloud. Backup your server (yes backup your server just in case).

Open your PHPStorm project and click Tools, Deployment and then Configuration.

Click the plus icon near the top left and choose SFTP

Screenshot showing add new deployment server (SFTP)

Name the deployment target (e.g “server (project)”)

Screenshot of an input box showing a server name

Enter your “server name” or IP and port
Enter your “ssh username” (ensure the SSH user had write access to the wwwroot folder and the web server can read the files written by this user)
Under password I chose “Key pair OpenSSH or Putty (as I had SSH details already setup in Putty details
You can add your ppk private key from Putty (use the puttygen program to conbvert ssh public and private kets to ppk format)
If you have a passphrase on your SSH key add it now
Enter your web servers remote path (for the project)
Enter your web server URL

Screenshot showing a server name, port, username, password, ssh file passphrase, root path and web server url.

I did SSH to my remote server and created the destination folder. This will ensure I can deploy code here (PHPStorm does not create the remote path fpor you ).

mkdir /wwwroot/php001
chown -R www-data:www-data /wwwroot/php001/

Click Test Connection

No we need to click the Mappings tab and add a mapping.

Local path is your local path
Deployment path is / (the web root path is carried forward from the previous tab)
Web path is the web path that is entered in the browser

Screenshot showing a manual file mapping of local and remote file locations

Click Add New Mapping. Now we are ready to deploy

Deploying code to the cloud

I right clicked on the root note in PHPStorm and created an index.php file.

I edited the index.php on my local machine and then click the Tools then Deployment and choose “Upload to fearby.com (php001)” menu.

Manual upload available in Tools menu then Deployment menu

The File Transfer output window showed the transfer progress.

Screenshot showing the file transfer window output saying the file uploaded.

I loaded https://fearby.com/php001/index.php in Google chrome. It worked.

Screenshot showing https://fearby.com/php001/index.php loaded in a bowser

Don’t forget to turn off Automatic uploads under Tools, Deployment menu.

'Screenshot showing Automatic updated turned on

Now when I create new files or change existing files they will auto upload.

Sourc Control

I will add this soon.

Shell Command

You can also open an SSH console to the server and run commands

e.g zip files

zip -r backup.zip .

I can also open a folder window in PHPStorm and show all remote files by clicking Tools then Deployment then Show Remote Files, Zip files can be easily downlaoded or other files uploaded. Nice.

Linux Client

I will review the Linux PHPStorm client soon.

Troubleshooting

Watch the Official guide on Deployment and Remote Hosts in PhpStorm – PhpStorm Video Tutorial

Good Luck. I hope this guide helps someone.

Version

1.2 Removed advertisements

1.1 Minor Updates

1.0 Initial Version

I am moving away from Apple hardware

February 3, 2019 by Simon

My Late 2012 Mac Book Pro Retina laptop is all but dead, it has many dead pixels and because of the poor cooling and is NOT a joy to use anymore. It does not “JUST WORK” and personally, I do not think “thinner” laptops can handle Australian summers as its hardware cooling it inadequate above 40c air temperatures.

My laptop processor would spend more time thermal throttling (at 104c) in Web Browsers and text editors that at normal speeds. Opening up productivity apps like Photoshop or Premiere Pro would send the laptop into meltdown.

Frequent high temps were common.

Attempted Fixes

Warning Disclaimer: My laptop is out of warranty and I know my way around the inside of a computer hardware without zapping it. Do not attempt to open your laptop unless you know what you are doing, have backed up your data and are prepared to brick your computer.

I removed dust from inside the laptop.
I tried to only use the laptop refrigerative air conditioning
I replaced the thermal paste on the CPU and GPU (3 times)
I reinstalled OSX Mojave and reset the SMC and PRAM multiple times.
I ran the fans at 100% (see post here), The fans were operating at full capacity and were not broken.

The stock thermal paste was crusty after 5 years. The plastic CPU/GPU cover was visibly cooked.

I ordered some new Thermal grizzly thermal paste, I had some older silicone paste on hand just in case.

After many reapplications of the Thermal Grizzly, the older silicone paste seemed to work the better???

After a few months, all of the fixes above did not seem to work. OSX Mojave would spin up the CPU and GPU into a frenzy overloading the single heat pipe within minutes.

Time to try some more drastic cooling modifications?

I tried improving the efficiency of the single (copper) heat pipe that is shared between an Intel i7 2.6 GHz and an Nvidia Video Card by removing the black paint by stripping the paint with acetone.

I manually removed paint from in between the heat sink fins with a LED to reveal the metal.

I reinstalled the heat pipe with high hopes? That looks nice 🙂

I removed the old thermal paste and added new paste. First I tried Thermal Grizzly Cryonaut. I re-applied the paste three separate times as each application was not that much better than the old crusty stock paste from Apple. Did I have a bad batch of Thermal Grizzly?, It seemed thick and not very viscous. I ended up using an old tube of silicone paste (the white stuff) as my Arctic Silver was too old to try and I did not want to order more.

With the silicone paste applied and the paint removed temperatures were about 15c lower at max, I still had frequent thermal throttling but at least I had a reserve buffer.

This was all before the Aussie Heatwaves and high temperatures soon returned.

Is there still room for improvement?

How heat pipes work

Heat pipes have an evaporating (hot part) and condensing zone (cool part) on the heat pipe. I noticed Apple’s “stock” condensing fins were small, would improving this zone help?. Time to improve the condensers zones by adding larger copper heat sinks to the bare side of the heat pipe.

I purchased a few copper Xeon/Sun server sized heat sinks and thermal epoxied them to the condensing end of the heat pipe. Yes, they would protrude out the bottom of the case but #Meh. I can fix that by extending the base of the laptop down and making it thicker (old school style).

The server heat sinks arrived

I cut the heat sinks in half.

I packed the fins with paper before cutting to ensure the cut did not damage the fins.

After cutting, I wiped the copper heat sinks with vinegar to restore the surface to a nice copper shine.

I tested the heat sink idea with silicone paste first

Temps were 25c lower, Now it’s time to use Arctic Silver Thermal Epoxy

I applied the Thermal Epoxy to the heat pipe (I temporary had foil strips above the fans so I did not block them while the epoxy dried.

I then stuck the heat sink’s to the heat pipe (with Arctic Silver Thermal Epoxy).

I toyed with a clear case but decided against it for static electricity and stability reasons.

I purchased a second Mac Book base for so I could cut holes for the heat sinks to protrude and use the original base to hide the modification.

I made a 30 mm base wall to so I could use it as a wall between the laptop base and the new 30mm lower base.

I added some 5-volt and 12 -volt fans inside the new extended 30mm base.

Finished Product

A normal looking Mac Book except for the 30mm lower base and internal 5V or 12V fans.

External power plugs on the left side, I will add lights at a later stage.

Are the temps lower?

Videos

Video: Mac Book Pro cooling mod, I can now watch 1080p videos without maxing the CPU

Video: Mac Book Pro cooling mod with external powered 5v or 12v fans

Conclusion

50c lower temps are nicer at idle but in Premiere Pro (exporting video) the laptop was still thermal throttling like mad and temps were terrible (100+). Lets not get started when I start some development VM’s

Conclusion 2 weeks later

This is still not a joy to use. I don’t think I have the right to expect a 5-year-old laptop to keep up running a CPU/GPU intensive OS and applications.

Time to buy a new computer, Apple still makes thin and over heating laptops by the looks of it?

Maybe I need to buy a fridge to stick a computer in a fridge to use these days?

YouTube users indicate Apple has a problem with heat.

What computer do I get next?

Not an Apple made one. I will be moving back to Windows for local development and Linux on servers

Dell Alienware has many heat pipes.

Acer Predator 500

I read a few reviews (e.g this one from Ultra book reviews) and Acer have good cooling.

MSI GT Series laptops look the best if cooling is important.

Or should I build a custom desktop with way more cores

CPU: Threadripper 2950X 16C 32T

SSD: M.2 SSD: Samsung 970 PRO 512GB
MOBO: Asus Zenith Extreme
Power: Corsair RM1000x 1000W
MEM: Quad 3600 Mhz
GPU: AMD Radeon VII Navi 3980

Thanks for reading.

v1.3 Added videos

v1.2 Updated alt tag descriptions

v1.1 Added “I will be moving back to Windows for local development and Linux on servers”

1.0 Initial Draft

No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

August 5, 2018 by Simon

No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Aside

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Now on with the post.

Murphy’s Law

I recently had an issue where I set up a website for a friend. I invested 6 hours into setting up..

I setup…

Debian OS
NGINX Web Server
MySQL Database
PHP 7.2
PHP-FPM Child Workers
HTTPS Certificate
Security (Firewall/Headers/SSH, WordPress, Plugins etc).
Installed WordPress and Plugins
Setup DNSSEC
Etc

I had tested GTMetrix scores = less than 1 second. Security headers were tested and I was happy with the site.

The server and backups were automatically deleted after 7 days while I was away from my keyboard because I assumed the account was valid and had credits.

Lesson Learned

Always have a backup (of the server, setup/ www, MySQL etc).
Script setups (Ansible, Puppet or Scripts) to sate time redeploying if need be.
Backups are not always available.
Do have setup documented (Check)
Do have a disaster plan

I have guides on setting up a server on UpCloud, AWS, Vultr, Digital Ocean but setting up can be rather repetitive so how can you prevent resetting up servers?

Why Plan for the Worst

Companies disappear.
Some hosts go down.
Some hosts have weird trial modes and internal process that could take your site down.
Human error?
Murphy’s Law

How I will prevent this in future

I am building a Java desktop app for Windows/OSX/Linux app that will deploy and set up on UpCloud/Vultr/Digital Ocean providers and allow for 1 click deploy and backup and restore.
I am going to re-establish replication between servers with RSync etc.
I am going to start to automate installs and environments.
I am going to set up a hot (ready to go) Green/Blue mirrored environments (www and DB server) on different providers in case of emergency. Then I can set the active live servers with DNS (blog posts soon).
Consider a server farm (same provider or different providers)

I hope this guide helps someone.

Please consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.0 Initial Post

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

June 25, 2018 by Simon

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

If you have not read my previous posts I have now moved my blog from Vultr to the awesome UpCloud host (signup using this link to get $25 free credit).

Recently I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Spoiler: UpCloud performance is great.

Why Upgrade

I have 1 CPU, 1 GB memory and 50GB storage and it is running well? I have PHP child workers set up and have set up the preferred use of memory over swap file usage.

Before UpCloud, when I had 512MB ram on Vultr I had multiple NGINX crashed a day so I used a bash script and scheduled a cron job to clear memory cache when memory fell below 100MB (view the script here). To further increase the speed of the WordPress I have configured the OS to use memory over the disk. About once a day free memory falls below 100MB (this is not a problem as my script clears cached items automatically).

I’d like to add more memory as I am working on some things (watch this space) and I will use the extra memory. I’d prefer the server is set up now for the expected workload.

How to Upgrade

This is how I upgraded from 1xCPU/2GB Memory/50GB Storage/2TB Transferred Data to a 2 CPU/4GB Mmeory/80GB Storage/4TB Transferred Data server.

UpCloud Pricing: https://www.upcloud.com/pricing/

Upgrade an UpCloud VM

I shut down my existing VM. Read this guide to setup a VM.

shutdown -P

Login to the UpCloud dashboard, select your server (confirm the server has shut down) and click General Settings, choose the upgrade and click Update.

I confirmed the upgrade options (2x CPU, 4096 MB Memory).

Click Update

After 10 seconds you can start your server from the UpCloud server.

I confirmed the CPU upgrade was visible in the VM

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
stepping        : 2
microcode       : 0x1
cpu MHz         : 3099.978
cache size      : 16384 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 6199.95
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
stepping        : 2
microcode       : 0x1
cpu MHz         : 3099.978
cache size      : 16384 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 6199.95
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Software Tweaks Post Upgrade.

I added these settings to the top of /etc/nginx/nginx.conf to ensure the extra CPU was used.

worker_processes auto;
worker_cpu_affinity auto;

I increased PHP FPM ( /etc/php/7.2/fpm/php.ini ) to increase the memory usage and child workers. I doubled child workers and max memeory limit.

memory_limit = 3072M
pm.max_children = 80
pm.start_servers = 30
pm.min_spare_servers = 10
pm.max_spare_servers = 30

I restarted NGINX and PHP

nginx -t
nginx -s reload
/etc/init.d/nginx restart
service php7.2-fpm restart

I tweaked WordPress max memory limits

define( 'WP_MEMORY_LIMIT','3072M');
define( 'WP_MAX_MEMORY_LIMIT','3072M');

MySQL Tweaks: I logged into MySQL

mysql -u root -p

I ran “SHOW GLOBAL STATUS” to view stats

mysql> SHOW GLOBAL STATUS;
+-----------------------------------------------+--------------------------------------------------+
| Variable_name                                 | Value                                            |
+-----------------------------------------------+--------------------------------------------------+
| Aborted_clients                               | 0                                                |
| Aborted_connects                              | 0                                                |
| Binlog_cache_disk_use                         | 0                                                |
| Binlog_cache_use                              | 0                                                |
| Binlog_stmt_cache_disk_use                    | 0                                                |
| Binlog_stmt_cache_use                         | 0                                                |
| Bytes_received                                | 3179986                                          |
| Bytes_sent                                    | 223872114                                        |
| Com_admin_commands                            | 0                                                |
| Com_assign_to_keycache                        | 0                                                |
| Com_alter_db                                  | 0                                                |
| Com_alter_db_upgrade                          | 0                                                |
| Com_alter_event                               | 0                                                |
| Com_alter_function                            | 0                                                |
| Com_alter_instance                            | 0                                                |
| Com_alter_procedure                           | 0                                                |
| Com_alter_server                              | 0                                                |
| Com_alter_table                               | 0                                                |
| Com_alter_tablespace                          | 0                                                |
| Com_alter_user                                | 0                                                |
| Com_analyze                                   | 0                                                |
| Com_begin                                     | 0                                                |
| Com_binlog                                    | 0                                                |
| Com_call_procedure                            | 0                                                |
| Com_change_db                                 | 284                                              |
| Com_change_master                             | 0                                                |
| Com_change_repl_filter                        | 0                                                |
| Com_check                                     | 0                                                |
| Com_checksum                                  | 0                                                |
| Com_commit                                    | 0                                                |
| Com_create_db                                 | 0                                                |
| Com_create_event                              | 0                                                |
| Com_create_function                           | 0                                                |
| Com_create_index                              | 0                                                |
| Com_create_procedure                          | 0                                                |
| Com_create_server                             | 0                                                |
| Com_create_table                              | 0                                                |
| Com_create_trigger                            | 0                                                |
| Com_create_udf                                | 0                                                |
| Com_create_user                               | 0                                                |
| Com_create_view                               | 0                                                |
| Com_dealloc_sql                               | 0                                                |
| Com_delete                                    | 18                                               |
| Com_delete_multi                              | 0                                                |
| Com_do                                        | 0                                                |
| Com_drop_db                                   | 0                                                |
| Com_drop_event                                | 0                                                |
| Com_drop_function                             | 0                                                |
| Com_drop_index                                | 0                                                |
| Com_drop_procedure                            | 0                                                |
| Com_drop_server                               | 0                                                |
| Com_drop_table                                | 0                                                |
| Com_drop_trigger                              | 0                                                |
| Com_drop_user                                 | 0                                                |
| Com_drop_view                                 | 0                                                |
| Com_empty_query                               | 0                                                |
| Com_execute_sql                               | 0                                                |
| Com_explain_other                             | 0                                                |
| Com_flush                                     | 0                                                |
| Com_get_diagnostics                           | 0                                                |
| Com_grant                                     | 0                                                |
| Com_ha_close                                  | 0                                                |
| Com_ha_open                                   | 0                                                |
| Com_ha_read                                   | 0                                                |
| Com_help                                      | 0                                                |
| Com_insert                                    | 342                                              |
| Com_insert_select                             | 0                                                |
| Com_install_plugin                            | 0                                                |
| Com_kill                                      | 0                                                |
| Com_load                                      | 0                                                |
| Com_lock_tables                               | 0                                                |
| Com_optimize                                  | 0                                                |
| Com_preload_keys                              | 0                                                |
| Com_prepare_sql                               | 0                                                |
| Com_purge                                     | 0                                                |
| Com_purge_before_date                         | 0                                                |
| Com_release_savepoint                         | 0                                                |
| Com_rename_table                              | 0                                                |
| Com_rename_user                               | 0                                                |
| Com_repair                                    | 0                                                |
| Com_replace                                   | 0                                                |
| Com_replace_select                            | 0                                                |
| Com_reset                                     | 0                                                |
| Com_resignal                                  | 0                                                |
| Com_revoke                                    | 0                                                |
| Com_revoke_all                                | 0                                                |
| Com_rollback                                  | 0                                                |
| Com_rollback_to_savepoint                     | 0                                                |
| Com_savepoint                                 | 0                                                |
| Com_select                                    | 16358                                            |
| Com_set_option                                | 849                                              |
| Com_signal                                    | 0                                                |
| Com_show_binlog_events                        | 0                                                |
| Com_show_binlogs                              | 0                                                |
| Com_show_charsets                             | 0                                                |
| Com_show_collations                           | 0                                                |
| Com_show_create_db                            | 0                                                |
| Com_show_create_event                         | 0                                                |
| Com_show_create_func                          | 0                                                |
| Com_show_create_proc                          | 0                                                |
| Com_show_create_table                         | 0                                                |
| Com_show_create_trigger                       | 0                                                |
| Com_show_databases                            | 3                                                |
| Com_show_engine_logs                          | 0                                                |
| Com_show_engine_mutex                         | 0                                                |
| Com_show_engine_status                        | 0                                                |
| Com_show_events                               | 0                                                |
| Com_show_errors                               | 0                                                |
| Com_show_fields                               | 921                                              |
| Com_show_function_code                        | 0                                                |
| Com_show_function_status                      | 0                                                |
| Com_show_grants                               | 0                                                |
| Com_show_keys                                 | 1                                                |
| Com_show_master_status                        | 0                                                |
| Com_show_open_tables                          | 0                                                |
| Com_show_plugins                              | 0                                                |
| Com_show_privileges                           | 0                                                |
| Com_show_procedure_code                       | 0                                                |
| Com_show_procedure_status                     | 0                                                |
| Com_show_processlist                          | 0                                                |
| Com_show_profile                              | 0                                                |
| Com_show_profiles                             | 0                                                |
| Com_show_relaylog_events                      | 0                                                |
| Com_show_slave_hosts                          | 0                                                |
| Com_show_slave_status                         | 0                                                |
| Com_show_status                               | 6                                                |
| Com_show_storage_engines                      | 0                                                |
| Com_show_table_status                         | 0                                                |
| Com_show_tables                               | 2                                                |
| Com_show_triggers                             | 0                                                |
| Com_show_variables                            | 6                                                |
| Com_show_warnings                             | 1                                                |
| Com_show_create_user                          | 0                                                |
| Com_shutdown                                  | 0                                                |
| Com_slave_start                               | 0                                                |
| Com_slave_stop                                | 0                                                |
| Com_group_replication_start                   | 0                                                |
| Com_group_replication_stop                    | 0                                                |
| Com_stmt_execute                              | 4                                                |
| Com_stmt_close                                | 4                                                |
| Com_stmt_fetch                                | 0                                                |
| Com_stmt_prepare                              | 4                                                |
| Com_stmt_reset                                | 0                                                |
| Com_stmt_send_long_data                       | 4                                                |
| Com_truncate                                  | 2                                                |
| Com_uninstall_plugin                          | 0                                                |
| Com_unlock_tables                             | 0                                                |
| Com_update                                    | 70                                               |
| Com_update_multi                              | 0                                                |
| Com_xa_commit                                 | 0                                                |
| Com_xa_end                                    | 0                                                |
| Com_xa_prepare                                | 0                                                |
| Com_xa_recover                                | 0                                                |
| Com_xa_rollback                               | 0                                                |
| Com_xa_start                                  | 0                                                |
| Com_stmt_reprepare                            | 0                                                |
| Connection_errors_accept                      | 0                                                |
| Connection_errors_internal                    | 0                                                |
| Connection_errors_max_connections             | 0                                                |
| Connection_errors_peer_address                | 0                                                |
| Connection_errors_select                      | 0                                                |
| Connection_errors_tcpwrap                     | 0                                                |
| Connections                                   | 292                                              |
| Created_tmp_disk_tables                       | 1124                                             |
| Created_tmp_files                             | 30                                               |
| Created_tmp_tables                            | 1369                                             |
| Delayed_errors                                | 0                                                |
| Delayed_insert_threads                        | 0                                                |
| Delayed_writes                                | 0                                                |
| Flush_commands                                | 1                                                |
| Handler_commit                                | 6094                                             |
| Handler_delete                                | 33                                               |
| Handler_discover                              | 0                                                |
| Handler_external_lock                         | 38571                                            |
| Handler_mrr_init                              | 0                                                |
| Handler_prepare                               | 0                                                |
| Handler_read_first                            | 2299                                             |
| Handler_read_key                              | 134761                                           |
| Handler_read_last                             | 237                                              |
| Handler_read_next                             | 310119                                           |
| Handler_read_prev                             | 2733                                             |
| Handler_read_rnd                              | 222350                                           |
| Handler_read_rnd_next                         | 472820                                           |
| Handler_rollback                              | 0                                                |
| Handler_savepoint                             | 0                                                |
| Handler_savepoint_rollback                    | 0                                                |
| Handler_update                                | 15605                                            |
| Handler_write                                 | 17310                                            |
| Innodb_buffer_pool_dump_status                | Dumping of buffer pool not started               |
| Innodb_buffer_pool_load_status                | Buffer pool(s) load completed at 180624 23:38:01 |
| Innodb_buffer_pool_resize_status              |                                                  |
| Innodb_buffer_pool_pages_data                 | 1035                                             |
| Innodb_buffer_pool_bytes_data                 | 16957440                                         |
| Innodb_buffer_pool_pages_dirty                | 0                                                |
| Innodb_buffer_pool_bytes_dirty                | 0                                                |
| Innodb_buffer_pool_pages_flushed              | 1936                                             |
| Innodb_buffer_pool_pages_free                 | 7144                                             |
| Innodb_buffer_pool_pages_misc                 | 13                                               |
| Innodb_buffer_pool_pages_total                | 8192                                             |
| Innodb_buffer_pool_read_ahead_rnd             | 0                                                |
| Innodb_buffer_pool_read_ahead                 | 0                                                |
| Innodb_buffer_pool_read_ahead_evicted         | 0                                                |
| Innodb_buffer_pool_read_requests              | 306665                                           |
| Innodb_buffer_pool_reads                      | 950                                              |
| Innodb_buffer_pool_wait_free                  | 0                                                |
| Innodb_buffer_pool_write_requests             | 26509                                            |
| Innodb_data_fsyncs                            | 1229                                             |
| Innodb_data_pending_fsyncs                    | 0                                                |
| Innodb_data_pending_reads                     | 0                                                |
| Innodb_data_pending_writes                    | 0                                                |
| Innodb_data_read                              | 16273920                                         |
| Innodb_data_reads                             | 1078                                             |
| Innodb_data_writes                            | 2857                                             |
| Innodb_data_written                           | 53379584                                         |
| Innodb_dblwr_pages_written                    | 1275                                             |
| Innodb_dblwr_writes                           | 109                                              |
| Innodb_log_waits                              | 0                                                |
| Innodb_log_write_requests                     | 450                                              |
| Innodb_log_writes                             | 585                                              |
| Innodb_os_log_fsyncs                          | 793                                              |
| Innodb_os_log_pending_fsyncs                  | 0                                                |
| Innodb_os_log_pending_writes                  | 0                                                |
| Innodb_os_log_written                         | 664064                                           |
| Innodb_page_size                              | 16384                                            |
| Innodb_pages_created                          | 56                                               |
| Innodb_pages_read                             | 988                                              |
| Innodb_pages_written                          | 1936                                             |
| Innodb_row_lock_current_waits                 | 0                                                |
| Innodb_row_lock_time                          | 0                                                |
| Innodb_row_lock_time_avg                      | 0                                                |
| Innodb_row_lock_time_max                      | 0                                                |
| Innodb_row_lock_waits                         | 0                                                |
| Innodb_rows_deleted                           | 2                                                |
| Innodb_rows_inserted                          | 19219                                            |
| Innodb_rows_read                              | 249102                                           |
| Innodb_rows_updated                           | 77                                               |
| Innodb_num_open_files                         | 81                                               |
| Innodb_truncated_status_writes                | 0                                                |
| Innodb_available_undo_logs                    | 128                                              |
| Key_blocks_not_flushed                        | 0                                                |
| Key_blocks_unused                             | 12751                                            |
| Key_blocks_used                               | 645                                              |
| Key_read_requests                             | 321877                                           |
| Key_reads                                     | 648                                              |
| Key_write_requests                            | 196                                              |
| Key_writes                                    | 150                                              |
| Locked_connects                               | 0                                                |
| Max_execution_time_exceeded                   | 0                                                |
| Max_execution_time_set                        | 0                                                |
| Max_execution_time_set_failed                 | 0                                                |
| Max_used_connections                          | 3                                                |
| Max_used_connections_time                     | 2018-06-24 23:43:48                              |
| Not_flushed_delayed_rows                      | 0                                                |
| Ongoing_anonymous_transaction_count           | 0                                                |
| Open_files                                    | 229                                              |
| Open_streams                                  | 0                                                |
| Open_table_definitions                        | 206                                              |
| Open_tables                                   | 786                                              |
| Opened_files                                  | 502                                              |
| Opened_table_definitions                      | 208                                              |
| Opened_tables                                 | 817                                              |
| Performance_schema_accounts_lost              | 0                                                |
| Performance_schema_cond_classes_lost          | 0                                                |
| Performance_schema_cond_instances_lost        | 0                                                |
| Performance_schema_digest_lost                | 0                                                |
| Performance_schema_file_classes_lost          | 0                                                |
| Performance_schema_file_handles_lost          | 0                                                |
| Performance_schema_file_instances_lost        | 0                                                |
| Performance_schema_hosts_lost                 | 0                                                |
| Performance_schema_index_stat_lost            | 0                                                |
| Performance_schema_locker_lost                | 0                                                |
| Performance_schema_memory_classes_lost        | 0                                                |
| Performance_schema_metadata_lock_lost         | 0                                                |
| Performance_schema_mutex_classes_lost         | 0                                                |
| Performance_schema_mutex_instances_lost       | 0                                                |
| Performance_schema_nested_statement_lost      | 0                                                |
| Performance_schema_prepared_statements_lost   | 0                                                |
| Performance_schema_program_lost               | 0                                                |
| Performance_schema_rwlock_classes_lost        | 0                                                |
| Performance_schema_rwlock_instances_lost      | 0                                                |
| Performance_schema_session_connect_attrs_lost | 0                                                |
| Performance_schema_socket_classes_lost        | 0                                                |
| Performance_schema_socket_instances_lost      | 0                                                |
| Performance_schema_stage_classes_lost         | 0                                                |
| Performance_schema_statement_classes_lost     | 0                                                |
| Performance_schema_table_handles_lost         | 0                                                |
| Performance_schema_table_instances_lost       | 0                                                |
| Performance_schema_table_lock_stat_lost       | 0                                                |
| Performance_schema_thread_classes_lost        | 0                                                |
| Performance_schema_thread_instances_lost      | 0                                                |
| Performance_schema_users_lost                 | 0                                                |
| Prepared_stmt_count                           | 0                                                |
| Qcache_free_blocks                            | 1                                                |
| Qcache_free_memory                            | 16760152                                         |
| Qcache_hits                                   | 0                                                |
| Qcache_inserts                                | 0                                                |
| Qcache_lowmem_prunes                          | 0                                                |
| Qcache_not_cached                             | 16355                                            |
| Qcache_queries_in_cache                       | 0                                                |
| Qcache_total_blocks                           | 1                                                |
| Queries                                       | 19164                                            |
| Questions                                     | 19155                                            |
| Select_full_join                              | 0                                                |
| Select_full_range_join                        | 0                                                |
| Select_range                                  | 2677                                             |
| Select_range_check                            | 0                                                |
| Select_scan                                   | 2098                                             |
| Slave_open_temp_tables                        | 0                                                |
| Slow_launch_threads                           | 0                                                |
| Slow_queries                                  | 0                                                |
| Sort_merge_passes                             | 12                                               |
| Sort_range                                    | 4859                                             |
| Sort_rows                                     | 244452                                           |
| Sort_scan                                     | 854                                              |
| Ssl_accept_renegotiates                       | 0                                                |
| Ssl_accepts                                   | 0                                                |
| Ssl_callback_cache_hits                       | 0                                                |
| Ssl_cipher                                    |                                                  |
| Ssl_cipher_list                               |                                                  |
| Ssl_client_connects                           | 0                                                |
| Ssl_connect_renegotiates                      | 0                                                |
| Ssl_ctx_verify_depth                          | 0                                                |
| Ssl_ctx_verify_mode                           | 0                                                |
| Ssl_default_timeout                           | 0                                                |
| Ssl_finished_accepts                          | 0                                                |
| Ssl_finished_connects                         | 0                                                |
| Ssl_server_not_after                          |                                                  |
| Ssl_server_not_before                         |                                                  |
| Ssl_session_cache_hits                        | 0                                                |
| Ssl_session_cache_misses                      | 0                                                |
| Ssl_session_cache_mode                        | NONE                                             |
| Ssl_session_cache_overflows                   | 0                                                |
| Ssl_session_cache_size                        | 0                                                |
| Ssl_session_cache_timeouts                    | 0                                                |
| Ssl_sessions_reused                           | 0                                                |
| Ssl_used_session_cache_entries                | 0                                                |
| Ssl_verify_depth                              | 0                                                |
| Ssl_verify_mode                               | 0                                                |
| Ssl_version                                   |                                                  |
| Table_locks_immediate                         | 11962                                            |
| Table_locks_waited                            | 0                                                |
| Table_open_cache_hits                         | 19395                                            |
| Table_open_cache_misses                       | 817                                              |
| Table_open_cache_overflows                    | 12                                               |
| Tc_log_max_pages_used                         | 0                                                |
| Tc_log_page_size                              | 0                                                |
| Tc_log_page_waits                             | 0                                                |
| Threads_cached                                | 2                                                |
| Threads_connected                             | 1                                                |
| Threads_created                               | 3                                                |
| Threads_running                               | 1                                                |
| Uptime                                        | 2944                                             |
| Uptime_since_flush_status                     | 2944                                             |
+-----------------------------------------------+--------------------------------------------------+
353 rows in set (0.00 sec)

Read more on SHOW GLOBAL STATUS here. Read more on the values here.

I can see NO major errors here (possibly due to UpClouds awesome disk IO) so I won’t be making memory tweaks in MySQL. Sign Up using this link and get $25 credit free on UpCloud and see for yourself how fast they are.

Configure Ubuntu System Memory Usage

Edit /etc/sysctl.conf

Add the following to allow things to sit in ram longer

vm.vfs_cache_pressure=50

Snip from: https://www.kernel.org/doc/Documentation/sysctl/vm.txt

This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim.  Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are."

Read these pages here and here regarding setting MySQL memory.

Reboot

shutdown -r now

Resize the disk

The 2x CPU, 4GB memory plan comes with 80GB storage allowance. My disk at present is 50GB and I will update the size soon following this guide soon.

Quick Benchmark

I used loader.io to load 500 users to access my site in 1 minute.

The benchmark worked with no errors.

This benchmark was performed with no Cloudflare caching. I should get Cloudflare caching working again to lower the average response time. I loaded my website manually in Google Chrome while loader.io threw 500 users at my site and it loaded very fast.

Conclusion

After a few days, I checked my memory logs and there were no low memory triggers (just normal internal memory management triggers). Ubuntu was happier.

This graph was taken before I set “vm.vfs_cache_pressure” so I will update this graph in a few days.

I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

1.0 Initial Post

Backup

Post Index

1. NextCloud System Requirements

2. Creating a new Virtual Machine at UpCloud

UpCloud Chat Support

Configuring the server with Putty

3. Updating Ubuntu

Confirming the 2x storage disks

4. Installing Common Software Packages

5. Securing SSH with the Google Authentication PAM module

6. Installing a Firewall

7. Installing NGINX and DNS

8. Installing PHP/PHP-FPM

9. Installing MySQL

10. Nixstats

11. CronTab Updates

12. Misc Security Stuff

Installing NextCloud

Misc Setup

Nextcloud External Security Scan

Adding Two-Factor Authentication (YubiKeys)

Nextcloud Overview

Syncing Files from my PC to Nextcloud

Nextcloud Windows App

Nextcloud Conclusion

Troubleshooting

Deleting a MySQL Database

SUMo to the Rescue

SUMo Trial Limitations

SUMo Pro Scan

SUMo Settings

Getting Application Updates

Updating Software

Checking Application Update Status

SUMo Right Click Actions

After a few hours updating apps

All Apps Updated

Export

Conclusion

Saving a dollar

Starting Acronis True Image 2020

Creating a Backup

Backup Destination

File Musical Chair’s

Now that I have 800GB free I can backup up my C:\ Disk

Backup in Progress

Creating Rescue (Recovery) DVD Media?

Creating Rescue (Recovery) USB Media?

Restoring a backup from the DVD Rescue Media

Restoring a backup from the USB Rescue Media

Backup Protection

Backup Schedules

Backblaze

Other Features

Active Protection

Cloud Backup

Archive large or old files

Cloud Sync

Other Tools

Acronis True Image (Conclusion)

Pros

Cons

Disclaimer

Troubleshooting

DVD Rescue Media

Bonus: Windows System Restore

Create a System Restore Image

Restoring a restore point

Bonus: Clean Windows 10 Install

Bonus: How I prepare all files needed for a clean install

Finding Installed Software with SUMo

Links

Creating a Back Blaze Account

Choose a Backblaze Plan

Installing the Backblaze Software

Backblaze Preferences

Backblaze Portal

Initial Backup time

Restore Files from Backblaze

Other Links