Code, Security and Server Stuff

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

"If you're not still learning, you're already dying."
- Ryan Holiday - Ego is the Enemy

Follow me on Twitter: @FearbySoftware

View All Posts.

Cloud

I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.

by

I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance. Here is what I did to set up a complete Ubuntu 18.04 system (NGINX, PHP, MySQL, WordPress etc). This is not a paid review (just me documenting my steps over 2 days).

Background (CPanel hosts)

In 1999 I hosted my first domain (www.fearby.com) on a host in Seattle (for $10 USD a month), the host used CPanel and all was good.  After a decade I was using the domain more for online development and the website was now too slow (I think I was on dial-up or ADSL 1 at the time). I moved my domain to an Australian host (for $25 a month).

After 8 years the domain host was sold and performance remained mediocre. After another year the new host was sold again and performance was terrible.

Advertisement:



Page load times were near 30 seconds.

I started receiving Resource Limit Is Reached warnings (basically this was a plot by the new CPanel host to say “Pay us more and this message will go away”).

This is a screenshot of my old host giving a Usage Limit Exceeded error

The straw that broke the camel’s back was their demand of $150/year for a dodgy SSL certificate.

This is a screenshot of a crappy C rated SSL certificate (measured at ssl labs)

I needed to move to a self-managed server where I was in control.

Advertisement:



Buying a Domain Name

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Self Managed Server

I found a good web IDE ( http://www.c9.io/ ) that allowed me to connect to a cloud VM.  C9 allowed me to open many files and terminal windows and reconnect to them later. Don’t get excited, though, as AWS have purchased C9 and it’s not the same.

C9 IDE

C9 IDE

I spun up a Digital Ocean Server at the closest data centre in Singapore. Here was my setup guide creating a Digital Ocean VM, connecting to it with C9 and configuring it. I moved my email to G Suite and moved my WordPress to Digital Ocean (other guides here and here).

I was happy since I could now send emails via CLI/code, set up free SSL certs, add second domain email to G Suite and Secure G Suite. No more usage limit errors either.

Self-managing servers requires more work but it is more rewarding (flexible, faster and cheaper).  Page load times were now near 20 seconds (10-second improvement).

Latency Issue

Over 6 months, performance on Digital Ocean (in Singapore) from Australia started to drop (mentioned here).  I tried upgrading the memory but that did not help (latency was king).

Moved the website to Australia

I moved my domain to Vultr in Australia (guide here and here). All was good for a year until traffic growth started to increase.

Blog Growth

I tried upgrading the memory on Vultr and I setup PHP child workers, setup Cloudflare.

GT Metrix scores were about a “B” and Google Page Speed Scores were in the lower 40’s. Page loads were about 14 seconds (5-second improvement).

Tweaking WordPress

I set up an image compression plugin in WordPress then set up a cloud image compression and CDN Plugin from the same vendor.  Page Speed info here.

GT Metrix scores were now occasionally an “A” and Page Speed Scores were in the lower 20’s. Page loads were about 3-5 seconds (10-second improvement).

Mixed bag from Vultr (more optimisation and performance improvements were needed).

This screenshot is showing poor www.gtmetrix.com scores , pool google page speed index scores and upgrading from 1GB to 2GB memory on my server.

Google Chrome Developer Console Audit Results on Vultr hosted website were not very good (I stopped checking as nothing helped).

This is a screenshot showing poor site performance (screenshot taken in Google Dev tools audit feature)

The problem was the Vultr server (400km away in Sydney) was offline (my issue) and everything above (adding more memory, adding 2x CDN’s (EWWW and Cloudflare), adding PHP Child workers etc) did not seem to help???

Enter UpCloud…

Recently, a friend sent a link to a blog article about a host called “UpCloud” who promised “Faster than SSD” performance.  This can’t be right: “Faster than SSD”? I was intrigued. I wanted to check it out as I thought nothing was faster than SSD (well, maybe RAM).

I signed up for a trial and ran a disk IO test (read the review here) and I was shocked. It’s fast. Very fast.

Summary: UpCloud was twice as fast (Disk IO and CPU) as Vultr (+ an optional $4/m firewall and $3/m for 1x backup).

This is a screenshot showing Vultr.com servers getting half the read and write disk io performance compared to upcloud.com.

fyi: Labels above are K Bytes per second. iozone loops through all file sizes from 4 KB to 16,348 KB and measures through the reads per second. To be honest, the meaning of the numbers doesn’t interest me, I just want to compare apples to apples.

This is am image showing iozone results breakdown chart (kbytes per sec on vertical axis, file size in horizontal axis and transfer size on third access)

(image snip from http://www.iozone.org/ which explains the numbers)

I might have to copy my website on UpCloud and see how fast it is.

Where to Deploy and Pricing

UpCloud Pricing: https://www.upcloud.com/pricing/

UpCloud Pricing

UpCloud does not have a data centre in Australia yet so why choose UpCloud?

Most of my site’s visitors are based in the US and UpCloud have disk IO twice as fast as Vultr (win-win?).  I could deploy to Chicago?

This image sows most of my visitors are in the US

My site’s traffic is growing and I need to ensure the site is fast enough in the future.

This image shows that most of my sites visitors are hitting my site on week days.

Advertisement:



Creating an UpCloud VM

I used a friend’s referral code and signed up to create my first VM.

FYI: use my Referral code and get $25 free credit.  Sign up only takes 2 minutes.

https://www.upcloud.com/register/?promo=D84793

When you click the link above you will receive 25$ to try out serves for 3 days. You can exit his trail and deposit $10 into UpCloud.

Trial Limitations

The trial mode restrictions are as following:

* Cloud servers can only be accessed using SSH, RDP, HTTP or HTTPS protocols
* Cloud servers are not allowed to send outgoing e-mails or to create outbound SSH/RDP connections
* The internet connection is restricted to 100 Mbps (compared to 500 Mbps for non-trial accounts)
* After your 72 hours free trial, your services will be deleted unless you make a one-time deposit of $10

UpCloud Links

The UpCloud support page is located here: https://www.upcloud.com/support/

More UpCloud links to read:

Signing up to UpCloud

Navigate to https://upcloud.com/signup and add your username, password and email address and click signup.

New UpCloud Signup Page

Add your address and payment details and click proceed (you don’t need to pay anything ($1 may be charged and instantly refunded to verify the card)

Add address and payment details

That’s it, check yout email.

Signup Done

Look for the UpCloud email and click https://my.upcloud.com/

Check Email

Now login

Login to UpCloud

Now I can see a dashboard 🙂

UpCloud Dashboard

I was happy to see 24/7 support is available.

This image shows the www.upcloud.com live chat

I opted in for the new dashboard

UpCloud new new dashboard

Deploy My First UpCloud Server

This is how I deployed a server.

Note: If you are going to deploy a server consider using my referral code and get $25 credit for free.

Under the “deploy a server” widget I named the server and chose a location (I think I was supposed to use an FQDN name -e.g., “fearby.com”). The deployment worked though. I clicked continue, then more options were made available:

  1. Enter a short server description.
  2. Choose a location (Frankfurt, Helsinki, Amsterdam, Singapore, London and Chicago)
  3. Choose the number of CPU’s and amount of memory
  4. Specify disk number/names and type (MaxIOPS or HDD).
  5. Choose an Operating System
  6. Select a Timezone
  7. Define SSH Keys for access
  8. Allowed login methods
  9. Choose hardware adapter types
  10. Where the send the login password

Deploy Server

FYI: How to generate a new SSH Key (on OSX or Ubuntu)

Output

Did the key export? (yes)

> /temp# ls /temp/ -al
> drwxr-xr-x 2 root root 4096 Jun 9 15:33 .
> drwxr-xr-x 27 root root 4096 Jun 8 14:25 ..
> -rw——- 1 user user 1766 Jun 9 15:33 example_rsa
> -rw-r–r– 1 user user 396 Jun 9 15:33 example_rsa.pub

“example_rsa” is the private key and “example_rsa.pub “is the public key.

  • The public key needs to be added to the server to allow access.
  • The private key needs to be added to any local ssh program used for remote access.

Initialisation script (after deployment)

I was pleased to see an initialization script section that calls actions after the server is deployed. I configured the initialisation script to pull down a few GB of backups from my Vultr website in Sydney (files now removed).

This was my Initialisation script:

Confirm and Deploy

I clicked “Confirm and deploy” but I had an alert that said trial mode can only deploy servers up to 1024MB of memory.

This image shows I cant deploy servers with 2/GB in trial modeExiting UpCloud Trial Mode

I opened the dashboard and clicked My Account then Billing, I could see the $25 referral credit but I guess I can’t use that in Trial.

I exited trial mode by depositing $10 (USD).

View Billing Details

Make a manual 1-time deposit of $10 to exit trial mode.

Deposit $10 to exit the trial

FYI: Server prices are listed below (or view prices here).

UpCloud Pricing

Now I can go back and deploy the server with the same settings above (1x CPU, 2GB Memory, Ubuntu 18.04, MaxIOPS Storage etc)

Advertisement:



Deployment takes a few minutes and depending on how you specified a password may be emailed to you.

UpCloud Server Deployed

The server is now deployed; now I can connect to it with my SSH program (vSSH).  Simply add the server’s IP, username, password and the SSH private key (generated above) to your ssh program of choice.

fyi: The public key contents start with “ssh-rsa”.

This image shows me connecting to my sever via ssh

I noticed that the initialisation script downloaded my 2+GB of files already. Nice.

UpCloud Billing Breakdown

I can now see on the UpCloud billing page in my dashboard that credit is deducted daily (68c); at this rate, I have 49 days credit left?

Billing Breakdown

I can manually deposit funds or set up automatic payments at any time 🙂

UpCloud Backup Options

You do not need to setup backups but in case you want to roll back (if things stuff up), it is a good idea. Backups are an additional charge.

I have set up automatic daily backups with an auto deletion after 2 days

To view backup scheduled click on your deployed server then click backup

List of UpCloud Backups

Note: Backups are charged at $0.056 for every GB stored – so $5.60 for every 100GB per month (half that for 50GB etc)

You can take manual backups at any time (and only be charged for the hour)

UpCloud Firewall Options

I set up a firewall at UpCloud to only allow the minimum number of ports (UpCloud DNS, HTTP, HTTPS and My IP to port 22).  The firewall feature is charged at $0.0056 an hour ($4.03 a month)

I love the ability to set firewall rules on incoming, destination and outgoing ports.

To view your firewall click on your deployed server then click firewall

UpCloud firewall

Update: I modified my firewall to allow inbound ICMP (IPv4/IPv6) and UDP (IPv4/IPv6) packets.

(Note: Old firewall screenshot)

Firewall Rules Allow port 80, 443 and DNS

Because my internet provider has a dynamic IP, I set up a VPN with a static IP and whitelisted it for backdoor access.

Local Ubuntu ufw Firewall

I duplicated the rules in my local ufw (2nd level) firewall (and blocked mail)

UpCloud Download Speeds

I pulled down a 1.8GB Ubuntu 18.08 Desktop ISO 3 times from gigenet.com and the file downloaded in 32 seconds (57MB/sec). Nice.

Install Common Ubuntu Packages

I installed common Ubuntu packages.

Timezone

I checked the server’s time (I thought this was auto set before I deployed)?

I reset the time to Australia/Sydney.

Now the timezone is set 🙂

Shell History

I increased the shell history.

SSH Login

I created a ~/.ssh/authorized_keys file and added my SSH public key to allow password-less logins.

I added my pubic ssh key, then exited the ssh session and logged back in. I can now log in without a password.

Install NGINX

nginx/1.14.0 is now installed.

A quick GT Metrix test.

This image shows awesome static nginx performance ratings of of 99%

Install MySQL

Run these commands to install and secure MySQL.

Securing the MySQL server deployment.
> Would you like to setup VALIDATE PASSWORD plugin?: n
> New password: **********************************************
> Re-enter new password: **********************************************
> Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
> Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
> Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
> Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
> Success.

I disabled the validate password plugin because I hate it.

MySQL Ver 14.14 Distrib 5.7.22 is now installed.

Set MySQL root login password type

Set MySQL root user to authenticate via “mysql_native_password”. Run the “mysql” command.

Now let’s set the root password authentication method to “mysql_native_password”

Check authentication method.

Now we need to flush permissions.

Done.

Advertisement:



Install PHP

Install PHP 7.2

PHP 7.2.5, Zend Engine v3.2.0 with Zend OPcache v7.2.5-1 is now installed. Do update PHP frequently.

I made the following changes in /etc/php/7.2/fpm/php.ini

> cgi.fix_pathinfo=0
> max_input_vars = 1000
> memory_limit = 1024M
> max_file_uploads = 20M
> post_max_size = 20M

Install PHP Modules

sudo apt-get install php-pear php7.2-curl php7.2-dev php7.2-mbstring php7.2-zip php7.2-mysql php7.2-xml

Install PHP FPM

Configure PHP FPM config.

Edit /etc/php/7.2/fpm/php.ini

> cgi.fix_pathinfo=0
> max_input_vars = 1000
> memory_limit = 1024M
> max_file_uploads = 20M
> post_max_size = 20M

Reload php sudo service.

Install PHP Modules

Configuring NGINX

If you are not comfortable editing NGINX config files read here, here and here.

I made a new “www root” folder, set permissions and created a default html file.

I edited the “root” key in “/etc/nginx/sites-enabled/default” file and set the root a new location (e.g., “/www-root”)

I added these performance tweaks to /etc/nginx/nginx.conf

> worker_cpu_affinity auto;
> worker_rlimit_nofile 100000

I add the following lines to “http {” section in /etc/nginx/nginx.conf

Check NGINX Status

Install Open SSL that supports TLS 1.3

This is a work in progress. The steps work just fine for me on Ubuntu 16.04. but not Ubuntu 18.04.?

Installing Adminer MySQL GUI

I will use the PHP based Adminer MySQL GUI to export and import my blog from one server to another. All I needed to do is install it on both servers (simple 1 file download)

Use Adminer to Export My Blog (on Vultr)

On the original server open Adminer (http) and..

  1. Login with the MySQL root account
  2. Open your database
  3. Choose “Save” as the output
  4. Click on Export

This image shows the export of the wordpress adminer page

Save the “.sql” file.

I used Adminer on the UpCloud server to Import My Blog

FYI: Depending on the size of your database backup you may need to temporarily increase your upload and post sizes limits in PHP and NGINX before you can import your database.

Edit /etc/php/7.2/fpm/php.ini
> max_file_uploads = 100M
> post_max_size =100M

And Edit: /etc/nginx/nginx.conf
> client_max_body_size 100M;

Don’t forget to reload NGINX config and restart NGINX and PHP. Take note of the maximum allowed file size in the screenshot below. I temporarily increased my upload limits to 100MB in order to restore my 87MB blog.

Now I could open Adminer on my UpCloud server.

  1. Create a new database
  2. Click on the database and click Import
  3. Choose the SQL file
  4. Click Execute to import it

Import MuSQL backup with Adminer

Don’t forget to create a user and assign permissions (as required – check your wp-config.php file).

Import MySQL Database

Tip: Don’t forget to lower the maximum upload file size and max post size after you import your database,

Cloudflare DNS

I use Cloudflare to manage DNS, so I need to tell it about my new server.

You can get your server’s IP details from the UpCloud dashboard.

Find IP

At Cloudflare update your DNS details to point to the server’s new IPv4 (“A Record”) and IPv6 (“AAAA Record”).

Cloudflare DNS

Advertisement:



Domain Error

I waited an hour and my website was suddenly unavailable.  At first, I thought this was Cloudflare forcing the redirection of my domain to HTTP (that was not yet set up).

DNS Not Replicated Yet

I chatted with UpCloud chat on their webpage and they kindly assisted me to diagnose all the common issues like DNS values, DNS replication, Cloudflare settings and the error was pinpointed to my NGINX installation.  All NGINX config settings were ok from what we could see?  I uninstalled NGINX and reinstalled it (and that fixed it). Thanks UpCloud Support 🙂

Reinstalled NGINX

I reinstalled NGINX and reconfigured /etc/nginx/nginx.conf (I downloaded my SSL cert from my old server just in case).

Here is my /etc/nginx/nginx.conf file.

Here is my /etc/nginx/sites-available/default file (fyi, I have not fully re-setup TLS 1.3 yet so I commented out the settings)

SSL Labs SSL Certificate Check

All good thanks to the config above.

SSL Labs

Install WP-CLI

I don’t like setting up FTP to auto-update WordPress plugins. I use the WP-CLI tool to manage WordPress installations by the command line. Read my blog here on using WP-CLI.

Download WP-CLI

Move WP-CLI to the bin folder as “wp”

Test wp

Update WordPress Plugins

Now I can run “wp plugin update” to update all WordPress plugins

Update WordPress Core

WordPress core file can be updated with “wp core update”

Troubleshooting: Use the flag “–allow-root “if wp needs higher access (unsafe action though).

Install PHP Child Workers

I edited the following file to setup PHP child workers /etc/php/7.2/fpm/pool.d/www.conf

Changes

> pm = dynamic
> pm.max_children = 40
> pm.start_servers = 15
> pm.min_spare_servers = 5
> pm.max_spare_servers = 15
> pm.process_idle_timeout = 30s;
> pm.max_requests = 500;
> php_admin_value[error_log] = /var/log/www-fpm-php.www.log
> php_admin_value[memory_limit] = 512M

Advertisement:



Restart PHP

Test NGINX config, reload NGINX config and restart NGINX

Output (14 workers are ready)

Memory Tweak (set at your own risk)

vm.swappiness = 1

Setting swappiness to a value of 1 all but disables the swap file and tells the Operating System to aggressively use ram, a value of 10 is safer. Only set this if you have enough memory available (and free).

Possible swappiness settings:

> vm.swappiness = 0 Swap is disabled. In earlier versions, this meant that the kernel would swap only to avoid an out of memory condition when free memory will be below vm.min_free_kbytes limit, but in later versions, this is achieved by setting to 1.[2]> vm.swappiness = 1 Kernel version 3.5 and over, as well as Red Hat kernel version 2.6.32-303 and over: Minimum amount of swapping without disabling it entirely.
> vm.swappiness = 10 This value is sometimes recommended to improve performance when sufficient memory exists in a system.[3]
> vm.swappiness = 60 The default value.
> vm.swappiness = 100 The kernel will swap aggressively.

The “htop” tool is a handy memory monitoring tool to “top”

Also you can use good old “watch” command to show near-live memory usage (auto-refreshes every 2 seconds)

Script to auto-clear the memory/cache

As a habit, I am setting up a cronjob to check when free memory falls below 100MB, then cache is automatically cleared (freeing memory).

Script Contents: clearcache.sh

I set the cronjob to run every 15 mins, I added this to my cronjob.

Sample log output

I will check the log (/scripts/clearcache.log) in a few days and view the memory trends.

After 1/2 a day Ubuntu 18.04 is handling memory just fine, no externally triggered cache clears have happened 🙂

Free memory over time

I used https://crontab.guru/every-hour to set the right schedule in crontab.

I rebooted the VM.

Update: I now use Nixstats monitoring

Swap File

FYI: Here is a handy guide on viewing swap file usage here. I’m not using swap files so it is only an aside.

After the system rebooted I checked if the swappiness setting was active.

Yes, swappiness is set.

File System Tweaks – Write Back Cache (set at your own risk)

First, check your disk name and file system

Take note of your disk name (e.g vda1)

I used TuneFS to enable writing data to the disk before writing to the journal. tunefs is a great tool for setting file system parameters.

Warning (snip from here): “I set the mode to journal_data_writeback. This basically means that data may be written to the disk before the journal. The data consistency guarantees are the same as the ext3 file system. The downside is that if your system crashes before the journal gets written then you may loose new data — the old data may magically reappear.”

Warning this can corrupt your data. More information here.

I ran this command.

I edited my fstab to append the “writeback,noatime,nodiratime” flags for my volume after a reboot.

Edit FS Tab:

I added “writeback,noatime,nodiratime” flags to my disk options.

Updating Ubuntu Packages

Show updatable packages.

Update Packages.

Unattended Security Updates

Read more on Ubuntu 18.04 Unattended upgrades here, here and here.

Install Unattended Upgrades

Enable Unattended Upgrades.

Now I configure what packages not to auto update.

Edit /etc/apt/apt.conf.d/50unattended-upgrades

Find “Unattended-Upgrade::Package-Blacklist” and add packages that you don’t want automatically updated, you may want to manually update these (and monitor updates).

I prefer not to auto-update critical system apps (I will do this myself).

FYI: You can find installed packages by running this command:

Enable automatic updates by editing /etc/apt/apt.conf.d/20auto-upgrades

Edit the number at the end (the number is how many days to wait before updating) of each line.

> APT::Periodic::Update-Package-Lists “1”;
> APT::Periodic::Download-Upgradeable-Packages “1”;
> APT::Periodic::AutocleanInterval “7”;
> APT::Periodic::Unattended-Upgrade “1”;

Set to “0” to disable automatic updates.

The results of unattended-upgrades will be logged to /var/log/unattended-upgrades

Update packages now.

Almost done.

I Rebooted

Advertisement:



GT Metrix Score

I almost fell off my chair. It’s an amazing feeling hitting refresh in GT Metrix and getting sub-2-second score consistently (and that is with 17 assets loading and 361KB of HTML content)

0.9sec load times

WebPageTest.org Test Score

Nice. I am not sure why the effective use of CDN has an X rating as I have the EWWW CDN and Cloudflare. First Byte time is now a respectable “B”, This was always bad.

Update: I found out the longer you set cache delays in Cloudflare the higher the score.

Web Page Test

GT Metrix has a nice historical breakdown of load times (night and day).

Upcloud Site Speed in GTMetrix

Google Page Speed Insight Desktop Score

I benchmarked with https://developers.google.com/speed/pagespeed/insights/

This will help with future SEO rankings. It is well known that Google is pushing fast servers.

100% Desktop page speed score

Google Chrome 70 Dev Console Audit (Desktop)

100% Chrome Audit Score

This is amazing, I never expected to get this high score.  I know Google like (and are pushing) sub-1-second scores.

My site is loading so well it is time I restored some old features that were too slow on other servers

  • I disabled Lazy loading of images (this was not working on some Android devices)
  • I re-added the News Widget and news images.

GTMetrix and WebpageTest sores are still good (even after adding bloat)

Benchmarks are still good

My WordPress site is not really that small either

Large website

FYI: WordPress Plugins I use.

These are the plugins I use.

How I use these plugins to speed up my site.

  • I use EWWW Image Optimizer plugin to auto-compress my images and to provide a CDN for media asset deliver (pre-Cloudflare). Learn more about ExactDN and EWWW.io here.
  • I use Autoptimize plugin to optimise HTML/CSS/JS and ensure select assets are on my EWWW CDN. This plugin also removes WordPress Emojis, removed the use of Google Fonts, allows you to define pre-configured domains, Async Javascript-files etc.
  • I use BJ Lazy Load to prevent all images in a post from loading on load (and only as the user scrolls down the page).
  • GTmetrix for WordPress and Cloudflare plugins are for information only?
  • I use WP-Optimize to ensure my database is healthy and to disable comments/trackbacks and pingbacks.

Let’s Test UpCloud’s Disk IO in Chicago

Looks good to me, Read IO is a little bit lower than UpCloud’s Singapore data centre but still, it’s faster than Vultr.  I can’t wait for more data centres to become available around the world.

Why is UpCloud Disk IO so good?

I asked UpCloud on Twitter why the Disk IO was so good.

  • “MaxIOPS is UpCloud’s proprietary block-storage technology. MaxIOPS is physically redundant storage technology where all customer’s data is located in two separate physical devices at all times. UpCloud uses InfiniBand (!) network to connect storage backends to compute nodes, where customers’ cloud servers are running. All disks are enterprise-grade SSD’s. And using separate storage backends, it allows us to live migrate our customers’ cloud servers freely inside our infrastructure between compute nodes – whether it be due to hardware malfunction (compute node) or backend software updates (example CPU vulnerability and immediate patching).“

My Answers to Questions to support

Q1) What’s the difference between backups and snapshots (a Twitter user said Snapshots were a thing)

A1) Backups and snapshots are the same things with our infrastructure.

Q2) What are charges for backup of a 50GB drive?

A2) We charge $0.06 / GB of the disk being captured. But capture the whole disk, not just what was used. So for a 50GB drive, we charge $0.06 * 50 = $3/month. Even if 1GB were only used.

  • Support confirmed that each backup is charged (so 5 times manual backups are charged 5 times). Setting up a daily auto backup schedule for 2 weeks would create 14 billable backup charges.
  • I guess a 25GB server will be $1.50 a month

Q3) What are data charges if I go over my 2TB quota?

A3) Outgoing data charges are $0.056/GB after the pre-configured allowance.

Q4) What happens if my balance hits $0?

A4) You will get notification of low account balance 2 weeks in advance based on your current daily spend. When your balance reaches zero, your servers will be shut down. But they will still be charged for. You can automatically top-up if you want to assign a payment type from your Control Panel. You deposit into your balance when you want. We use a prepay model of payment, so you need to top up before using, not billing you after usage. We give you lots of chances to top-up.

Support Tips

  • One thing to note, when deleting servers (CPU, RAM) instances, you get the option to delete the storages separately via a pop-up window. Choose to delete permanently to delete the disk, to save credit. Any disk storage lying around even unattached to servers will be billed.
  • Charges are in USD.

I think it’s time to delete my domain from Vultr in Sydney.

Deleted my Vultr domain

I deleted my Vultr domain.

Delete Vultr Server

Done.

More Reading on UpCloud

https://www.upcloud.com/documentation/faq/

UpCloud Server Status

http://status.upcloud.com

Check out my new guide on Nixstats for awesome monitoring

What I would like

  1. Ability to name individual manual backups (tag with why I backed up).
  2. Ability to push user defined data from my VM to the dashboard
  3. Cheaper scheduled backups
  4. Sydney data centres (one day)

Advertisement:



Update: Post UpCloud Launch Tweaks (Awesome)

I had a look at https://www.webpagetest.org/ results to see where else I can optimise webpage delivery.

Optimisation Options

Disable dasjhicons.min.css (for unauthenticated WordPress users).

Find functions.php in the www root

Edit functions.php

Add the following

// Remove dashicons in frontend for unauthenticated users
add_action( 'wp_enqueue_scripts', 'bs_dequeue_dashicons' );
function bs_dequeue_dashicons() {
    if ( ! is_user_logged_in() ) {
        wp_deregister_style( 'dashicons' );
    }
}

HTTP2 Push

I added http2 to my listening servers

I tested a http2 push page by defining this in /etc/nginx/sites-available/default 

Once I tested that push (demo here) was working I then defined two files to push that were being sent from my server

I used the WordPress Plugin Autoptimize to remove Google font usage (this removed a number of files being loaded when my page loads).

I used the WordPress Plugin WP-Optimize plugin into to remove comments and disable pingbacks and trackbacks.

WordPress wp-config.php tweaks

Add 2FA Authentication to server logins.

I recently checked out YubiCo YubiKeys and I have secured my Linux servers with 2FA prompts at login. Read the guide here. I secured my WordPress too.

Tweaks Todo

  • Compress placeholder BJ Lazy Load Image (plugin is broken)
  • Solve 2x Google Analytics tracker redirects (done, switched to Matomo)

Conclusion

I love UpCloud’s fast servers, give them a go (use my link and get $25 free credit).

I love Cloudflare for providing a fast CDN.

I love ewww.io’s automatic Image Compression and Resizing plugin that automatically handles image optimisations and pre Cloudflare/first hit CDN caching.

Read my post about server monitoring with Nixstats here.

Let the results speak for themselves (sub <1 second load times).

Results

I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v2.1 Newer GTMetrix scores

v2.0 New UpCloud UI Update and links to new guides.

v1.9 Spelling and grammar

v1.8 Trial mode gotcha (deposit money ASAP)

v1.7 Added RSA Private key info

v1.7 – Added new firewall rules info.

v1.6 – Added more bloat to the site, still good.

v1.5 Improving Accessibility

v1.4 Added Firewall Price

v1.3 Added wp-config and plugin usage descriptions.

v1.2 Added GTMetrix historical chart.

v1.1 Fixed free typos and added final conclusion images.

v1.0 Added final results

v0.9 added more tweaks (http2 push, removing unwanted files etc)

v0.81 Draft  – Added memory usage chart and added MaxIOPS info from UpCloud.

v0.8 Draft post.

I set the cronjob to run every 15 mins, I added this to my cronjob.

Sample log output
I will check the log (/scripts/clearcache.log) in a few days and view the memory trends.
After 1/2 a day Ubuntu 18.04 is handling memory just fine, no externally triggered cache clears have happened 🙂
Free memory over time
I used https://crontab.guru/every-hour to set the right schedule in crontab.
I rebooted the VM.
Update: I now use Nixstats monitoring
Swap File
FYI: Here is a handy guide on viewing swap file usage here. I’m not using swap files so it is only an aside.
After the system rebooted I checked if the swappiness setting was active.
Yes, swappiness is set.
File System Tweaks – Write Back Cache (set at your own risk)
First, check your disk name and file system
Take note of your disk name (e.g vda1)
I used TuneFS to enable writing data to the disk before writing to the journal. tunefs is a great tool for setting file system parameters.
Warning (snip from here): “I set the mode to journal_data_writeback. This basically means that data may be written to the disk before the journal. The data consistency guarantees are the same as the ext3 file system. The downside is that if your system crashes before the journal gets written then you may loose new data — the old data may magically reappear.”
Warning this can corrupt your data. More information here.
I ran this command.
I edited my fstab to append the “writeback,noatime,nodiratime” flags for my volume after a reboot.
Edit FS Tab:
I added “writeback,noatime,nodiratime” flags to my disk options.
Updating Ubuntu Packages
Show updatable packages.
Update Packages.
Unattended Security Updates
Read more on Ubuntu 18.04 Unattended upgrades here, here and here.
Install Unattended Upgrades
Enable Unattended Upgrades.
Now I configure what packages not to auto update.
Edit /etc/apt/apt.conf.d/50unattended-upgrades
Find “Unattended-Upgrade::Package-Blacklist” and add packages that you don’t want automatically updated, you may want to manually update these (and monitor updates).
I prefer not to auto-update critical system apps (I will do this myself).
FYI: You can find installed packages by running this command:
Enable automatic updates by editing /etc/apt/apt.conf.d/20auto-upgrades
Edit the number at the end (the number is how many days to wait before updating) of each line.
> APT::Periodic::Update-Package-Lists “1”;
 > APT::Periodic::Download-Upgradeable-Packages “1”;
 > APT::Periodic::AutocleanInterval “7”;
 > APT::Periodic::Unattended-Upgrade “1”;
Set to “0” to disable automatic updates.
The results of unattended-upgrades will be logged to /var/log/unattended-upgrades
Update packages now.
Almost done.
I Rebooted
Advertisement:

 
 
GT Metrix Score
I almost fell off my chair. It’s an amazing feeling hitting refresh in GT Metrix and getting sub-2-second score consistently (and that is with 17 assets loading and 361KB of HTML content)
0.9sec load times
WebPageTest.org Test Score
Nice. I am not sure why the effective use of CDN has an X rating as I have the EWWW CDN and Cloudflare. First Byte time is now a respectable “B”, This was always bad.
Update: I found out the longer you set cache delays in Cloudflare the higher the score.
Web Page Test
GT Metrix has a nice historical breakdown of load times (night and day).
Upcloud Site Speed in GTMetrix
Google Page Speed Insight Desktop Score
I benchmarked with https://developers.google.com/speed/pagespeed/insights/
This will help with future SEO rankings. It is well known that Google is pushing fast servers.
100% Desktop page speed score
Google Chrome 70 Dev Console Audit (Desktop)
100% Chrome Audit Score
This is amazing, I never expected to get this high score.  I know Google like (and are pushing) sub-1-second scores.
My site is loading so well it is time I restored some old features that were too slow on other servers
  • I disabled Lazy loading of images (this was not working on some Android devices)
  • I re-added the News Widget and news images.
GTMetrix and WebpageTest sores are still good (even after adding bloat)
Benchmarks are still good
My WordPress site is not really that small either
Large website
FYI: WordPress Plugins I use.
These are the plugins I use.
How I use these plugins to speed up my site.
  • I use EWWW Image Optimizer plugin to auto-compress my images and to provide a CDN for media asset deliver (pre-Cloudflare). Learn more about ExactDN and EWWW.io here.
  • I use Autoptimize plugin to optimise HTML/CSS/JS and ensure select assets are on my EWWW CDN. This plugin also removes WordPress Emojis, removed the use of Google Fonts, allows you to define pre-configured domains, Async Javascript-files etc.
  • I use BJ Lazy Load to prevent all images in a post from loading on load (and only as the user scrolls down the page).
  • GTmetrix for WordPress and Cloudflare plugins are for information only?
  • I use WP-Optimize to ensure my database is healthy and to disable comments/trackbacks and pingbacks.
Let’s Test UpCloud’s Disk IO in Chicago
Looks good to me, Read IO is a little bit lower than UpCloud’s Singapore data centre but still, it’s faster than Vultr.  I can’t wait for more data centres to become available around the world.
Why is UpCloud Disk IO so good?
I asked UpCloud on Twitter why the Disk IO was so good.
  • “MaxIOPS is UpCloud’s proprietary block-storage technology. MaxIOPS is physically redundant storage technology where all customer’s data is located in two separate physical devices at all times. UpCloud uses InfiniBand (!) network to connect storage backends to compute nodes, where customers’ cloud servers are running. All disks are enterprise-grade SSD’s. And using separate storage backends, it allows us to live migrate our customers’ cloud servers freely inside our infrastructure between compute nodes – whether it be due to hardware malfunction (compute node) or backend software updates (example CPU vulnerability and immediate patching).“
My Answers to Questions to support
Q1) What’s the difference between backups and snapshots (a Twitter user said Snapshots were a thing)
A1) Backups and snapshots are the same things with our infrastructure.
Q2) What are charges for backup of a 50GB drive?
A2) We charge $0.06 / GB of the disk being captured. But capture the whole disk, not just what was used. So for a 50GB drive, we charge $0.06 * 50 = $3/month. Even if 1GB were only used.
  • Support confirmed that each backup is charged (so 5 times manual backups are charged 5 times). Setting up a daily auto backup schedule for 2 weeks would create 14 billable backup charges.
  • I guess a 25GB server will be $1.50 a month
Q3) What are data charges if I go over my 2TB quota?
A3) Outgoing data charges are $0.056/GB after the pre-configured allowance.
Q4) What happens if my balance hits $0?
A4) You will get notification of low account balance 2 weeks in advance based on your current daily spend. When your balance reaches zero, your servers will be shut down. But they will still be charged for. You can automatically top-up if you want to assign a payment type from your Control Panel. You deposit into your balance when you want. We use a prepay model of payment, so you need to top up before using, not billing you after usage. We give you lots of chances to top-up.
Support Tips
  • One thing to note, when deleting servers (CPU, RAM) instances, you get the option to delete the storages separately via a pop-up window. Choose to delete permanently to delete the disk, to save credit. Any disk storage lying around even unattached to servers will be billed.
  • Charges are in USD.
I think it’s time to delete my domain from Vultr in Sydney.
Deleted my Vultr domain
I deleted my Vultr domain.
Delete Vultr Server
Done.
More Reading on UpCloud
https://www.upcloud.com/documentation/faq/
UpCloud Server Status
http://status.upcloud.com
Check out my new guide on Nixstats for awesome monitoring
What I would like
  1. Ability to name individual manual backups (tag with why I backed up).
  2. Ability to push user defined data from my VM to the dashboard
  3. Cheaper scheduled backups
  4. Sydney data centres (one day)
Advertisement:

 
 
Update: Post UpCloud Launch Tweaks (Awesome)
I had a look at https://www.webpagetest.org/ results to see where else I can optimise webpage delivery.
Optimisation Options
Disable dasjhicons.min.css (for unauthenticated WordPress users).
Find functions.php in the www root
Edit functions.php
Add the following
HTTP2 Push
I added http2 to my listening servers
I tested a http2 push page by defining this in /etc/nginx/sites-available/default 
Once I tested that push (demo here) was working I then defined two files to push that were being sent from my server
I used the WordPress Plugin Autoptimize to remove Google font usage (this removed a number of files being loaded when my page loads).
I used the WordPress Plugin WP-Optimize plugin into to remove comments and disable pingbacks and trackbacks.
WordPress wp-config.php tweaks
Add 2FA Authentication to server logins.
I recently checked out YubiCo YubiKeys and I have secured my Linux servers with 2FA prompts at login. Read the guide here. I secured my WordPress too.
Tweaks Todo
  • Compress placeholder BJ Lazy Load Image (plugin is broken)
  • Solve 2x Google Analytics tracker redirects (done, switched to Matomo)
Conclusion
I love UpCloud’s fast servers, give them a go (use my link and get $25 free credit).
I love Cloudflare for providing a fast CDN.
I love ewww.io’s automatic Image Compression and Resizing plugin that automatically handles image optimisations and pre Cloudflare/first hit CDN caching.
Read my post about server monitoring with Nixstats here.
Let the results speak for themselves (sub <1 second load times).
Results
I hope this guide helps someone.
Please consider using my referral code and get $25 credit for free.
https://www.upcloud.com/register/?promo=D84793
Ask a question or recommend an article
     
Your Name (required)
 
Your Email (required)
 
Your Question
 
Revision History
v2.1 Newer GTMetrix scores
v2.0 New UpCloud UI Update and links to new guides.
v1.9 Spelling and grammar
v1.8 Trial mode gotcha (deposit money ASAP)
v1.7 Added RSA Private key info
v1.7 – Added new firewall rules info.
v1.6 – Added more bloat to the site, still good.
v1.5 Improving Accessibility
v1.4 Added Firewall Price
v1.3 Added wp-config and plugin usage descriptions.
v1.2 Added GTMetrix historical chart.
v1.1 Fixed free typos and added final conclusion images.
v1.0 Added final results
v0.9 added more tweaks (http2 push, removing unwanted files etc)
v0.81 Draft  – Added memory usage chart and added MaxIOPS info from UpCloud.
v0.8 Draft post.
How to back up an iPhone (including photos and videos) multiple ways
 by 
This guide is for and anyone who needs to back up their iPhone (complete device along with separate backups of photos and videos).
This is not a paid promo, I don’t get a kickback for mentioning the awesome app below.
Check our my other related posts
At the time of writing iTunes v12.9.5.7 was the latest version of iTunes.
iTunes is Apple official software for getting files too and from an iPhone, iPod or iPad. If rumours are right Apple will kill off iTunes software at the end of 2019. If Apple kills off the iTunes software I will update this guide in the future.
Disclaimer: I provide no warranty or support for this app, this is what happened to work for me.
Why write a guide on Backing up a mobile phone?
  • It’s not that simple an I get asked how to do this almost every week by someone.
  • No one thinks to backup up their photos until they fear their phone is lost or stolen.
  • I am about to publish a few posts on backing up (development machines, servers etc) with automated software and with free open source software so this post will be handy to link in what data you can back up in that post.
  • The last time I wrote an article on backups was in March 2016 and its not that great these days.
Golden rules of backing up.
  1. Backup to Three different locations.
  2. Two of the media need to be different.
  3. One of the locations needs to be offsite
  4. Test your backups (a backup is only good if you can restore).  (from @Daniel15)
  5. If the backup is encrypted make sure you have the decryption keys.  (from @Daniel15)
Backing up the official way (with iTunes)
Apple prefers you just pay them to extend your free 5GB iCloud storage and not worry about backups and not to worry yourself with manual steps.
Advertisement:
   


My wife’s iPhone is always popping up messages saying that her iCloud storage is nearly full and she should upgrade the free space limit.
Screenshot of iCloud setting on iPhone saying your iCloud is nearly full but you have used 5GB or 5GB?
I know she has 100GB+ images and movies on her iPhone but iCloud has only backed up 4GB of photos and videos. Do not trust any backup statuses unless you can verify all files from a restore.
For the record I use an Android phone (Android backup guide coming soon).
iCloud is a good idea for automatic backup but I prefer to see my photos and backup them up myself. Also, Apple iCloud is not free from troubles. If you want Apple to handle backups then I do suggest you upgrade your iCloud storage from the free 5GB to a more sensible 200GB or more.
Screenshot of https://support.apple.com/en-au/HT201318
Here in Australia Apple charge the following for extra storage.
  • 50GB = $1.49 a month
  • 200GB = $4.49 a month
  • 2TB (2,0000GB) = $14.99 a month
A rough estimate: 50GB is enough to store about 12,000 files (9,000 photos + 3,000 videos) *
  • * = based on stats from the backup below (depending on the size an length of your videos).
Screenshot of apple iCloud pricing 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month 
You can upgrade your paid Apple iCloud backup limit from your iCloud Storage Settings from your iPhone’s settings screen.
Screenshot of iPhone iCloud upgrade button.
IMHO: Apple does make it clear what you are getting before you purchase (but they do not push it enough and people assume their data is safe).
iPhone picture of 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month 
TIP: Please review your existing iPhone data usage and iCloud usage before it’s too late
I don’t want ot pay Apple for more iCloud storage
I like you, I am tight too but I do pay Google $2.49 a month for 100GB backup storage on my Android phone. Google Drive (Google storage is a bit cheaper than Apple).
Screenshot of Google Drive https://www.google.com/drive/
It is nice knowing your phone is auto backed up.
Install iTunes on Windows (10)
On your Windows 10 computer click the start button. If you don’t have a Microsoft Account go here and create one. Click the start button and type “itunes” (you don’t need to type into a box, just start typing when you click the start button)’.
Then click “iTunes, install app”
Screenshot of a Windows start menu and Get iTunes icon
Advertisement:
   


If you are logged into your Windows 10 store with your Microsoft ID click “Get” (if not you will need to login to the store).
Screenshot of me logged into the Windows Microsoft Store.
After you click ‘Get’, iTunes will start downloading.
Screenshot of the Windows Store downloading iTunes
When iTunes is downloaded Windows will install it. When it’s installed click ‘Launch’
iTunes is installed, Click launch screenshot
You will need to agree to Apple’s terms of service. Click ‘Agree’
Apple terms of service screenshot.
When iTunes opens click “Agree”
Screenshot of Apple iTunes asking if we can agree to share analytics data
Now login to iTunes with your Apple ID (if you don’t have one create one here)’.
Screenshot of iTunes open with an arrow pointing to the Account menu
Click the ‘Account’ then ‘Sign In’ menu.
Screenshot of the account then sign in menu
Login to iTunes with your Apple ID
Screenshot of the apple sign in box.
Optional: If you have Two Factor Authorisation (2FA) of Apple ID’s turned on  (you should) you will need to enter a 6 digit code.
Screenshot of an apple 6 digit 2FA code
Apple Two Factor Authorisation (2FA) will send one of your other devices a login code that you will need to use to login (confirm you own the account)
Screenshot of apples 2FA web page at https://support.apple.com/en-au/HT204915
TIP: Check out https://twofactorauth.org/ to see what other sites use Two Factor Authorisation (2FA). I use hardware Yubico YubiKeys to protect logins to WordPress, Linux and websites.
Now back to the article, iTunes should be ready to allow us to backup our iPhone.
Screenshot of me logged into iTunes
Before I continue I will click ‘Edit‘ then ‘Preferences‘ menu to view where iTunes will download media too (different ot backup data), not important but I just want ot see if it is not pointed ot my smaller C drive before I backup my phone
I changed the location to a larger S:\Drive’
Screenshot of Apple setting screen (tab Advanced)
I was greeted with a message asking me to confirm that I wanted to sync the iPhone that I just plugged in called ‘EllieRose’, I clicked ‘Continue’
Confirm access to the iPhone screenshot
On the iPhone I also clicked ‘Trust’ to allow iTunes to talk to it.
iPhone screenshot to allow iTunes to talk to it.
I was then prompted to download a software update to the iPhone, It appears this phone is not running the latest software
Screenshot of a question to update to iOS v 12.3.1
I was prompted to sync the iPhone to the computer
screenshot of a prompt to sync purchases from the iPhone to the computer
A backup of the phone was underway.
Screenshot of iTunes backup progress with a progress bar
If the screen above does not appear click the icon below to view the backup and restore iPhone menu.
Screenshot showing the iPhone Device button in iTunes
The iPhone was backing up to this folder on my computer:
C:\Users\Simon Fearby\Apple\MobileSync\Backup
I was not prompted for a location to back up so and I will move this backup folder after the backup completes (so my C: drive does not fill up).
My drives.
  • C:\ =  500GB drive (faster SSD)
  • S:\ = 2,000GB Drive (slower Magnetic)
Screenshot showing  iTunes backup and  the C:\Users\Simon Fearby\Apple\MobileSync\Backup folder
The iTunes iPhone backup is now complete (it took about 2 hours).
Screenshot of the complete backup
Now that the iPhone was backed up iTunes the Operating System update started to run.
Screenshot of iTunes verifying the iPhone update
Update verified.
Screenshot of iTunes updating the iPhone OS.
During the update the iPhone was unavailable.
Screenshot of iPhone installing firmware 
Done, the iPhone had updated it’s operating system in about 30 minutes.
Screenshot of iTunes showing the newer OS version.
Now lets see how much space I have available on my C Drive.
Screenshot of disk usage on my c and S drive.
It looks like the iPhone backup iTunes made was 60GB in size.
Advertisement:
   


Screenshot of Windows reporting the backup was 60GB in size.
Using Windows Explorer I moved the backup from..
C:\Users\Simon Fearby\Apple\MobileSync\Backup
to..
S:\Backup\AlisonsiPhone\iTunes\31May2019
(right click drag an drop action in explorer)
Screenshot of right click drag and drop move folder.
It took me about 20 minutes to move the 60GB folder.
Screenshot of thew Windows copy dialog progress.
TIP: You can (should) also copy this backup folder to..
Now this backup is available for me to restore in the future if I need it.
In the case of a restore, I  just need to move the backup into the iTunes expected location.
Copy from..
S:\Backup\AlisonsiPhone\iTunes\31May2019
..to..
C:\Users\Simon Fearby\Apple\MobileSync\Backup
The Catch
The catch with back-ups made with iTunes is they are useless if you wanted to restore individual files (say to find a photograph or video). iTunes backups are usually tens of thousands of files with random folders and filenames
Screenshot of iTunes nasty backup, i obfuscated all files to random guid filenames
If you want to JUST backup photos and videos read on.
How to backup just photos and videos from your iPhone.
This part of the guide needs a paid version of the iOS App Photo Transfer App. In Australia, the app is free but to turn it into a full version needed to restore everything  (via in-app purchase) it cost $10.99.
This app is well worth $10.99 AUD (it may be cheaper in other countries) to have it push photos an videos from your iPhone to a free companion Windows program (a Mac versions exist too).
Screenshot of http://phototransferapp.com/
Buy the iOS version and install it on your iPhone.
Then download the free Windows version: http://phototransferapp.com/win/
Extract the files from the zip file.
I extracted PhotoTransferapp.exe from PhotoTransferapp.zip 
When I run PhotoTransferApp.exe I get an error saying I need to install Adobe AIR run times.
Screenshot of me installing Adobe Air run times from https://get.adobe.com/air/
Go to https://get.adobe.com/air/ and download and install the Air run time.
Screenshot of https://get.adobe.com/air/
Install the Adobe Air run time.
Install Adobe Air From https://get.adobe.com/air/
I installed Adobe AIR and again reopened the PhotoTansferApp.exe and was prompted to allow access to the Windows Firewall.
I was on a home network so I enabled firewall access.
Screenshot of Photo Transfer App asking fore firewall access.
The Photo Transfer App prompted me to ‘Discover Devices‘ and to make sure the Transfer app is running on the iPhone.
The Photo Transfer App prompted me to 'Discover Devices' and to make sure the Transfer app is running on the iPhone.
Before I clicked ‘Discover Devices” on the Photo Transfer App on Windows I opened the Transfer app (mentioned above) on the iPhone.
Photo of the Transfer app on the iPhone
On the iPhone I clicked ‘Send‘
Click Send
The transfer app will ask for access to your photos, you will need to press OK.
Advertisement:
   


Screenshot of the transfer app asking for permissions to photos
On the iPhone again I clicked  ”Windows‘
On the iPhone again I clicked  ''Windows'
The iOS Transfer app now said I should run the ‘Photo Transfer App‘ on Windows.
The iOS Transfer app now said I should run the 'Photo Transfer App' on Windows.
TIP: You can transfer over WiFi (if your iPhone and Windows device is on the same WiFi Network) or you can transfer over a USB cable.
Screenshot: transfer via WiFi or USB?
Before I started the ‘Detect Device‘ or ‘transfer‘, I set the Backup path location to my S: drive by clicking settings in the bottom right and choosing a folder (as my C Drive is a bit small).
Screenshot of Photo Transfer App choosing a folder to backup to.
I noticed the port of 57777, I temporarily disabled the whole Windows Firewall just in case it prevents the photo backup.
In hindsight, it was not a good idea to disable the whole firewall, but because I was at home on a safe network I felt safe to do so. Next time I will not disable the firewall and see if this still works.
If you are on an internet cafe, school or university network do not disable your firewall.
I clicked ”Start” then typed ‘firewall‘ and clicked ‘Windows Defender Firewall‘
Screenshot Windows Defender firewall icon
I turned off my firewall.
Screenshot Firewall disabled.
I clicked ‘Detect Devices‘ and the iPhone ‘EllieRose‘ appeared on the left. I doubled clicked on the iPhone name and was prompted with an Authorization required message.
Screenshot: Authorisation required message.
I looked at the ‘Transfer‘ app on iOS and clicked ‘Yes, always‘ to allow access.
Screenshot: Authorisation yes/no on the iPhone
After 20 seconds I can see photos on my iPhone on Windows.
Screenshot, all of my iPhone images were appearing on Window.
I selected all camera albums to backup and clicked backup. I prompt to upgrade to the full version will appear, you will need to buy the upgrade.
Full backup happens only if you upgrade to a full version via an in app purchase
After the full version if purchased the backup will be allowed.
You can upgrade from the free to paid full version from the main screen of the Transfer app.
Screenshot get full access by upgrading the free app to a paid version
The upgrade In App Purchase is $10.99 AUD
Screenshot of a $10.99 In App Purchase on the iPhone
When you purchase the In App Purchase you can run the backup again.
Screenshot of all camera categories on the iPhone on windows
Advertisement:
   


Now I could see photos being copied from the iPhone to my defined backup folder.
Screenshot underway.
I went to bed as I knew there were about 100GB of files on the iPhone and this was going to take a while.
In the morning the backup was done.
Screenshot backup done.
26,000 files were backed up (over 100GB).
I now had a full iPhone backup made by iTunes and a copy of all photos and videos.
Screenshot showing the 2 backups (a) iTunes made and b) Photo Transfer App made)
I turned on the Firewall again.
Screenshot: I re enabled the firewall.
I now had 160GB of backed up photos, videos and phone backup.
160GB of files  from both backups.
Backup your iPhone backups to the cloud
The steps to do this two ways will be added soon.
I will add a section on how you can back up the iTunes and manually synced photo and video backups to the cloud automatically and a more complex but cheaper was for 0.005c per GB.
Watch this post.
How to backup and Android Phone
Article coming soon.
Other Links
Read the official iTunes/iCloud backup guide from Apple here: https://support.apple.com/en-au/HT203977
Backing up your computer automatically with BackBlaze software (no data limit)
Versions
v1.3 Added back blaze article link
v1.2 Added more images (from a phone that does not have the iOS app already)
v1.1 Added an Android heading.
v1.0 Initial Post
Connecting to a server via SSH with Putty
 by 
This post aims to show how you can connect to a remote VM server using  Telnet/SSH Secure shell with a free program called Putty on Windows. This not an advanced guide, I hope you find it useful.
You will learn how to connect (via Windows) to a remote computer (Linux) over the Telnet protocol using SSH (Secure Shell). Once you login you can remotely edit web pages, learn to code, install programs or do just about anything.
Advertisement:


Common Terms (Glossary)
  • Putty: Putty is a free program that allows you to connect to a server via Telnet. Putty can be downloaded from here.
  • Port: A port is a number given to a virtual lane on the internet (a port is similar to a frequency in radio waves but all ports share the same transport layer frequency on the internet). Older unencrypted webpages work on Port (lane 80), older mail worked on Port 25, encrypted web pages work on Port 443. Telnet (that SSH Secure Shell uses) used Port 22. Read about port numbers here.
  • SSH: SSH is a standard that allows you to securely connect to a server over the telnet protocol. Read more here.
  • Shell: Shell or Unix Shell is the name given to the interactive command line interface to Linux. Read more about the shell here.
  • Telnet: Telnet is a standard on the TCP/IP protocol that allows two-way communication between computers (all communicatin issent as characters and not graphics).  Read more on telnet here and read about the TCP protocols here and here.
  • VM: VM stands for Virtual Machine and is a name given to a server you can buy (but it is owned by someone else). Read more here.
Read about other common glossary terms used on the Inetre here: 
https://en.wikipedia.org/wiki/Glossary_of_Internet-related_terms
Background
If you want a webpage on the internet (or just a server to learn how to program) it’s easier to rent a VM for a few dollars a month and manage it yourself (with Telnet/SSH Secure Shell) than it is to buy a $5,000 server, place it in a data centre and pay for electricity and drive in every few days and update it. Remote management of VM servers via SSH/Secure Shell is the way for small to medium solutions.
  • A simple web hosting site may cost < $5 a month but is very limited.
  • A self-managed VM costs about $5 a month
  • A website service like Wix, Squarespace, Shopify or WordPress will cost about $30~99 a month.
  • A self-owned server will cost hundreds to thousands upfront.
There are pros and cons to all solutions above (e.g cost, security, scalability, performance, risk) but these are outside this post’s topic. I have deployed VMs on provides like AWS, Digital Ocean, Vultr and UpCloud for years. If you need to buy a VM you can use this link and get $25 free credit.
I used to use the OSX Operating System on Apple computers. I was used to using the VSSH software program to connect to servers deployed on UpCloud (using this method). With the demise of my old Apple Mac book (due to heat) I have moved back to using Windows (I am never using Apple hardware again until they solve the heat issues).
Also, I prefer to use Linux servers in the cloud (over say Windows) because I believe they are cheaper, faster and more secure.
Enough talking lets configure a connection.
Public and Private Keys?
Whenever you want to connect to a remote server via Telnet/SSH Secure Shell you will need a public and private key to encrypt communications between you and the remote server.
The public key is configured on your server (on Linux you add the public key to this file ~/.ssh/authorized_keys).
The private key is used by programs (usually on your local computer) to connect to the remote server.

How to create a Public and Private Key on Linux
I usually run this command on Ubuntu or Debian Linux to generate a public and private SSH key.
sudo ssh-keygen -t rsa -b 4096
The key below was generated for this post and is not used online. Keys are like physical keys, people who have them and know where to use them can use them.
Output:
Generating public/private rsa key pair.
Enter file in which to save the key (/username/.ssh/id_rsa): ./server
Enter passphrase (empty for no passphrase): ********
Enter same passphrase again: ********
Your identification has been saved in ./server.
Your public key has been saved in ./server.pub.
The key fingerprint is:
SHA256:sxfcyn4oHQ1ugAdIEGwetd5YhxB8wsVFxANRaBUpJF4 [email protected]
The key's randomart image is:
+---[RSA 4096]----+
| .oB**[email protected]       |
|  +.==B.+        |
| o .o+o+..       |
|  .. +..o...     |
|    o ..Sooo.    |
|         ++o.    |
|        .o+o     |
|        .oo .    |
|         ...     |
+----[SHA256]-----+

Advertisement:


The two files were created
server
server.pub
  • “server” is the private key
  • “server.pub“is the public key
Public/Private Key Contents
Public Key Contents (“server.pub”)
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC7Xo9bOCXJ7gVjP8tKOxHVId3KTo5I0VRU/kSRK3+mGd5VbDbQABo3tdWzYhzkjODzRS9TeL2dcLAQNNQKshi9IW5IGDS1NocSCLFQId5BFr9s3E79fkWqcZkKmwocepXOOZ91EDKgIFxviOzZKe99sdxxMoZzi1nxgVyXl4TnaelyiQxeKYniVs1iqDfYWQCxkKsmYit8TvGtOwrhLvKNh9362/y5ebpXVdFlRuB83eF7k8RHNYCQyOJJVx4cwnTIsAN0GMOwjuaOZbp7rR1d6k7RZmaApNRTbaWOXy32UiBST5TV/jXF2UL/4IBnn+yvCrM0v79e/3omgjlVVKfWByFzMv/YlBKCAX3xxtJQ9RkzTqseKupXmmJU0rik6Xuz31N2oyw4M7yJofSUGVCN0pnpKEvnKxqolfD9egdQy2XDaNioY7cvOO1qRegCKE0sDh1m5MzJWMhbDs7macSMyd6+0O5qWc/ByHy0G/mVbd8kO4jIuEzEs4IFkPCToEZp7KfkY7KRkOhccLbQ4ApCesUfBtGGAN1f33NnXCHae3Cx46nSd23fvgDZUVnjI47tNJH5Z8FNVlW/fp5Rgeu/aPUephnDX2IBxwIKQOmSTDY+nxU4V+c93H1gSOJfvqYbVKIAXKyN9Yh6LC44ZvLrL4q0TC0QlH2+kxMLnuj7zw== [email protected]
Private Key Contents (“server”), always keep the private key safe and never publish it.
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,D34670C40CE3778974BEF97094010597

b4oecyqLsWt9n+G12ldVNlaQxSKF1wSrlBPg6FGiHRauTCyreUwoI2dMOAkwnGmN
8fcy51fH7D3Kg0G9fWWNPd+oUDwZmrpB8Mv6Ndk4bLYZEbkNOFgvPwNre7edTBOD
JGZRdWqb+yrywgvz3iTXPNjNK5REU3u3JmD69jInFNo92j765QQKA4sFgEyD/8g+
zg8yefIQAhEsVELC5LXPPyuTfA+x0Q+040PqCJ+FCISJI1CeZjLwk7Fbe453Vj81
zaDsurl5X5gaRUlVjB2asr6etWdMLWcalX4Nbyj2A10L3J4ONjKq3Wc2muJ0Q6ES
oNqBaU2iHPlK8yK0TGj/ERfjaG1qdlhBcow0pSapRqGopXBuVBLVuyc2NHe5CCTk
Ezq+LZGsVYmiOIIY4QRJdEN/DVLFHRGK/xA9A7unm484zXIEO6wznE0DuCTtyZs0
luJ3bKLRcack3K1Dphq0LjSG4YxQlkHewa9k9AKpDPTqeeKKckySakiDCGPT6htk
VqaCKrApAt6GQ2hLVXZ0BFVN5A3WUJ5s+HpFvTUzHTNZcdsVS4PgxhuCtnSO/BdS
/G+ODc4aZJNYQD9QQfWUnxkgnQJCWJ+aBZtKF7eDPRYY7qD9jWxubDzrFplBkmAi
O+aX5N8dpU3lEty4INjyh5LpgZWSswjUhEKWi/c1k+Qd1gCWzYzwAq2BfpWcF8Z+
c+y9lQUKbq2yDlxReCIsfb/hda5k1HjgaUlhKbjWIITSlGqf/NE9i+vj0rQEMQXQ
mxBoilfLUPd5A1ttG5XvqC2ex5HBmjzCazZ13Z/2c/PkwicHBmrf5bKYHZp49niV
44n8tZRamCUv6HaJUaKR22MigOG/qGppGPodGeLNj1DFLYAEQ78SYcVhEqIICBo1
t1yaIemUq8MWXSZz1K3cP4FEXQcEziQxFLU/0DCE0P0mIU3MExUmjB/nVE8vxb5l
p3ej3yrRGe+P2neco2gttgaTEi6l/S+0TIiZNstnVPG48BPW71mwVg9XR1d+avO7
OpXt0UgocX0xp7zBgK2up8Ai6v66WwjoNgyvFe02aK4/+fSC+aJ5D6N7JVNxd/bn
Py4W8oLKnrE1PKtIfBw/aE+rgudaMIyuxCaLllRKyDxVPPiJFp2iFcH/Y+k+0vDa
xE9Jpdd0zOWkZyebAxrS8zAUUNNaTQ+rWkj/zORjE4ptHpdwdazzHoQwIs+1kjsv
e/+JEmoskH7XozLnxClVhhWMXWfgQsPWBqPnGzieW0tv9SeIAU/BLJCHJRhBMAT1
ugBtcda1VMlAPVroYtVyUdCxkYZqGfIDbKqtOvvuBgUIUe/HnC3ExQQycC9F05BH
RJibaM/11MLTcZSO7KOK65Dg2v3VBhe6rfDl4tTR0yOySPXCacb9aMt2pMPTEe0/
wU49wCefchfD2bsR3kXPpUqm+HbkHORpIwsMZfQO/8dooXYdiYUdzV9roXG6OGVQ
SsV/xR2lE3XrR71TBegfRnQirI8tj4psSor+yCj3qV936Oh31D96Z6P4glshibsG
ffWAO/TSdu5ZV+UVahh6bTozs+g+odUu/S48TeI1fk7lPlqwZdjoSHXUI2v1FAQ2
jSSywuZQxHlGhg6OeI052cxx3zcVyVVLFHhIrfvufNc3c3+KYhtyiSzBNYN1BrJi
xNXwlDS1jYWgRHkf9zbNBU0MLTYHjZZvO9Jpl/UhKKBdIvJFwmGmXS2lgU6slunJ
Ojp4tY1tbI520KOskV/OoqEfmhXh5fTlIconzoK1aLqxk1d0d65ONcxqVbAG79RN
b0Q5PgewSOgFlcZ7tEIZKAWsWVhjlFTSGRujdZVM1vZB9fCJesemai7HU0e4J+Do
tqvss8I2n6TPxlTYFzQ4w12pIiOzx/8cFLX78NLN8wQFElhhczeuW5HDAnmPxYhQ
eLY0HgDCFSvVAvGXo0j1gcBUcOr/LzZSsJhxsB7FKyrUjlmD/7Y45WoKJj41bKL+
y4+iDhXyLBiqVClRijsguwiCkmPFiR7Bng2pglS0oIWPWu1UbTJWVJPfuUTOBC+M
4/2fBtgFjUz8iUISs9ncEKkERlxodBIu+ekgLJZAigSMvUKfGE1YB1AA9x96VLjd
VJSjjWvnhMEoSwNzlNQ9+dhoD5Cg9zicgIIKnHnovYGOu8g9ZWfvhJFrKZgkfLRv
r2KgkWiHWpf0swiyGUOlGJDe39nMMkoxib7XE/J3VI3na1ZUOIf8kl9kdHXJ0R3C
2IjdbfiFHEDOrakp5oeVf8BbLK7RB8OlxgJAS47Byh8j97U7f13A5ZYlK3bkZ7E4
h7mCJQozgWP81ut0d9WUlcKp5M8yg2ctZ7h4oeG4Js4ceHqd19Z4P+1xWKwXcdmV
+uhiTftevTu3/UhYQVV4ck98C9pursJJYL5hTnIIpTSWIR+jSahhtzUy/upjugPp
cKi6eGlOkcHdKNRtiu7/IZqni85fC8PAwPZ93SICdiq6BpGaGWFh046weIJuflSK
Pd76+M70YRd+pkaRjJyFJ3hLyg7W5mlOb1+yBIlXKzpbch9B5E4dRHCcOsg4+v/9
exRgAnvUIhR/GpSySDDwgKHg8rAyjjoGeZFH3TJIemAAimyaR608a9tCn7SxVobs
UQlZ9WwC0dQIEv7mSvSige3imbybPtCoBHJAqsJqKCFJEDWbIF5l2VYZcfJUYaEI
oZAJHYGnZm33yQ6eSOusXJ2SnnGZ+ZsGO4bDVSwN20FkSt11gN8Wjrki9CxeVQp7
dWbKX1r/lZw74yUB4cYN23hgLJsdqvM7THzwlBkVtgV74RGY0qv59ecBUSQedlSK
dkOnkmoCiGRSNyf+ebijQaygnfK0ArG5wiRF/RQWiPFj7S6DHRxIOrXqcmvhJ7Ly
NApn9pPYyoZEAbk82MAXkapZ5+YLIKLjdNsYuKq5xVty+mc+FfxLWmZGX+QQinra
Z9DfY9KQw4rxJ/ju4ILnDrygm/QBsNFXBojOuzOIULt7c26s3d/47T+IXA4SIX4v
cPqYa6S3PU/Yoe5/Ya3tFxXmBXgEgVLZuujMs7dyCOAqLEyBEHYqIclp+TElWQLR
V660fczVXeedfd2tNBy1IBj1vhGa9j5mZLbFwTczykwCFfihLIrxSEc1MQA4CaSX
-----END RSA PRIVATE KEY-----

The Public and Private keys is used to encrypt all Telnet/SSH connections and traffic to your server. Keep these key’s private.
Advertisement:


fyi: Putty can create SSH Keys too
If you do not have a Linux computer or Linux server to generate keys the Putty generator can create keys too.
Puttygen generating a key based on the randomness of mouse movements.
I did not know Putty can create keys.
Do save the public and private key(s) that were generated in Puttygen (tip: PPK files are what we are after along with the public key later in this post).
Public keys are added to your server when you deploy them. On Linux, you can add new Pulic keys after deployment by adding them to this file “~/.ssh/authorized_keys” to allow people to log in.
Puttygen does format the keys differently than how Ubuntu generates them. Read more here. I’ll keep generating keys in Linux over Puttygen.
Output of the public and PPK files from Puttygen

Putty SSH Client on Windows
Putty is a free windows program that you can use to connect to serves via SSH. Download and install the Putty program.
Open Putty
Putty Icon
Default Putty User Interface.
Screenshot of the Putty Program
To create a connection add an exiting IP address (server name) and SSH port (22) to Putty.
Screenshot of an IP and port entered into putty
In Putty (note the tree view to the left of the image), You can set the auto login name to use to log into the remote server under the Connection the Data in the tree view item
Screenshot showing the SSH usename being added to putty under Connection then Data menu,

You can also set the username under the Connection then Rlogin section of Putty.
Set the usernmae undser rlogin area of putty
OK, lets add the private SSH Key to Putty.
Putty Screehshot showing no support for standard SSH keys (only PPK files)
It looks like Putty only supports PPK private key files not ones generated by Linux. I used to be able to use the private key in the VSSH program on OSX and add the private key to connect to the server over SSH.  Putty does not allow you to use Linux generated Private keys directly.
Convert your (Linux generated) private key to (Putty) PPK format with Puttygen
Putty comes with a Key Generator/Converter, you can open your existing RSA private key and convert it (or generate a new one).
TIP: If you generate a key in Puttygen don;t forget to ad’d it to your authorized host file in your remote server.
Open Puttygen
Puttygen icon
Click Conversions than Import Key and choose the private key you generated in Linux
Screenshot showing import RSA key to convert
The private key will be opened
Screenshot of imported RSA key
You can then save the private key as a PPK file.
Save the private key as a PPK file
Advertisement:


“server.ppk” Key contents
PuTTY-User-Key-File-2: ssh-rsa
Encryption: aes256-cbc
Comment: imported-openssh-key
Public-Lines: 12
AAAAB3NzaC1yc2EAAAADAQABAAACAQC7Xo9bOCXJ7gVjP8tKOxHVId3KTo5I0VRU
/kSRK3+mGd5VbDbQABo3tdWzYhzkjODzRS9TeL2dcLAQNNQKshi9IW5IGDS1NocS
CLFQId5BFr9s3E79fkWqcZkKmwocepXOOZ91EDKgIFxviOzZKe99sdxxMoZzi1nx
gVyXl4TnaelyiQxeKYniVs1iqDfYWQCxkKsmYit8TvGtOwrhLvKNh9362/y5ebpX
VdFlRuB83eF7k8RHNYCQyOJJVx4cwnTIsAN0GMOwjuaOZbp7rR1d6k7RZmaApNRT
baWOXy32UiBST5TV/jXF2UL/4IBnn+yvCrM0v79e/3omgjlVVKfWByFzMv/YlBKC
AX3xxtJQ9RkzTqseKupXmmJU0rik6Xuz31N2oyw4M7yJofSUGVCN0pnpKEvnKxqo
lfD9egdQy2XDaNioY7cvOO1qRegCKE0sDh1m5MzJWMhbDs7macSMyd6+0O5qWc/B
yHy0G/mVbd8kO4jIuEzEs4IFkPCToEZp7KfkY7KRkOhccLbQ4ApCesUfBtGGAN1f
33NnXCHae3Cx46nSd23fvgDZUVnjI47tNJH5Z8FNVlW/fp5Rgeu/aPUephnDX2IB
xwIKQOmSTDY+nxU4V+c93H1gSOJfvqYbVKIAXKyN9Yh6LC44ZvLrL4q0TC0QlH2+
kxMLnuj7zw==
Private-Lines: 28
DkpbM78GgGBSgfs9MsmZwDJj6HFXdoe+fCP1rLnwbE99mvU6Fbs23hXd+FsVdQbb
VR5tKTocV7tEwGjtLCHSTSF6gap0l4ww0Ecuvr/Dra2CJ2BsntyssBrWnlUT7OlA
M9zKQAzywAy4AHkph0YvH4l7BcJ5V1pUltm2JDTU6+iFqXDsstUUEDcQ4u0EalWU
EEsW+quNSwO0HBHvWY6N7tbiuEN9L+cFYIdsJEDfqM4hNi+7Ym+SQq5FOPyA6gXa
vhujsjPQAWI3TFxh7EIvsPDMCXxWHL6qaDvOMmPPTZDbEvm4nQ5Kax9jWacPILn7
ezc7ZAiZdDiFbkF3TLyuHx71mjChZgLoZLWYfXR3MBEEYnkNO/7oSMRUwDzEyWKW
ZgqdUtGg0cR+qWvaxQTDQsN/DjB7jGgnlreF92S8xSsbk5GgpZnTQ1V0cm3oecB+
+JP90K4Fi979gPWnwTfg6ZvmLUiVz3uBbvegkT9CVZhhZXSKq53H+SjTZfKBPrM9
NHGLkYr1WjToGR39LMrh4X3KChGewMFyuxtpkEQV60eCnHZBHgTco2A0yriRprOP
Ks4qJXOtZnsMYMesUDX9W5wLc4HcRvRh2UBPw/8bPz6mNrBk8j5SPIwBrPMIBejd
4IPoYezaEFKPg2bP7dn+Nftz5CGagcV2g+zhE615dsWzX1P7yu/1dTmz9LXaMmN6
d+zJE8TtjeaoW5NE1HVFlj9rknzJW7xQfokhS5hMkOg6J0AA6Pk13tupu8WHMkVB
x7nVu876f8tIbT8GzXCGgSl+zS7IJO3pt9T9QHIYa+T3oTIUqfBfK1WffUZwHMRn
Xn/VKUtIIIPiVfCtQuxSrTiJzQcoJ/yvfv62YAGv2LsDlBoHfXRdf6h3TCCCOVxT
WE45sbj3gJ1Cgjt1SEd/8A3hkstn2U2NKBI9gkB9H5BbDJoAXq6/4CkwaQvSEzs7
LK5btRlWop+7gqkyMPpgxv9li9IEDJ999ufMqxkFgOBkmkR5Si71elXRnwiKrjfU
Ce14iy7Dd7lb7IU9OEBjWFZlSigVEnc8klhGHDuxnojiW1ld7pUDIkAAbdTMOFON
abcpfNwcg5Y3l+1KwIQHuewAUuA9472jV4V9EAn7pJ7wgmYHbzMzg9Z9dM8h/3UI
axBzAW+cJM80gN+nZMbmDC9FkXV16GSuqC2iQUVGb2TIheAS7oCR+JFZFQNv0ytF
rGQ9K1wIGbMI4oDPcAid7DzrEXVl3d2x8MtwF/WzfHehVJD1h1uNwezLf1gBKyas
9GBfDOYwd8zgaL2H99GYD1Ba7TePJY81mx7m10eYdwDj1vCpboKE3cE6AyL7ki+4
Ix7GSzQs9NBckF9+8eVXe2T4Cc450hIoN0BWcxVUUdGCA1skZ1PczPs1z/ae4lxd
l5WmPy8Gyh7cnZpyqzvwAPSFDadkNP60eekfkRHyo4QyLhj7QZtO0kOgWhT3CHma
FjZ5jJu59U/4gc0TpQ8ra3vgKQKudloExsg027+34nR98dN+zzUj4S2C/J34W98C
DEEu/SO7nfW/a2UARXBKWCbS+3j24zHc9dbgX2tZoAoInUvRGiSOsLVsMhDiBoyb
wWoNxrKPR3Fi5zZ+GfDUgUGpZoW/b54KnFouIHBYbI41Gkh4vj6lxOGh/sb3SPHd
Wg6EN/0z/mer3bG0a2/ZHKYA5KGWRXWYvYLz4Je8fb/egBrSU6BztwSNeilzA9lI
J4BO7pzXECnWYutB14UxHw==
Private-MAC: 12298fa865ac574da81898252e83b812200cba59

Now the PPK key can be added to Putty for any server connection that uses the public key. Use the right key for the right server though.
Add the private key to a Putty server by clicking Connection, SSH, AUTH section and browing to the PPK file.
Screenshot showing the PPK key file added to Putty
Advertisement:


Now we need to save the connection, click back on the Session note at the top of the treeview, type a server name and click Save
Save Putty connection.
Connecting to your sever via Telnet/SSH wiht Putty.
Once you have added a server name, port, usernames and private key to Putty you can double click the server list item to connect to your server.
You will see a message about accepting the public key from the server. Click Yes. This fingerprint will be the same fingerprint that was shown when you generated the keys (if not maybe someone is hacking in the middle of your local computer and server)
Putty messgae box asking to to remember the public key
Hopefully, you will now have full access to your server with the account you logged in with.
Screenshot of an Ubuntu screen after login
Happy Coding.
Alternatives to self-managed VM’s
I will always run self-managed server (and configure it myself) as its the most economical way to build a fast and secure server in my humble opinion.
I have blogged about alternatives but these solutions always sacrifice something and costs are usually higher and performance can be slower.
I am also lucky enough I can do this as a hobby and its not my day job. when you self manage a VM you will have endless tasks or securing your server and tweaking but its fun.
More Reading
Read some useful Linux commands here and read my past guides here. If you want to buy a domain name click here.
If you are bored and want to learn more about SSH Secure shell read this.
Related Blog Posts
Advertisement:


Version: Draft 1.0 Initial Draft
How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains
 by 
This is a quick guide that will show you how you can connect to a cloud server via SFTP with the PHPStorm IDE from Jet Brians and deploy files from your localhost to the cloud.  This is my opinion, I am not paid to promote PHPStorm or UpCloud.
Advertisement:
   


Pre-Requisites/Assumptions
This guide will assume you already have (or know how to)..
I am using Windows 10 Home (with IIS Web Server (document root redirected to S:\Code\) and a pre built Ubuntu Cloud servers.
IIS pointing to S:\Code
Why no FTP? I do not create FTP servers on my serves to increase security and I only access servers via SSH via white-listed IP’s and then authenticate with hardware 2FA keys from YubiCo (read me 2FA guide here and also  how to secure *nix servers and WordPress with 2FA).
Background
2 years ago I used to use the Cloud 9 IDE to connect to, and code files on cloud servers and life was good. I could configure and connect to servers, drag and drop files, run bash scripts from a web page and close the Cloud 9 browsers tabs, travel hundreds of kilometres and log back into C9 and all code and bash scripts would reappear.
Here is the Cloud 9 IDE showing code on the left and a Browser on the right.
C9.IDE showing code on the left and web page on the right
With Cloud 9 code could be easily accessed, edited and run.
See screenshot of code running from a Cloud 9 hosted server with properties windows on the right.
C9 IDE showing  a bash terminal windows and code
Advertisement:
   


Regrettably, I cancelled my $9/m subscription to Cloud 9 after a minor stroke and since then I have gone back to a terminal screen to code and transfer files. In the last 2 years.
Screenshot of the putty program connected to a Ubuntu box editing a file with nano
Uploading and downloading files on mass is painful via pure SSH.
AWS has since purchased Cloud 9 and I am not sure if it will ditch support of non-AWS servers in the future. AWS is good but servers are very expensive for what you get IMHO.  I have found Disk IO on UpCloud is awesome (also UpCloud support is great (I am not paid to say that)).
PHPStorm IDE?
A quick Google of IDE’s like Cloud 9 mentioned PHPStorm from Jet Brains. I have used IntelliJ IDEA from Jet Brains before and PHPStorm seems to be very popular.
Go to 
https://www.jetbrains.com/phpstorm/ and see the features of PHPStorm
Watch PHPStorm in action
 
Whats new in PHPStorm 2019.1
 
Advertisement:
   


Install PHPStorm
Visit https://www.jetbrains.com/phpstorm/download/ and download and install PHPStorm (free trial 30 days)
System requirements
  • Microsoft 10/8/7/Vista/2003/XP (incl. 64-bit)
  • 2 GB RAM minimum
  • 4 GB RAM recommended
  • 1024×768 minimum screen resolution
Pricing
PHPStorm is $8.90/m for individual use (or $19.90/m commercial). For the first 12 months of uninterrupted subscription payments qualify you for receiving a perpetual fallback license (20% discount for an uninterrupted subscription for a 2nd year, 40% discount for an uninterrupted subscription for 3rd year onwards).
PHPStorm work on Windows, OSX or Linux. This great an I use Windows locally and Linux remotely but I’m keen to use Linux locally to match local and remote dev environments.
Official PHPStorm Pricing Page: 
https://www.jetbrains.com/phpstorm/buy/#edition=commercial
fyi: Jet Brains has free licencing for individual use for Students and faculty members.
Creating a Project
Open PHPStorm and select “Create New Project”.
Create New Project screen
Advertisement:
   


Choose a project type on the left (e.g “PHP Empty Project“) and choose a location to save too (I chose “S:\code\php001) on my local machine. I chose “S:\code\php001” on my local machine.
New Project screenshot asking for a name and save location
Choose a folder to save to.
Choose a project and location to save to locally
Click “OK” to create the project.
PHPStorm will have created a project for you. You will notice a “.idea“folder under the location you saved with these files.
  • misc.xml
  • modules.xml
  • php001.xml
  • wordspace.xml
Do not delete these files.
Advertisement:
   


Creating your first PHP file
You can right click on the project root and select New then PHP File
Right click on the root in the tree view then new PHP File
 Or clicking the File then New menu choosing PHP File.
File new PHP File dialog
Name the file (e.g index.php)
Naming a file index.php
The file has been created and its available in my localhost web server.
Screenshot showing PHPStorm with index.php, S:\Code showing index.php and http://localhpst/php001/index.php loading
Advertisement:
   


Creating a Deploy Target
Now we need to specify a deploy target in PHPStorm to push the file changes to the cloud. Backup your server (yes backup your server just in case).
Open your PHPStorm project and click Tools, Deployment and then Configuration.
Click Tools, Deployment and then Configuration.
Click the plus icon near the top left and choose SFTP
Screenshot showing add new deployment server (SFTP)
Advertisement:
   


Name the deployment target (e.g “server (project)”)
Screenshot of an input box showing a server name
  • Enter your “server name” or IP and port
  • Enter your “ssh username” (ensure the SSH user had write access to the wwwroot folder and the web server can read the files written by this user)
  • Under password I chose “Key pair OpenSSH or Putty (as I had SSH details already setup in Putty details
  • You can add your ppk private key from Putty (use the puttygen program to conbvert ssh public and private kets to ppk format)
  • If you have a passphrase on your SSH key add it now
  • Enter your web servers remote path (for the project)
  • Enter your web server URL
Screenshot showing a server name, port, username, password, ssh file passphrase, root path and web server url. 
I did SSH to my remote server and created the destination folder. This will ensure I can deploy code here (PHPStorm does not create the remote path fpor you ).
mkdir /wwwroot/php001
chown -R www-data:www-data /wwwroot/php001/
Advertisement:
   


Click Test Connection
Test Successful screenshot
No we need to click the Mappings tab and add a mapping.
  • Local path is your local path
  • Deployment path is / (the web root path is carried forward from the previous tab)
  • Web path is the web path that is entered in the browser
Screenshot showing a manual file mapping of local and remote file locations
Click Add New Mapping. Now we are ready to deploy
Deploying code to the cloud
I right clicked on the root note in PHPStorm and created an index.php file.
Creating an index.php file by file new
Advertisement:
   


I edited the index.php on my local machine and then click the Tools then Deployment and choose “Upload to fearby.com (php001)” menu.
Manual upload available in Tools menu then Deployment menu
The File Transfer output window showed the transfer progress.
Screenshot showing the file transfer window output saying the file uploaded.
I loaded https://fearby.com/php001/index.php in Google chrome. It worked.
Screenshot showing https://fearby.com/php001/index.php loaded in a bowser
Advertisement:
   


Don’t forget to turn off Automatic uploads under Tools, Deployment menu.
'Screenshot showing Automatic updated turned on
Now when I create new files or change existing files they will auto upload.
Sourc Control
I will add this soon.
Shell Command
You can also open an SSH console to the server and run commands
e.g zip files
zip -r backup.zip .
I can also open a folder window in PHPStorm and show all remote files by clicking Tools then Deployment then Show Remote Files, Zip files can be easily downlaoded or other files uploaded. Nice.
Screenshot showing remote files
Linux Client
I will review the Linux PHPStorm client soon.
Advertisement:
   


Troubleshooting
Watch the Official guide on Deployment and Remote Hosts in PhpStorm – PhpStorm Video Tutorial
 
Good Luck. I hope this guide helps someone.
Version 
1.2 Removed advertisements
1.1 Minor Updates
1.0 Initial Version
Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution
 by 
This is how I replacing Google Analytics with Piwik/Matomo for a locally hosted privacy-focused open source analytics solution
Advertisement:

 
 
Aside
I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. PHP is my programming language of choice.
Now on with the post
Google Analytics
I will fully admit Google Analytics is good. I posted this a while ago on how you can set up Google Analytics on your site.
Google Analytics has some great charts and graphs. Simple to set up and easy to use.
Analitics Home
My site traffic is growing and I would prefer to hold my own analytics on user data. Matomo is an analytics solution that stays on my server and not in the hands of Google.
Blog Growth
Google Analytics can be Slow
Sometimes the Google Analytics server is slow (affecting the speed of my server). I blogged recently about speeding up a WordPress site here and Google Servers were not adding expiry headers on assets.
I did log a ticket with Google to fix this and the experience was terrible.
Support for Google Analytics is terrible
Gogole Analytics support of terrible
GT Metrix scores show poor delivery of tracking assets.
Google Slow Assets
Privacy
After the Cambridge Analytica fiasco (that made me decide to delete facebook) sending analytics to Google is not a good idea.
I am not saying Google is evil but I want my site’s visitors tracking data to remain local.
Website Speed Benchmark before installing Matomo
I can load my site in 1.3 seconds at best, 1.5 seconds on average and 2.0 seconds at worst. My site is loading 11 assets.
GTmetrix 1.3 second page load time
Page Speed Scores
GTMerix page speed load times
Y Slow Scores, Gogol Assets are reporting no expiry headers (slowing down scores)
GTMetrix yslow load times
Advertisement:

 
 
Google Analytics tracking assets are slow.
Gmetrix waterfall list
Optimizations to be made
Browser caching is not possible with Google Analytics.
Gogole lacking browser caching
Missing Expiry Headers (I can see a Google Tag Manager server is slowing down my servers benchmark score)
Google lacking Expiry Headers
Why Mamoto (instead of Google Analytics)
I came across
Mamoto was mentioned
I visited https://matomo.org/
Mamoto webpage
Snip
> Take care of running Matomo yourself by installing it on your own server. There is no cost for Matomo itself but you need a server and update Matomo & your server regularly to keep it fast and secure. Need help? The Matomo team provides free help resources and paid support.
Mamoto Setup Instruction Guide
Source Code
Source code is available.
> Matomo is the leading open alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites, apps & the IoT and visualise this data and extract insights. Privacy is built-in. We love Pull Requests! https://matomo.org/
https://github.com/matomo-org/matomo
Installation Guide
I read the installation guide here https://matomo.org/docs/installation/
You can view the change log here https://matomo.org/changelog/
Downloading Mamoto
I logged into my server via SSH and downloaded the 18MB download to the desired folder
I unzipped the zip file
I loaded the URL where Matoto was installed (e.g “https://fearby.com/folder/subfolder/matomo/”)
I received this well-crafted error.
Matomo File Permission Error
Raw Output
I refreshed the page after running the commands above on my site (via SSH)
Matomo Setup Step 1
A system check was performed. I installed when PHP 7.2.11 was the latest, PHP 7.2.12 or higher might be available. Follow my guide to update PHP on Ubuntu.
System Check
I had one Issue with Freetype not being installed.
Install Freetype
I solved this error by installing freetype
Output
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'freetype-tools' for glob 'freetype*'
Note, selecting 'freetype2-demos' for glob 'freetype*'
The following NEW packages will be installed:
  freetype2-demos
0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded.
Need to get 123 kB of archives.
After this operation, 728 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 freetype2-demos amd64 2.8.1-2ubuntu2 [123 kB]
Fetched 123 kB in 0s (965 kB/s)
Selecting previously unselected package freetype2-demos.
(Reading database ... 122574 files and directories currently installed.)
Preparing to unpack .../freetype2-demos_2.8.1-2ubuntu2_amd64.deb ...
Unpacking freetype2-demos (2.8.1-2ubuntu2) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up freetype2-demos (2.8.1-2ubuntu2) ...
Then I installed “php-gd”
Output:
I refreshed the Matomo setup wizard page, Freetype is now installed 🙂
FreeType is installed
Advertisement:

 
 
Database Settings
For the life of me, I could not get Matomo to talk to a database on another server so I set it up on my localhost.
I used this guide to help in mysql CLI to create the database and users.
Enter Matomo Database settings
Commands in mysql to create a database and user and assign the user to the database. If you are not comfortable with MySql CLI you can use Adminder GUI.
I used this PHP code to test connecting to the dedicated server before using the localhost
<?php
$servername = "localhost";
$username = "databaseuser";
$password = "#################";
$dbname = "tbdatabasename";

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);

// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
} else {
        echo "Connection Success";
}

$conn->close();
?>
Database created ok
Database OK
I created a Matomo user then I grabbed the javascript tracking ID code so I could paste this into WordPress.
Matomo Tracking ID
I opened my WordPress theme settings and deleted the Google tracking tags and added the Matomo tracking code.
Delete Google Tracking Tags
I added the Matomo tracking javascript in the head section.
The dashboard is up and collecting data.
Matomo Dashboard
Some reports are missing data so I will come back later.
After 1 week I could see data
Matomo is not collecting daya
Advertisement:

 
 
Securing Mamoto
I read this guide here to secure Matomo
Opt Out Tracking
I enabled Opt Out Tracking in the Mamoto settings and added the generated opt-out code to my front page and at the bottom or all existing articles.
I had to allow iframe tags on my site by adding this header in NGINX (previously I blocked iframes)
add_header X-Frame-Options sameorigin
Add Opt Out Tracking Code to WordPress.
Matomo Opt Out Added to WordPRess widgets
I updated my privacy page and my GDPR notification bar. Now visitors will see a opt out of tracking on the front page and all article pages.
Opt out of tracking enabled
SMTP Settings
I added my GSuite mail server settings to enable sending of reports via email. I loaded my old guide here to get the GSuite SMTP settings.
GSuite SMTP Settings Added
I enabled force https on the Mamoto application (edited: config/config.ini.php file)
[General]
...
force_ssl = 1
Matomo Plugins (Marketplace)
I opened the System then Plugins section of Matomo to open the Marketplace
Plugins
I installed these plugins
  • Force SSL
  • HidePasswordReset
  • Google Authenticator
  • Device Pixel Ratio
  • Bandwidth
  • Js Tracker Force Async
  • Treemap Visualization
  • Security Info
  • Custom Alerts
  • IP Reports
  • Live Tab
  • etc
Updating PHP
Matomo Admin (Panel – Security/Diagnostics) section will report if your PHP gets out of date.
Matomo warning of PHP being out of date
Hardening Advice
I enabled 2fA Authorisation at logins (Google Analytics Plugin).
Matomo 2fa Login screenshot
Read my guide here on hardware 2FA YubiCo YubiKeys here.
php.ini hardening changes
Matomo also recommended some php.ini file changes.
> open_basedir – open_basedir is disabled. When this is enabled, only files that are in the given directory/directories and their subdirectories can be read by PHP scripts. You should consider turning this on. Keep in mind that other web applications not written in PHP will not be restricted by this setting.
> upload_tmp_dir – upload_tmp_dir is disabled, or is set to a common world-writable directory. This typically allows other users on this server to access temporary copies of files uploaded via your PHP scripts. You should set upload_tmp_dir to a non-world-readable directory
This may break your WordPress so enable at your own risk. I might move Mamoto to a dedicated “analytics” subdomain then enable these options.
Advertisement:

 
 
Troubleshooting
I had to run this command when installing Device Pixel Ratio, Device Network Information, Bandwidth plugins
Output:
GTMetrix (After)
GT Metrix reports that my site is not slower (still 1.5 seconds)
GTMetrix After Pagespeed
I can see that some JavaScript is not being picked up by CDN.
GTMetrix After YSlow
Also 2 More files loading (when compared to Google Analytics)
2 More Files
Time to add the Mamoto files to my CDN.
Adding Matomo Resources to a CDN
I read this Matomo forum post.
I copied these 2 assets to my WordPress wp-content folder (my WordPress CDN ewww.io will then upload them to the CDN).
I have cache everything enabled in ewww.io and this will copy the javascript assets ot my CDN.  I will need to manually update these js files each time a Matomo update is installed.
I change my Matomo tracker code to include the new CDN location
<!-- Matomo -->
<script type="text/javascript">
  var _paq = _paq || [];
  /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
  _paq.push(['trackPageView']);
  _paq.push(['enableLinkTracking']);
  (function() {
    var u="//fearby.com/utils/matomo/";
    _paq.push(['setTrackerUrl', u+'piwik.php']);
    _paq.push(['setSiteId', '1']);
    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript'; g.async=true; g.defer=true; g.src='https://fearby-com.exactdn.com/wp-content/piwik.js'; s.parentNode.insertBefore(g,s);
  })();
</script>
<!-- End Matomo Code -->


Advertisement:

 
 
I could not find out how to change the location of my (now CDN cached https://fearby-com.exactdn.com/wp-content/optOut.js) so I temporarily disabled the opt-out form on my front page.
todo: Find out how to change the CDN location of optOut.js and re-enabled the form.
All assets are loading from CDN.
GT Metrix shows my site loads in 1.4 seconds
Analytics Reporting
Graphs are not as pretty as Google Analytics but they are working.
Matomo is not collecting daya
Mobile Reporting
Mobile reporting is good too.
Screenshot of the Matomo Mobile app
Updating Matomo Plugins 
Don’t forget to update your plugins from the Matomo dashboard.
Updating Matomo (Core)
Matomo has an official guide on how to update Matomo here.
I do not have FTP so I will perform the manual three step update.
But before I do that I will manually backup my web server and database server just in case.
I backed up my Matomo config (I SSH”ed to the server)
$ cd /www-root/matomo-root/
$ cp ./config.ini.php ./config.ini.3.x.x.php
I navigated to the folder above my Matomo folder
Advertisement:

 
 
 
$ cd ..
$ cd ..
I downloaded Matomo
$ wget https://builds.matomo.org/matomo.zip
I unzipped the zip file
$ unzip -o matomo.zip
I removed the matomo.zip file
$ rm matomo.zip
I loaded the Matomo Login page again and was prompted to update the database.
Matomo Database Update Required
Matomo reported it was updated Successfully.
Matomo was updated message
Oops and error in config error appeared when I tried to login.
Matomo Error in config
Oh, Do I need to replace the config file with my backed up config file?
(edit: Yes Matomo say to do this, my bad)
Ten seconds later I accidentally deleted all my config files (I had zero backups), the next 2 minutes were spend shutting down my servers (web and db) and restoring them from backup. Thanks goodness UpCloud are awesome hosts.
I now had to restore my servers and repeat the steps but this time restore my config file before logging back in.
I did this but had thew same error
> An error occurred
 > Authentication object cannot be found in the container. Maybe the Login plugin is not activated?
 > You can activate the plugin by adding:
 > Plugins[] = Login under the [Plugins] section in your config/config.ini.php
I checked my replaced config.ini.php and it did have
> [PluginsInstalled]
 > PluginsInstalled[] = “Login”
I googled and found this page that said reset your password (this was not an option as Matomo was not loading)
I logged into mysql with my Matomo user
> mysql -u matomodbusername -p
 > Enter password:
 > Welcome to the MySQL monitor. Commands end with ; or \g.
 > Server version: 5.7.xxxx
> Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
> Oracle is a registered trademark of Oracle Corporation and/or its
 affiliates. Other names may be trademarks of their respective
 owners.
> Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
> mysql> show databases;
 +——————–+
 | Database |
 +——————–+
 | information_schema |
 | matomodb |
 +——————–+
 > 2 rows in set (0.00 sec)
The account and database seems ok.
Advertisement:

 
 
 
I tried “FLUSH PRIVILEGES;” with no luck
I tried to sop mysql but it was locked
It was late so I rebooted my server (it did not come back up after a few minutes, I forced a reboot)
I still had an “Authentication object cannot be found in the container.” error when trying to login to Matomo???
I re-checked the “config.ini.php” file after reding threads at the Matomo Forums
$ sudo nano /www-root/matomo-root/config.ini.php
“Plugins[] = “Login”” was not in the “[Plugins]” area of the file???  I added it, saved the change and was able to reload the Matomo GUI.
I checked some key reports.
Visitors over time:
Visitors over time report
Visitor Location Map
Visitor Location Map
Visitor Overview
Visitor Overview
Out links Clicked
Out links Clicked
Nice
I subscribed to the Matomo newsletter here to keep up to date with Matomo update releases: https://matomo.org/newsletter/
Advertisement:

 
 
 
Good luck and I hope this guide helps someone
Ask a question or recommend an article
     
Your Name (required)
 
Your Email (required)
 
Your Question
 
Revision History
v1.2 Hardening info
v1.1 Updating Matomo
v1.0 Initial post
Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
 by 
This is a draft post showing how you can monitor the performance of a server (or servers) with NixStats and receive alerts by SMS, Push, Email, Telegram etc
fyi: This is not a paid post, this is just me using the NixStats software to monitor my servers and send alerts.
Advertisement:

 
 
Finding a good host
If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.
Buy a domain name here
Domain names for just 88 cents!
Monitoring Servers
The post below will show you how you can monitor servers online with https://nixstats.com/ and send alerts when resources reach limits or servers fail.
https://nixstats.com/
I signed up and started a Nixstats (14 day free trial).
Start Nixstats Trial
After I created an account I was emailed by Nixtsats with agent install instructions for Linux (1 line). I was also advised to add contacts and to set up alerts.
I logged into the Nixstats settings and set up…
  • My Timezone
  • Default reporting period
  • First name and Surname
  • Reporting emails
  • etc
Nixstats Subscription Upgrade
Subscription options
  • Free (5 monitors, 1 server, 24-hour data retention etc)
  • Founder (25 monitors, 10 servers, 30-day data retention etc)
  • Business (100 monitors, 15 servers, 60-day data retention etc)
Subscription Options
I enabled the limited founder subscription so I can monitor 10x servers (this deal is too good to miss). I tried creating a status page myself last year and it is terribly hard.
Subscriptiosn page
I am now out of the free trial period 🙂 Let’s start monitoring many servers.
Subscription Activated
I enabled two factor Auth to Nixstats logins
Nixstats Two Factor Auth
I created a Nixstats API key for future use (watch this space)
Create API Key
I installed the Nixstats agent (the dashboard gave a 1 line command you can run as root to install the agent (on Linux)).
Instal Nixstats Agent
FYI: Command (######################## is a number linked to your account)
Output
nixstatsagent.sh 100%[====================================================>] 37.80K –.-KB/s in 0.1s
2018-10-02 09:53:56 (338 KB/s) – ‘nixstatsagent.sh’ saved [38708/38708]
Found Ubuntu …
 Installing …
 Installing Python2-PIP …
 Installing nixstatsagent …
 Generation a server id …
 Got server_id: ######################
 Creating and starting service
 Created symlink /etc/systemd/system/multi-user.target.wants/nixstatsagent.service -> /etc/systemd/system/nixstatsagent.service.
 Created the nixstatsagent service
Server Dashboard
 below is a summary of all connected servers ( https://nixstats.com/dashboard/servers ).
Server Sumamry
Monitor Setup
I set up a number of monitors to monitor ping replies and https traffic
Monitors
Advanced Monitoring
I can also set the monitor credentials, timeouts, retries, auth methods, max redirects and frequency. If you server blocks login or resource GET attempts you may need to whitelist IP’s. IP’s of monitoring servers are located here https://nixstats.com/whitelist.php
Monitoring advanced options
Monitor Summary
The default dashboard is very informative. Feel free to create your own dashboards that focus on your own infastructure or apps.
Monitor Summary
Individual Server Reports
Advertisement:

 
 
You can click on a server and monitor it in detail.
Nixstats Graphs
Server Memory Graphs
Long-term memory graphs.
Memory Graph
Install Optional Nixstats Plugins
Nixstats offers many plugins to monitor software that is installed on your server (e.g NGINX, MySQL, PHP etc).
1) NGINX Monitoring (Plugin)
To enable NGINX monitoring I read https://help.nixstats.com/en/article/monitoring-nginx-50nu7f/
I edited my NGINX sites-enabled config.
I added the following
server {
    listen 127.0.0.1:8080;
    server_name localhost;
    location /status_page {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }
}
I tested, reloaded and restarted NGINX
The status page will only be available on the local machine, I tested the page on the local machine
It’s Working.
I edit /etc/nixstats.ini
I remove comments before these lines to enable the plugin
I ran the following command to see if NGINX monitoring is possible
Output
nginx:
{
    "accepts": 39,
    "accepts_per_second": 0.0,
    "active_connections": 6,
    "handled": 39,
    "handled_per_second": 0.0,
    "reading": 0,
    "requests": 119,
    "requests_per_second": 0.0,
    "waiting": 5,
    "writing": 1
}
It’s Working
I restarted the nixstatsagent
I can now view NGINX properties like active_connections in my dashboard. 🙂
2) Enable PHP-FPM Monitoring (Plugin)
Looks like a PHP-FPM monitoring was recently added lets set that up too. Read my guide on setting up PHP child workers here.
To enable PHP-FPM monitoring I read https://help.nixstats.com/en/article/monitoring-php-fpm-1tlyur6/
I edited my PHP-FPM ini file
I added the following line
Restart PHP
I added the following to /etc/nginx/sites-enabled/default localhost server block added above Note I use php 7.2 below (read more here).
server {
    listen 127.0.0.1:8080;
    server_name localhost;
location /status_phpfpm {
 access_log off;
 allow 127.0.0.1;
 deny all;
 include fastcgi_params;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 fastcgi_pass unix:/run/php/php7.2-fpm.sock;
 log_not_found off;
 }
 }
I tested, reloaded and restarted NGINX
I restart PHP-FPM
Enabled the plugin in /etc/nixstats.ini
I tested the status page
Output:
{
	"pool":"www",
	"process manager":"static",
	"start time":1538654543,
	"start since":178,
	"accepted conn":28,
	"listen queue":0,
	"max listen queue":0,
	"listen queue len":0,
	"idle processes":49,
	"active processes":1,
	"total processes":50,
	"max active processes":2,
	"max children reached":0,
	"slow requests":0
}
Advertisement:

 
 
I tested the agent
Output:
phpfpm:
{
    "accepted_conn": 51,
    "accepted_conn_per_second": 0.0,
    "active_processes": 1,
    "idle_processes": 49,
    "listen_queue": 0,
    "listen_queue_len": 0,
    "max_active_processes": 2,
    "max_children_reached": 0,
    "max_listen_queue": 0,
    "pool": "www",
    "process_manager": "static",
    "slow_requests": 0,
    "start_since": 318,
    "start_time": 1538654543,
    "total_processes": 50
}
I can now query PHP-FPM status values in Nixstats 🙂
Query PHP-FPM
Enable MySQL Monitoring (Plugin)
To enable MySQL monitoring I read https://help.nixstats.com/en/article/monitoring-mysql-1frskd8/
I edited the nixstats.ini
I enabled the mysql section in nixstats.ini and added my mysql credentials
[mysql]
enabled=yes
username=mysqluser
password=#######################
host=127.0.0.1
database=mysql
port=3306
socket=null
I ran this command to test MySQL querying
nixstatsagent --test mysql
mysql:
Load error: No module named MySQLdb
I had an error.
A quick Google revealed I had to install a mysql python module.
Allow localhost to connect to MySQL
Edit /etc/mysql.cnf and allow all localhost and external connections (I could not bind to localhost and an external IP at the same time)
bind-address    = 0.0.0.0
TIP: Ensure you have firewalled access to your MySQL server, never open it up without protection.
Let’s try again
nixstatsagent --test mysql
Output
Nice,
I restart MySQL
I restart my Nixstats service
service nixstatsagent restart
Now let’s monitor MySQL in Nixstats
I can now view MySQL metrix
MySQL MEtrix
Status Page
Nixstats allows you to create a status page ( https://nixstats.com/pages/overview ) where you can add any servers or monitors to that page. This stats page is truly awesome, it builds a live status page based on data coming from your installed agents.
You can even set up a custom subdomain that points to a Nixstats hosted status page too (e.g https://status.yourdomain.com).
FYI: An SSL certificate on your staus page may take a few hours to set up. Don’t panic if it is not instantatly available.
Custom Status Page
Nice.
This saves doing it yourself. The status page will look like it running on your server.
Status Page
You can create a status page that automatically aggregates collected data from monitors and displays them in a nice layout.
Status Page
This is great, I used to do my own status pages but not anymore.
Alerts
I added a contact so I could receive alerts. I could then add my mobile, email and PushOver key (to receive push notifications) and Telegram Bot API token.
Contact
Test Alerts
I sent a test alert to each service against the contact.
Test Alerts
Advertisement:

 
 
I activated a Pushover licence on my Android device for about $7.49 AUD (one off) to ensure I keep getting Push Notifications.
Bought licence for PushOver
Nixstats have links that show you how you can create a Telegram Bot and Pushover.net account.
Pushover will cost about $5 USD one off per device (see faq).
I created the following alerts
Alert: Disk Usage higher than 90%
Alert Disk Usage higher than 90%
Alert: Load greater than 90 per cent for 1 minute
Alert load greater than 90 percent for 1 minute
Alert: Less than 5 percent memory free.
Alert less than 5 percent memory free
Summary of alerts.
Alert Sumamry
I also added a CPU reached 95% one for 5 mins alert too (but it’s not pictured above)
I forgot to specify alert recipients and methods for each alert so I edited each alert and added the contact.
Selected Alert Recipients and methods
Now it’s time to test the alerts.
I shut down a server to test alerts
Alerts to my defined Email, SMS, Push and Telegram are working a treat 🙂
Alerts Working
After I rebooted the server I also received alerts about the server being back up.
The status page showed the server that was offline too.
Server Offline
Nice
Troubleshooting
I had an issue instaling the agent on Debian
I ran the following command
nixstatsagent.sh: line 508: [: Installer exited with error code 0. See nixstatsagent.log for details.: integer expression expected
An error occured, please check the install log file (nixstatsagent.log)!
Contents of nixstatsagent.log
I asked the Nixstats chat help and I was advised I had a dead repository (I removed this (editing the dead repo in the appropriate file in /etc/apt/) and all was ok)
I had trouble testing my Telegram alerts but it was my fault as I forgot to follow the bot account I created. Telegram does not allow message from a user (bot) unless you follow them.
A chat with the Nixstats staff sorted me out. Thanks, Nixstats chat team.
Nixstats chat
I had an issue with a missing python mysql package
I solved it by instaling python-mysqldb
Nixstats Help
Nixstats have a help subdomain: https://help.nixstats.com/en/
Nixstats Help
Error Logs Plugin
I did ask Nixstats on Twitter and they said they are working on a logging plugin, I can’t wait for that.
I now have access to beta log features and can see log tabs in Nixstats
I had or check the version of my rsyslogd
I edited: /etc/rsyslog.d/31-nixstats.conf
I pasted
##########################################################
### Rsyslog Template for Nixstats ###
##########################################################
$WorkDirectory /var/spool/rsyslog # where to place spool files
 $ActionQueueFileName fwdRule1 # unique name prefix for spool files
 $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
 $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
 $ActionQueueType LinkedList # run asynchronously
 $ActionResumeRetryCount -1 # infinite retries if host is down
template(name=”NixFormat” type=”string”
 string=”<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [[email protected] tag=\”rsyslog\”] %msg%\n”
 )
action(type=”omfwd” protocol=”tcp” target=”log.nixstats.com” port=”514″ template=”NixFormat”)
 #################END CONFIG FILE#########################
I restarted the rsyslog service
Live Log Output
I can see a live log from (unknown) logs.
I can see the firewall blocking access to certain ports.
Live Log
Search logs
Search
Blacklist Checking (Beta) 
Nixstats tweeted “We just launched a new blacklist check feature. Monitor your IP and hostname reputation. Free during the Beta!”
I enabled it.
I received a Blacklist notification
I requested removal at junkmailfilter.com
Thanks Nixstats
Conclusion
This is one of the best software packages I have seen in a while. I have developed status 
pages in the past (no more). Great work Nixstats.
Please check out Nixstats today, they are awesome. Signup for a free account and consider the limited time founder plan (it’s a bargain).
Nixstats Live chat support is awesome
Server Plug
If you need a server, consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.
https://www.upcloud.com/register/?promo=D84793
Advertisement:

 
 
Ask a question or recommend an article
     
Your Name (required)
 
Your Email (required)
 
Your Question
 
Revision History
v1.3 Blacklist beta
V1.2 Logs beta
v1.1 Logging Tweet
v1.0 Initial Post
Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx
 by 
This is a quick post that shows how I set up the “Feature-Policy”, “Referrer-Policy” and “Content Security Policy” headers in Nginx to tighter security and privacy.
Advertisement:

 
 
Aside
If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.
Buy a domain name here
Domain names for just 88 cents!
Now on with the post.
Add a Feature Policy Header
Upon visiting https://securityheaders.com/ I found references to a Feature-Policy header (WC3 internet standard) that allows you to define what browse features you webpage can use along with other headers.
Google mentions the Feature-Policy header here.
Browser features that we can enable or block with feature-policy headers.
  • geolocation
  • midi
  • notifications
  • push
  • sync-xhr
  • microphone
  • camera
  • magnetometer
  • gyroscope
  • speaker
  • vibrate
  • fullscreen
  • payment
Feature Policy Values
  • * = The feature is allowed in documents in top-level browsing contexts by default, and when allowed, is allowed by default to documents in nested browsing contexts.
  • self = The feature is allowed in documents in top-level browsing contexts by default, and when allowed, is allowed by default to same-origin domain documents in nested browsing contexts, but is disallowed by default in cross-origin documents in nested browsing contexts.
  • none = The feature is disallowed in documents in top-level browsing contexts by default and is also disallowed by default to documents in nested browsing contexts.
My Final Feature Policy Header
I added this header to Nginx
This essentially disables all browser features when visitors access my site
add_header Feature-Policy "geolocation none;midi none;notifications none;push none;sync-xhr none;microphone none;camera none;magnetometer none;gyroscope none;speaker self;vibrate none;fullscreen self;payment none;";
I reloaded Nginx config and restart Nginx
Feature-Policy Results
I verified my feature-policy header with https://securityheaders.com/
Feature Policy score from https://securityheaders.com/?q=fearby.com&followRedirects=on
Nice, Feature -Policy is now enabled.
Now I need to enable the following headers
Add a Referrer Policy Header
I added this header configuration in Nginx to prevent referrers being leaked over insecure protocols.
Referrer-Policy Results
Again, I verified my referrer policy header with https://securityheaders.com/
Referrer Policy resu;ts from https://securityheaders.com/?q=fearby.com&followRedirects=on
Done, now I just need to setup Content Security Policy.
Advertisement:

 
 
Add a Content Security Policy header
I read my old guide on Beyond SSL with Content Security Policy, Public Key Pinning etc before setting up a Content Security policy again (I had disabled it a while ago). Setting a fully working CSP is very complex and if you don’t want to review CSP errors and modify the CSP over time this may not be for you.
Read more about Content Security Policy here: https://content-security-policy.com/
I added my old CSP to Nginx
I then imported the CSP into https://report-uri.com/home/generate and enabled more recent CSP values.
I restarted Nginx
I loaded the Google Developer Console to see any CSP errors when loading my site.
CPS Errors
I enabled reporting of CSP errors to https://fearby.report-uri.com/r/d/csp/enforce 
Fyi: Content Security Policy OWASP Cheat Sheet.
You can validate CSP with https://cspvalidator.org
Now I won’t have to check my Chrome Developer Console and visitors to my site will report errors. I can see my site’s visitors CSP errors at https://report-uri.com/
report-cri.com Report
Content Security Policy Results
I reviewed the reported errors and made some more CSP changes. I will continue to lock down my CSP and make more changes before making this CSP policy live.
I verified my header with https://securityheaders.com/
Security Headers report from https://securityheaders.com/?q=https%3A%2F%2Ffearby.com&followRedirects=on
Advertisement:

 
 
Testing Policies
TIP: Use the header name of “Content-Security-Policy-Report-Only” instead of “Content-Security-Policy” to report errors before making CSP changes live.
I did not want to go live too soon, I had issues with some WordPress plugins not working in the WordPress admins screens.
Reviewing Errors
Do check your reported errors and update your CSP often, I had a post with a load of Twitter related errors.
Do check report-uri errors.
I hope this guide helps someone.
Please consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.
https://www.upcloud.com/register/?promo=D84793
Ask a question or recommend an article
     
Your Name (required)
 
Your Email (required)
 
Your Question
 
Revision History
V1.3 https://cspvalidator.org
v1.2 OWASP Cheat Sheet.
v1.1 added info on WordPress errors.
v1.0 Initial Post
Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space
 by 
Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space
Advertisement:

 
 
If you have not read my previous posts I have now moved my blog from Vultr to the awesome UpCloud host (signup using this link to get $25 free credit).
Recently I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.
Spoiler: UpCloud performance is great.
Upcloud Site Speed in GTMetrix
Why Upgrade
I have 1 CPU, 1 GB memory and 50GB storage and it is running well?  I have PHP child workers set up and have set up the preferred use of memory over swap file usage.
View of htop querying processes on Ubuntu
Before UpCloud, when I had 512MB ram on Vultr I had multiple NGINX crashed a day so I used a bash script and scheduled a cron job to clear memory cache when memory fell below 100MB (view the script here).  To further increase the speed of the WordPress I have configured the OS to use memory over the disk.  About once a day free memory falls below 100MB (this is not a problem as my script clears cached items automatically).
Graph of memory falling below 100MB every day
I’d like to add more memory as I am working on some things (watch this space) and I will use the extra memory. I’d prefer the server is set up now for the expected workload.
How to Upgrade
This is how I upgraded from 1xCPU/2GB Memory/50GB Storage/2TB Transferred Data to a 2 CPU/4GB Mmeory/80GB Storage/4TB Transferred Data server.
UpCloud Pricing: https://www.upcloud.com/pricing/
Pricing table form https://www.upcloud.com/pricing/
Upgrade an UpCloud VM
I shut down my existing VM. Read this guide to setup a VM.
Login to the UpCloud dashboard, select your server (confirm the server has shut down) and click General Settings, choose the upgrade and click Update.
Upgrade the Server, shut it down the server and choose upgrade
I confirmed the upgrade options (2x CPU, 4096 MB Memory).
Confirm Upgrade Options
Click Update
Upgrade Applied
After 10 seconds you can start your server from the UpCloud server.
I confirmed the CPU upgrade was visible in the VM
Software Tweaks Post Upgrade.
I added these settings to the top of /etc/nginx/nginx.conf to ensure the extra CPU was used.
I increased PHP FPM ( /etc/php/7.2/fpm/php.ini ) to increase the memory usage and child workers. I doubled child workers and max memeory limit.
I restarted NGINX and PHP
I tweaked WordPress max memory limits
MySQL Tweaks: I logged into MySQL
I ran “SHOW GLOBAL STATUS” to view stats
mysql> SHOW GLOBAL STATUS;
+-----------------------------------------------+--------------------------------------------------+
| Variable_name                                 | Value                                            |
+-----------------------------------------------+--------------------------------------------------+
| Aborted_clients                               | 0                                                |
| Aborted_connects                              | 0                                                |
| Binlog_cache_disk_use                         | 0                                                |
| Binlog_cache_use                              | 0                                                |
| Binlog_stmt_cache_disk_use                    | 0                                                |
| Binlog_stmt_cache_use                         | 0                                                |
| Bytes_received                                | 3179986                                          |
| Bytes_sent                                    | 223872114                                        |
| Com_admin_commands                            | 0                                                |
| Com_assign_to_keycache                        | 0                                                |
| Com_alter_db                                  | 0                                                |
| Com_alter_db_upgrade                          | 0                                                |
| Com_alter_event                               | 0                                                |
| Com_alter_function                            | 0                                                |
| Com_alter_instance                            | 0                                                |
| Com_alter_procedure                           | 0                                                |
| Com_alter_server                              | 0                                                |
| Com_alter_table                               | 0                                                |
| Com_alter_tablespace                          | 0                                                |
| Com_alter_user                                | 0                                                |
| Com_analyze                                   | 0                                                |
| Com_begin                                     | 0                                                |
| Com_binlog                                    | 0                                                |
| Com_call_procedure                            | 0                                                |
| Com_change_db                                 | 284                                              |
| Com_change_master                             | 0                                                |
| Com_change_repl_filter                        | 0                                                |
| Com_check                                     | 0                                                |
| Com_checksum                                  | 0                                                |
| Com_commit                                    | 0                                                |
| Com_create_db                                 | 0                                                |
| Com_create_event                              | 0                                                |
| Com_create_function                           | 0                                                |
| Com_create_index                              | 0                                                |
| Com_create_procedure                          | 0                                                |
| Com_create_server                             | 0                                                |
| Com_create_table                              | 0                                                |
| Com_create_trigger                            | 0                                                |
| Com_create_udf                                | 0                                                |
| Com_create_user                               | 0                                                |
| Com_create_view                               | 0                                                |
| Com_dealloc_sql                               | 0                                                |
| Com_delete                                    | 18                                               |
| Com_delete_multi                              | 0                                                |
| Com_do                                        | 0                                                |
| Com_drop_db                                   | 0                                                |
| Com_drop_event                                | 0                                                |
| Com_drop_function                             | 0                                                |
| Com_drop_index                                | 0                                                |
| Com_drop_procedure                            | 0                                                |
| Com_drop_server                               | 0                                                |
| Com_drop_table                                | 0                                                |
| Com_drop_trigger                              | 0                                                |
| Com_drop_user                                 | 0                                                |
| Com_drop_view                                 | 0                                                |
| Com_empty_query                               | 0                                                |
| Com_execute_sql                               | 0                                                |
| Com_explain_other                             | 0                                                |
| Com_flush                                     | 0                                                |
| Com_get_diagnostics                           | 0                                                |
| Com_grant                                     | 0                                                |
| Com_ha_close                                  | 0                                                |
| Com_ha_open                                   | 0                                                |
| Com_ha_read                                   | 0                                                |
| Com_help                                      | 0                                                |
| Com_insert                                    | 342                                              |
| Com_insert_select                             | 0                                                |
| Com_install_plugin                            | 0                                                |
| Com_kill                                      | 0                                                |
| Com_load                                      | 0                                                |
| Com_lock_tables                               | 0                                                |
| Com_optimize                                  | 0                                                |
| Com_preload_keys                              | 0                                                |
| Com_prepare_sql                               | 0                                                |
| Com_purge                                     | 0                                                |
| Com_purge_before_date                         | 0                                                |
| Com_release_savepoint                         | 0                                                |
| Com_rename_table                              | 0                                                |
| Com_rename_user                               | 0                                                |
| Com_repair                                    | 0                                                |
| Com_replace                                   | 0                                                |
| Com_replace_select                            | 0                                                |
| Com_reset                                     | 0                                                |
| Com_resignal                                  | 0                                                |
| Com_revoke                                    | 0                                                |
| Com_revoke_all                                | 0                                                |
| Com_rollback                                  | 0                                                |
| Com_rollback_to_savepoint                     | 0                                                |
| Com_savepoint                                 | 0                                                |
| Com_select                                    | 16358                                            |
| Com_set_option                                | 849                                              |
| Com_signal                                    | 0                                                |
| Com_show_binlog_events                        | 0                                                |
| Com_show_binlogs                              | 0                                                |
| Com_show_charsets                             | 0                                                |
| Com_show_collations                           | 0                                                |
| Com_show_create_db                            | 0                                                |
| Com_show_create_event                         | 0                                                |
| Com_show_create_func                          | 0                                                |
| Com_show_create_proc                          | 0                                                |
| Com_show_create_table                         | 0                                                |
| Com_show_create_trigger                       | 0                                                |
| Com_show_databases                            | 3                                                |
| Com_show_engine_logs                          | 0                                                |
| Com_show_engine_mutex                         | 0                                                |
| Com_show_engine_status                        | 0                                                |
| Com_show_events                               | 0                                                |
| Com_show_errors                               | 0                                                |
| Com_show_fields                               | 921                                              |
| Com_show_function_code                        | 0                                                |
| Com_show_function_status                      | 0                                                |
| Com_show_grants                               | 0                                                |
| Com_show_keys                                 | 1                                                |
| Com_show_master_status                        | 0                                                |
| Com_show_open_tables                          | 0                                                |
| Com_show_plugins                              | 0                                                |
| Com_show_privileges                           | 0                                                |
| Com_show_procedure_code                       | 0                                                |
| Com_show_procedure_status                     | 0                                                |
| Com_show_processlist                          | 0                                                |
| Com_show_profile                              | 0                                                |
| Com_show_profiles                             | 0                                                |
| Com_show_relaylog_events                      | 0                                                |
| Com_show_slave_hosts                          | 0                                                |
| Com_show_slave_status                         | 0                                                |
| Com_show_status                               | 6                                                |
| Com_show_storage_engines                      | 0                                                |
| Com_show_table_status                         | 0                                                |
| Com_show_tables                               | 2                                                |
| Com_show_triggers                             | 0                                                |
| Com_show_variables                            | 6                                                |
| Com_show_warnings                             | 1                                                |
| Com_show_create_user                          | 0                                                |
| Com_shutdown                                  | 0                                                |
| Com_slave_start                               | 0                                                |
| Com_slave_stop                                | 0                                                |
| Com_group_replication_start                   | 0                                                |
| Com_group_replication_stop                    | 0                                                |
| Com_stmt_execute                              | 4                                                |
| Com_stmt_close                                | 4                                                |
| Com_stmt_fetch                                | 0                                                |
| Com_stmt_prepare                              | 4                                                |
| Com_stmt_reset                                | 0                                                |
| Com_stmt_send_long_data                       | 4                                                |
| Com_truncate                                  | 2                                                |
| Com_uninstall_plugin                          | 0                                                |
| Com_unlock_tables                             | 0                                                |
| Com_update                                    | 70                                               |
| Com_update_multi                              | 0                                                |
| Com_xa_commit                                 | 0                                                |
| Com_xa_end                                    | 0                                                |
| Com_xa_prepare                                | 0                                                |
| Com_xa_recover                                | 0                                                |
| Com_xa_rollback                               | 0                                                |
| Com_xa_start                                  | 0                                                |
| Com_stmt_reprepare                            | 0                                                |
| Connection_errors_accept                      | 0                                                |
| Connection_errors_internal                    | 0                                                |
| Connection_errors_max_connections             | 0                                                |
| Connection_errors_peer_address                | 0                                                |
| Connection_errors_select                      | 0                                                |
| Connection_errors_tcpwrap                     | 0                                                |
| Connections                                   | 292                                              |
| Created_tmp_disk_tables                       | 1124                                             |
| Created_tmp_files                             | 30                                               |
| Created_tmp_tables                            | 1369                                             |
| Delayed_errors                                | 0                                                |
| Delayed_insert_threads                        | 0                                                |
| Delayed_writes                                | 0                                                |
| Flush_commands                                | 1                                                |
| Handler_commit                                | 6094                                             |
| Handler_delete                                | 33                                               |
| Handler_discover                              | 0                                                |
| Handler_external_lock                         | 38571                                            |
| Handler_mrr_init                              | 0                                                |
| Handler_prepare                               | 0                                                |
| Handler_read_first                            | 2299                                             |
| Handler_read_key                              | 134761                                           |
| Handler_read_last                             | 237                                              |
| Handler_read_next                             | 310119                                           |
| Handler_read_prev                             | 2733                                             |
| Handler_read_rnd                              | 222350                                           |
| Handler_read_rnd_next                         | 472820                                           |
| Handler_rollback                              | 0                                                |
| Handler_savepoint                             | 0                                                |
| Handler_savepoint_rollback                    | 0                                                |
| Handler_update                                | 15605                                            |
| Handler_write                                 | 17310                                            |
| Innodb_buffer_pool_dump_status                | Dumping of buffer pool not started               |
| Innodb_buffer_pool_load_status                | Buffer pool(s) load completed at 180624 23:38:01 |
| Innodb_buffer_pool_resize_status              |                                                  |
| Innodb_buffer_pool_pages_data                 | 1035                                             |
| Innodb_buffer_pool_bytes_data                 | 16957440                                         |
| Innodb_buffer_pool_pages_dirty                | 0                                                |
| Innodb_buffer_pool_bytes_dirty                | 0                                                |
| Innodb_buffer_pool_pages_flushed              | 1936                                             |
| Innodb_buffer_pool_pages_free                 | 7144                                             |
| Innodb_buffer_pool_pages_misc                 | 13                                               |
| Innodb_buffer_pool_pages_total                | 8192                                             |
| Innodb_buffer_pool_read_ahead_rnd             | 0                                                |
| Innodb_buffer_pool_read_ahead                 | 0                                                |
| Innodb_buffer_pool_read_ahead_evicted         | 0                                                |
| Innodb_buffer_pool_read_requests              | 306665                                           |
| Innodb_buffer_pool_reads                      | 950                                              |
| Innodb_buffer_pool_wait_free                  | 0                                                |
| Innodb_buffer_pool_write_requests             | 26509                                            |
| Innodb_data_fsyncs                            | 1229                                             |
| Innodb_data_pending_fsyncs                    | 0                                                |
| Innodb_data_pending_reads                     | 0                                                |
| Innodb_data_pending_writes                    | 0                                                |
| Innodb_data_read                              | 16273920                                         |
| Innodb_data_reads                             | 1078                                             |
| Innodb_data_writes                            | 2857                                             |
| Innodb_data_written                           | 53379584                                         |
| Innodb_dblwr_pages_written                    | 1275                                             |
| Innodb_dblwr_writes                           | 109                                              |
| Innodb_log_waits                              | 0                                                |
| Innodb_log_write_requests                     | 450                                              |
| Innodb_log_writes                             | 585                                              |
| Innodb_os_log_fsyncs                          | 793                                              |
| Innodb_os_log_pending_fsyncs                  | 0                                                |
| Innodb_os_log_pending_writes                  | 0                                                |
| Innodb_os_log_written                         | 664064                                           |
| Innodb_page_size                              | 16384                                            |
| Innodb_pages_created                          | 56                                               |
| Innodb_pages_read                             | 988                                              |
| Innodb_pages_written                          | 1936                                             |
| Innodb_row_lock_current_waits                 | 0                                                |
| Innodb_row_lock_time                          | 0                                                |
| Innodb_row_lock_time_avg                      | 0                                                |
| Innodb_row_lock_time_max                      | 0                                                |
| Innodb_row_lock_waits                         | 0                                                |
| Innodb_rows_deleted                           | 2                                                |
| Innodb_rows_inserted                          | 19219                                            |
| Innodb_rows_read                              | 249102                                           |
| Innodb_rows_updated                           | 77                                               |
| Innodb_num_open_files                         | 81                                               |
| Innodb_truncated_status_writes                | 0                                                |
| Innodb_available_undo_logs                    | 128                                              |
| Key_blocks_not_flushed                        | 0                                                |
| Key_blocks_unused                             | 12751                                            |
| Key_blocks_used                               | 645                                              |
| Key_read_requests                             | 321877                                           |
| Key_reads                                     | 648                                              |
| Key_write_requests                            | 196                                              |
| Key_writes                                    | 150                                              |
| Locked_connects                               | 0                                                |
| Max_execution_time_exceeded                   | 0                                                |
| Max_execution_time_set                        | 0                                                |
| Max_execution_time_set_failed                 | 0                                                |
| Max_used_connections                          | 3                                                |
| Max_used_connections_time                     | 2018-06-24 23:43:48                              |
| Not_flushed_delayed_rows                      | 0                                                |
| Ongoing_anonymous_transaction_count           | 0                                                |
| Open_files                                    | 229                                              |
| Open_streams                                  | 0                                                |
| Open_table_definitions                        | 206                                              |
| Open_tables                                   | 786                                              |
| Opened_files                                  | 502                                              |
| Opened_table_definitions                      | 208                                              |
| Opened_tables                                 | 817                                              |
| Performance_schema_accounts_lost              | 0                                                |
| Performance_schema_cond_classes_lost          | 0                                                |
| Performance_schema_cond_instances_lost        | 0                                                |
| Performance_schema_digest_lost                | 0                                                |
| Performance_schema_file_classes_lost          | 0                                                |
| Performance_schema_file_handles_lost          | 0                                                |
| Performance_schema_file_instances_lost        | 0                                                |
| Performance_schema_hosts_lost                 | 0                                                |
| Performance_schema_index_stat_lost            | 0                                                |
| Performance_schema_locker_lost                | 0                                                |
| Performance_schema_memory_classes_lost        | 0                                                |
| Performance_schema_metadata_lock_lost         | 0                                                |
| Performance_schema_mutex_classes_lost         | 0                                                |
| Performance_schema_mutex_instances_lost       | 0                                                |
| Performance_schema_nested_statement_lost      | 0                                                |
| Performance_schema_prepared_statements_lost   | 0                                                |
| Performance_schema_program_lost               | 0                                                |
| Performance_schema_rwlock_classes_lost        | 0                                                |
| Performance_schema_rwlock_instances_lost      | 0                                                |
| Performance_schema_session_connect_attrs_lost | 0                                                |
| Performance_schema_socket_classes_lost        | 0                                                |
| Performance_schema_socket_instances_lost      | 0                                                |
| Performance_schema_stage_classes_lost         | 0                                                |
| Performance_schema_statement_classes_lost     | 0                                                |
| Performance_schema_table_handles_lost         | 0                                                |
| Performance_schema_table_instances_lost       | 0                                                |
| Performance_schema_table_lock_stat_lost       | 0                                                |
| Performance_schema_thread_classes_lost        | 0                                                |
| Performance_schema_thread_instances_lost      | 0                                                |
| Performance_schema_users_lost                 | 0                                                |
| Prepared_stmt_count                           | 0                                                |
| Qcache_free_blocks                            | 1                                                |
| Qcache_free_memory                            | 16760152                                         |
| Qcache_hits                                   | 0                                                |
| Qcache_inserts                                | 0                                                |
| Qcache_lowmem_prunes                          | 0                                                |
| Qcache_not_cached                             | 16355                                            |
| Qcache_queries_in_cache                       | 0                                                |
| Qcache_total_blocks                           | 1                                                |
| Queries                                       | 19164                                            |
| Questions                                     | 19155                                            |
| Select_full_join                              | 0                                                |
| Select_full_range_join                        | 0                                                |
| Select_range                                  | 2677                                             |
| Select_range_check                            | 0                                                |
| Select_scan                                   | 2098                                             |
| Slave_open_temp_tables                        | 0                                                |
| Slow_launch_threads                           | 0                                                |
| Slow_queries                                  | 0                                                |
| Sort_merge_passes                             | 12                                               |
| Sort_range                                    | 4859                                             |
| Sort_rows                                     | 244452                                           |
| Sort_scan                                     | 854                                              |
| Ssl_accept_renegotiates                       | 0                                                |
| Ssl_accepts                                   | 0                                                |
| Ssl_callback_cache_hits                       | 0                                                |
| Ssl_cipher                                    |                                                  |
| Ssl_cipher_list                               |                                                  |
| Ssl_client_connects                           | 0                                                |
| Ssl_connect_renegotiates                      | 0                                                |
| Ssl_ctx_verify_depth                          | 0                                                |
| Ssl_ctx_verify_mode                           | 0                                                |
| Ssl_default_timeout                           | 0                                                |
| Ssl_finished_accepts                          | 0                                                |
| Ssl_finished_connects                         | 0                                                |
| Ssl_server_not_after                          |                                                  |
| Ssl_server_not_before                         |                                                  |
| Ssl_session_cache_hits                        | 0                                                |
| Ssl_session_cache_misses                      | 0                                                |
| Ssl_session_cache_mode                        | NONE                                             |
| Ssl_session_cache_overflows                   | 0                                                |
| Ssl_session_cache_size                        | 0                                                |
| Ssl_session_cache_timeouts                    | 0                                                |
| Ssl_sessions_reused                           | 0                                                |
| Ssl_used_session_cache_entries                | 0                                                |
| Ssl_verify_depth                              | 0                                                |
| Ssl_verify_mode                               | 0                                                |
| Ssl_version                                   |                                                  |
| Table_locks_immediate                         | 11962                                            |
| Table_locks_waited                            | 0                                                |
| Table_open_cache_hits                         | 19395                                            |
| Table_open_cache_misses                       | 817                                              |
| Table_open_cache_overflows                    | 12                                               |
| Tc_log_max_pages_used                         | 0                                                |
| Tc_log_page_size                              | 0                                                |
| Tc_log_page_waits                             | 0                                                |
| Threads_cached                                | 2                                                |
| Threads_connected                             | 1                                                |
| Threads_created                               | 3                                                |
| Threads_running                               | 1                                                |
| Uptime                                        | 2944                                             |
| Uptime_since_flush_status                     | 2944                                             |
+-----------------------------------------------+--------------------------------------------------+
353 rows in set (0.00 sec)
Read more on SHOW GLOBAL STATUS here. Read more on the values here.
I can see NO major errors here (possibly due to UpClouds awesome disk IO) so I won’t be making memory tweaks in MySQL. Sign Up using this link and get $25 credit free on UpCloud and see for yourself how fast they are.
Configure Ubuntu System Memory Usage
Edit /etc/sysctl.conf
Add the following to allow things to sit in ram longer
Snip from: https://www.kernel.org/doc/Documentation/sysctl/vm.txt
This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim.  Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are."
Read these pages here and here regarding setting MySQL memory.
Reboot
Resize the disk
The 2x CPU, 4GB memory plan comes with 80GB storage allowance.  My disk at present is 50GB and I will update the size soon following this guide soon.
Upgrade disk from 50gb to 80GB soon
Quick Benchmark
I used loader.io to load 500 users to access my site in 1 minute.
HTOP showing 2x busy CPU's running at 60%
The benchmark worked with no errors.
Loader.io Success with 500 concurrent users
This benchmark was performed with no Cloudflare caching. I should get Cloudflare caching working again to lower the average response time. I loaded my website manually in Google Chrome while loader.io threw 500 users at my site and it loaded very fast.
Conclusion
After a few days, I checked my memory logs and there were no low memory triggers (just normal internal memory management triggers). Ubuntu was happier.
No Low memory low triggers
This graph was taken before I set “vm.vfs_cache_pressure” so I will update this graph in a few days.
I hope this guide helps someone.
Please consider using my referral code and get $25 credit for free.
https://www.upcloud.com/register/?promo=D84793
Ask a question or recommend an article
     
Your Name (required)
 
Your Email (required)
 
Your Question
 
Revision History
1.0 Initial Post
How to use the UpCloud API to manage your UpCloud servers
 by 
How to use the UpCloud API to manage your UpCloud servers.
Advertisement:

 
 
If you have not read my previous posts I have now moved my blog etc to the awesome UpCloud host. Sign up using this link to get $25 free credit.
I recently compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here).
Here is my blog post on moving from Vultr to UpCloud.
Spoiler: UpCloud performance is great.
Upcloud Site Speed in GTMetrix
I have never had an UpCloud page load take longer than 2 seconds since moving.
UpCloud API
UpCloud has an API that we can opt into to using where we can manage servers. Read the official UpCloud API documentation here.
The API allows you to control:
  • Accounts
  • Pricing
  • Zones
  • Timezones
  • Plans
  • Servers
  • Storages
  • IP-Addresses
  • Firewall
  • Tags
  • etc
Create a sub-account to query the API 
You should create a new user account (in the UpCloud dasbboard) just for API access. I created two accounts for use on my server and on my home laptop and my server (and set a limiting IP(s) that can access it).
Create a Sub Account for API Access
Login to your UpCloud account (create an account here and get $25 free credit),
  1. Click My Accounts,
  2. Click User Accounts,
  3. Click Change on your user and enable API connections.
  4. TIP: Set up an IP rule to limit access to your API for security (I set up a VPN to get a static IP on my dynamic IP Internet host at home)).
  5. Save the changes
Enable API Connections
TIP: Lock down the account to have the minimum permissions required.
e.g
  • Disable access to the control panel (Untick).
  • Allow API Connections (Tick) and specify an IP
  • Disable access to billing contact (Untick).
  • Disable access to billing section in the control panel (Untick).
  • Disable allowing of emails to billing contact (Untick).
  • Allow or Remove access to all server (or manually add access to desired servers)
  • Allow or Remove access to modify storage (or manually allow or remove access to desired storage)
  • etc
Lock down the account to the minimum needed
Save the account.
Now let’s make our first API call
I use OSX and I use the awesome Paw API testing tool from https://paw.cloud (This is not a plug, they are awesome). Postman is a popular API testing tool too. Any good programing language or CLI will allow you to send API requests.
First, let’s prepare the authorization string (this is a Base64 encoded combination of your username and password) read more here.
  1. Head over to https://www.base64encode.org/
  2. Click the Encode tab
  3. Add your “username:password” (without the quotes).
  4. Click Encode
A Base64 string will be outputted 🙂
e.g > eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk
fyi
You can encode also Encode and Decode Base64 from the Ubuntu Command line
Encode Base64 from the CLI Sample
Decode Base64 from the CLI Sample
Now we can add an “Authorization Basic” token to the API request in Paw.
Authorization Header added with my base64 token.
A quick test of the UpCloud Prices API endpoint https://api.upcloud.com/1.2/price reveals the API is working.
Add Authorization Token
I can now see a full breakdown of my service prices in JSON 🙂
Query My Account
OK, Let’s see how much credit I have left by querying the https://api.upcloud.com/1.2/account, I duplicated the item in Paw and changed the URL to https://api.upcloud.com/1.2/account but no data returned?
I had to enable “Access to Billing section in Control Panel” for the user before this data returned from the API (make sense).
> HTTP/1.1 200 OK
Query (GET)
GET /1.2/account HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic *******************************************
Output
HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:23:32 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 91
Server: Apache

{
   "account" : {
      "credits" : 2500.00,
      "username" : "yourapiusername"
   }
}
“2500.00” = cents ($25)
Query All of Your Servers
Ok, Let’s get server information by querying https://api.upcloud.com/1.2/server
Query (GET)
GET /1.2/server HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic ##############base64hash##############
Output
HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:32:22 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 1154
Server: Apache

{
   "servers" : {
      "server" : [
         {
            "core_number" : "1",
            "hostname" : "server1nameredacted.com",
            "license" : 0,
            "memory_amount" : "2048",
            "plan" : "1xCPU-2GB",
            "plan_ipv4_bytes" : "3472464313",
            "plan_ipv6_bytes" : "166293599",
            "state" : "started",
            "tags" : {
               "tag" : [
                  "tag1"
               ]
            },
            "title" : "server1nameredacted.com",
            "uuid" : "########-####-####-####-############",
            "zone" : "us-chi1"
         },
         {
            "core_number" : "1",
            "hostname" : "server2nameredacted.com",
            "license" : 0,
            "memory_amount" : "1024",
            "plan" : "1xCPU-1GB",
            "plan_ipv4_bytes" : "198911",
            "plan_ipv6_bytes" : "19742",
            "state" : "started",
            "tags" : {
               "tag" : [
                  "tag2"
               ]
            },
            "title" : "server1nameredacted.com",
            "uuid" : "########-####-####-####-############",
            "zone" : "us-chi1"
         }
      ]
   }
}
Query Server Information
I have redated the UUID’s for my servers but once you know them you can query them by hitting https://api.upcloud.com/1.2/server/########-####-####-####-############
Query (GET)
Output
HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:45:14 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 1656
Server: Apache

{
   "server" : {
      "boot_order" : "cdrom,disk",
      "core_number" : "1",
      "firewall" : "on",
      "host" : redacted,
      "hostname" : "server1nameredacted.com",
      "ip_addresses" : {
         "ip_address" : [
            {
               "access" : "private",
               "address" : "##.#.#.###",
               "family" : "IPv4"
            },
            {
               "access" : "public",
               "address" : "###.###.###.###",
               "family" : "IPv4",
               "part_of_plan" : "yes"
            },
            {
               "access" : "public",
               "address" : "####:####:####:####:####:####:########",
               "family" : "IPv6"
            }
         ]
      },
      "license" : 0,
      "memory_amount" : "2048",
      "nic_model" : "virtio",
      "plan" : "1xCPU-2GB",
      "plan_ipv4_bytes" : "3519033266",
      "plan_ipv6_bytes" : "168200052",
      "state" : "started",
      "storage_devices" : {
         "storage_device" : [
            {
               "address" : "virtio:0",
               "boot_disk" : "0",
               "part_of_plan" : "yes",
               "storage" : "########-####-####-####-############",
               "storage_size" : 50,
               "storage_title" : "system",
               "type" : "disk"
            }
         ]
      },
      "tags" : {
         "tag" : [
            "fearby"
         ]
      },
      "timezone" : "Australia/Sydney",
      "title" : "server1nameredacted.com",
      "uuid" : "########-####-####-####-############",
      "video_model" : "cirrus",
      "vnc" : "off",
      "vnc_password" : "#########################",
      "zone" : "us-chi1"
   }
}
The servers name, IPv4 and IPV6 network adapters are listed, CPU(s), Memory, Disk Sized and UUID’s are all visible 🙂
Surprisingly the VNC password is visible (enabling access to the root console).
TIP: Ensure your API account is safe and secure.
Query Storage Information
Now, Let’s query the storage with the GUID from above by querying https://api.upcloud.com/1.2/storage/########-####-####-####-############
Query (GET)
GET /1.2/storage/########-####-####-####-############ HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic  ##############base64hash##############
Output
HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:53:36 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 559
Server: Apache

{
   "storage" : {
      "access" : "private",
      "backup_rule" : {},
      "backups" : {
         "backup" : [
            "########-####-####-####-############"
         ]
      },
      "license" : 0,
      "part_of_plan" : "yes",
      "servers" : {
         "server" : [
            "########-####-####-####-############"
         ]
      },
      "size" : 50,
      "state" : "online",
      "tier" : "maxiops",
      "title" : "system",
      "type" : "normal",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}
I can see information about the storage’s assigned server and backups 🙂
Query Backup Information
Backup storage can be queried with the same storge API endpoint https://api.upcloud.com/1.2/storage/########-####-####-####-############
Query (GET)
Output
HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 05:01:11 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 412
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-16T04:47:56Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "online",
      "title" : "On-Demand Backup",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}
Rename Backup
One thing that I would like to be able to do is to rename on-demand backups in the UpCloud dashboard (this is not a feature yet) but I can rename manual backup’s in the API though 🙂
Boring “On-Demand Backup” label.
Rename Backups Not possible in the GUI
I tried sending JSON to https://api.upcloud.com/1.2/storage/########-####-####-####-############ to rename a backup but kept getting an error?
JSON
{
 > “storage”: {
 > “title”: “Latest manual backup , Working NGINX, PHP, MySQL w Tweaks”,
 > “size”: “50”
 > }
 > }
Result
> “error_code” : “CONTENT_TYPE_INVALID”,
 > “error_message” : “The Content-Type header has an invalid value.”
I googled and found an old manual for UpClouds API (official support here).
I added these missing content type headers (108 was the length in chars of the payload)
> Content-Type: application/json; Charset=UTF-8'
> Content-Length: 108
Still no go?
I think the content length value is wrong, more here.
I fixed it, it turned out I had a semicolon in the Content-Type value. The JSON RFC always assumes that Content-Type is UTF8 encoded (more here).
This Fails
This Works
Content-Type: application/json
Now I can rename my Backup (storage). I manually calculated the length of the JSON payload and added a “Content-Length” header and value.
Query (PUT)
Output
HTTP/1.1 202 ACCEPTED
Date: Sun, 17 Jun 2018 05:47:02 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 453
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-16T04:47:56Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "online",
      "title" : "Latest manual backup , Working NGINX, PHP, MySQL w Tweaks",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}
Success 🙂
Backup Renamed
Create a Backup
Backups can be performed with a “/backup” added to the end of the query string.
Query (POST)
Output
Success (UpCloud GUI)
Conclusion
UpCloud does have great API docs.
I can easily integrate this into bash scripts to manage my servers via API and a future Java app for managing servers.
Paw does give CURL output to allow me to copy working API’s for use in BASH 🙂
More to come
  1. BASH Script to Deploy and configure a server on UpCloud via Initalization scripts (or manual) (1 week)
  2. JAVA App to manage your server (3 months)
If you are signing up for UpCloud please consider using my referral code and get $25 credit for free.
Read my setup guide here.
https://www.upcloud.com/register/?promo=D84793
I hope this guide helps someone.
Ask a question or recommend an article
     
Your Name (required)
 
Your Email (required)
 
Your Question
 
Revision History
V1.1 updated typo
v1.0 Initial Post.
Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4
 by 
How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4
Advertisement:

 
 
Read Part 1, Part 2, Part 3 or Part 4
I ran the MySQL benchmark preparation command again (no problem this time).
sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=###################### prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...
Test table and records created
Test Records Created
Now I can benchmark MySQL on my main server.
sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=################################# --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
RAW Output
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            336210
        write:                           0
        other:                           48030
        total:                           384240
    transactions:                        24015  (400.09 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 336210 (5601.24 per sec.)
    other operations:                    48030  (800.18 per sec.)

Test execution summary:
    total time:                          60.0242s
    total number of events:              24015
    total time taken by event execution: 480.0242
    per-request statistics:
         min:                                  1.79ms
         avg:                                 19.99ms
         max:                                141.00ms
         approx.  95 percentile:              37.49ms

Threads fairness:
    events (avg/stddev):           3001.8750/27.36
    execution time (avg/stddev):   60.0030/0.01
Results
queries performed (in 60 seconds):
  • read: 336210
  • other: 48030
  • total: 384240
I decided to add an index to see if I can speed this query up (read the MySQL index page here). I added an index (in Adminer) on the columns “Id” and “pad” for the sbtest table in the test database
I restarted the MySQL process
mysql restart
[ ok ] Restarting mysql (via systemctl): mysql.service.
I ran the same benchmark again.
Raw Output
sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=########################## --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            426538
        write:                           0
        other:                           60934
        total:                           487472
    transactions:                        30467  (507.69 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 426538 (7107.67 per sec.)
    other operations:                    60934  (1015.38 per sec.)

Test execution summary:
    total time:                          60.0110s
    total number of events:              30467
    total time taken by event execution: 479.9124
    per-request statistics:
         min:                                  5.75ms
         avg:                                 15.75ms
         max:                                138.57ms
         approx.  95 percentile:              25.10ms

Threads fairness:
    events (avg/stddev):           3808.3750/8.70
    execution time (avg/stddev):   59.9891/0.00
Results
The quick index added 20% extra throughput on queries 🙂
Mysql before and after an index
Don’t forget to delete your test database
DROP DATABASE `test`;
Viewing MySQL Index Usage (on the “test” database)
Query to show Index stats for a table ‘test’
SELECT
 OBJECT_SCHEMA as 'Database', OBJECT_NAME as 'Table', 
 INDEX_NAME as 'Index', 
 COUNT_STAR, 
 SUM_TIMER_WAIT,  MIN_TIMER_WAIT, AVG_TIMER_WAIT, MAX_TIMER_WAIT, 
 COUNT_READ, 
 SUM_TIMER_READ, MIN_TIMER_READ, AVG_TIMER_READ, MAX_TIMER_READ,  
 COUNT_FETCH, SUM_TIMER_FETCH, MIN_TIMER_FETCH, AVG_TIMER_FETCH, MAX_TIMER_FETCH
FROM 
 performance_schema.table_io_waits_summary_by_index_usage
WHERE 
 object_schema = 'test'
I can see the MySQL PRIMARY index is getting used 🙂
Index Summary
Read more in viewable query stats (columns) here.
Other System Information Tools
Show processor information
cat /proc/cpuinfo
Output
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 61
model name      : Virtual CPU a7769a6388d5
stepping        : 2
microcode       : 0x1
cpu MHz         : 2394.454
cache size      : 16384 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4788.90
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
Memory Information
You can assign 512MB, 1GB, 2GB or more memory to a server on Vultr, Read my guide here on upgrading resources for Vultr VM’s here.
Only upgrade your server’s memory when server processes demand it, there is no need to pay for extra idle memory. Read my older guides on upgrading Digital Ocean and AWS servers.
I use the htop utility to monitor memory and processes. The memory usage will depend on how you have configured your server to use connection pools in code, MySQL or services.  Also what memory demands do you get in pean bandwidth times?
HTOP
You can check your server memory details on Ubuntu with this command
cat /proc/meminfo
Output
MemTotal:        2048104 kB
MemFree:           96176 kB
MemAvailable:     693072 kB
Buffers:          183476 kB
Cached:           526124 kB
SwapCached:            0 kB
Active:          1467220 kB
Inactive:         243228 kB
Active(anon):    1070464 kB
Inactive(anon):    27004 kB
Active(file):     396756 kB
Inactive(file):   216224 kB
Unevictable:        3652 kB
Mlocked:            3652 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                64 kB
Writeback:             0 kB
AnonPages:       1004504 kB
Mapped:           114664 kB
Shmem:             94192 kB
Slab:             192692 kB
SReclaimable:     171892 kB
SUnreclaim:        20800 kB
KernelStack:        3072 kB
PageTables:        20528 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1024052 kB
Committed_AS:    2424332 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    247808 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       67440 kB
DirectMap2M:     2029568 kB
Use Memory or Disk (Swap)
You can configure the use of Memory over Disk by configuring your/etc/sysctl.conf file (setting value “vm.swappiness”)
You can check your swap file settings by running the following command
cat /proc/sys/vm/swappiness
1
Or By running
sysctl vm.swappiness
vm.swappiness = 1
Set a new swap file value by editing /etc/sysctl.conf
sudo nano /etc/sysctl.conf
Set the following to use more ram over the swap disk.
vm.swappiness = 1
Read about swappiness values here: https://en.wikipedia.org/wiki/Swappiness
Service Performance
Performance (and allocated resources) depends on the demands of your operating system and installed software
What operating system do you have?
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial
View NGINX Status, how much memory does it use?
/etc/init.d/nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:25 AEST; 1 weeks 3 days ago
     Docs: man:nginx(8)
 Main PID: #### (nginx)
    Tasks: 3
   Memory: 58.9M
      CPU: 33min 11.515s
   CGroup: /system.slice/nginx.service
           ├─#### nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─#### nginx: worker process
           └─#### nginx: cache manager process
PHP (and Child Worker) status how much memory does it use and how many child workers do you have? Read my add PHP child workers post here (and update to PHP 7.2 here)
sudo service php7.2-fpm status
● php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:26 AEST; 1 weeks 3 days ago
     Docs: man:php-fpm7.2(8)
 Main PID: #### (php-fpm7.2)
   Status: "Processes active: 0, idle: 20, Requests: 75911, slow: 0, Traffic: 0.1req/sec"
    Tasks: 21
   Memory: 694.2M
      CPU: 20h 49min 45.132s
   CGroup: /system.slice/php7.2-fpm.service
           ├─ #### php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
           ├─ #### php-fpm: pool www-acc
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-acc
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           └─ #### php-fpm: pool www-usr
MySQL Status
sudo service mysql status
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:27 AEST; 1 weeks 3 days ago
 Main PID: ##### (mysqld)
    Tasks: 35
   Memory: 405.9M
      CPU: 2h 17min 31.822s
   CGroup: /system.slice/mysql.service
           └─#### /usr/sbin/mysqld
Shared VM Hosts
One of the biggest impacts (after server latency) for your server is not the disk performance but the number of hosts/websites on the server who are also using the disk and server resources.
Reverse IP Lookup
I have 80 other web servers on my server (based on a reverse lookup).
I may move to a dedicated box when I can afford it.
Security
Above all else ensure that security is number 1 priority and make performance second priority.
Scan your site with Zap, Qualys and Kali Linux. Performance means nothing if you are hacked.
website-report
Simulated concurrent users
You can use Siege to test the maximum concurrent users accessing your site before the server starts to drop connections.
FYI: If you use Cloudflare (you should) this may not work as it will block connections.
Install Siege
sudo apt-get install siege
Test  your server with 10 concurrent serves for 1 minute
siege -t1m c10 'https://yourserver.com/'
Results
siege -t1m c10 'https://yourserver.com/'
** SIEGE 3.0.8
** Preparing 15 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:                    417 hits
Availability:                 100.00 %
Elapsed time:                  59.01 secs
Data transferred:               8.24 MB
Response time:                  1.62 secs
Transaction rate:               7.07 trans/sec
Throughput:                     0.14 MB/sec
Concurrency:                   11.46
Successful transactions:         417
Failed transactions:               0
Longest transaction:            2.26
Shortest transaction:           1.49
Keep upping the connections (from 10 above) to a limit where connections start dropping.
I tried 25 then 50 concurrent users hitting a server on Digital Ocean and it did not fail.
Conclusion
  • Choose a server near your customers
  • Change hosts if one is faster and cheaper
  • Measure or benchmark your server (and compare over time).
  • Use Cloudflare
Create your own server today
  • Create your own server on Vultr here.
  • Create your own server on Digital Ocean here.
  • Create your own server on UpCloud here.
And remember you can install the Runcloud server management dashboard here.
I hope this guide helps someone.
 
< Previous
Read Part 1, Part 2, Part 3 or Part 4