Linux

Updating NGINX to the development branch to get more frequent updates and features over the stable branch

November 20, 2018 by Simon

Updating NGINX to the development branch (on Ubuntu) to get more frequent updates and features over the stable branch

Aside

I have a number of guides on moving away from CPanel, Setting up VM’s on UpCloud, AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. View all recent posts here https://fearby.com/all/

Now on with the post

Warning

Backup your Nginx and Server before making any changes. The Nginx development branch is quite stable but anything can happen. If your site is mission critical then stay on the stable branch.

Nginx Branches

By default, you will most likely get the stable branch of Nginx when instaling and updating Nginx. I have been running the stable version for the last few years but was made aware of a DDoS vulnerability in Nginx.

Here is a good write-up on development merges into the stable branch.

Nginx Updates

Widely-used #Nginx server releases versions 1.15.6 and 1.14.1 to patch two HTTP/2 implementation vulnerabilities that might cause excessive memory consumption (CVE-2018-16843) & CPU usage (CVE-2018-16844), allowing a remote attacker to perform #DoS attackhttps://t.co/1Z3JoghoBr pic.twitter.com/qQ3pOFD1Lk
— The Hacker News (@TheHackersNews) November 9, 2018

I was aware recently of a DDoS bug affecting Nginx and the recommendation was to update ot Nginx 1.15.6 development branch (or 1.14.1 stable branch).

A few days ago no 1.14.1 update was available but a 1.15.6 was, should I switch to the development branch to get updates earlier?

Reminder to update your #nginx installations to the 1.14.1 stable or the 1.15.6 mainline versions for critical security patches released this week. #NGINXPlus customers, see instructions for updating based on the patch released 10/30 https://t.co/KitsOWIJkb
— NGINX, Inc. (@nginx) November 8, 2018

Recent Nginx Changes

Here are the recent changes to Nginx: http://nginx.org/en/CHANGES

Changes with nginx 1.15.6                                        06 Nov 2018

    *) Security: when using HTTP/2 a client might cause excessive memory
       consumption (CVE-2018-16843) and CPU usage (CVE-2018-16844).

    *) Security: processing of a specially crafted mp4 file with the
       ngx_http_mp4_module might result in worker process memory disclosure
       (CVE-2018-16845).

    *) Feature: the "proxy_socket_keepalive", "fastcgi_socket_keepalive",
       "grpc_socket_keepalive", "memcached_socket_keepalive",
       "scgi_socket_keepalive", and "uwsgi_socket_keepalive" directives.

    *) Bugfix: if nginx was built with OpenSSL 1.1.0 and used with OpenSSL
       1.1.1, the TLS 1.3 protocol was always enabled.

    *) Bugfix: working with gRPC backends might result in excessive memory
       consumption.


Changes with nginx 1.15.5                                        02 Oct 2018

    *) Bugfix: a segmentation fault might occur in a worker process when
       using OpenSSL 1.1.0h or newer; the bug had appeared in 1.15.4.

    *) Bugfix: of minor potential bugs.


Changes with nginx 1.15.4                                        25 Sep 2018

    *) Feature: now the "ssl_early_data" directive can be used with OpenSSL.

    *) Bugfix: in the ngx_http_uwsgi_module.
       Thanks to Chris Caputo.

    *) Bugfix: connections with some gRPC backends might not be cached when
       using the "keepalive" directive.

    *) Bugfix: a socket leak might occur when using the "error_page"
       directive to redirect early request processing errors, notably errors
       with code 400.

    *) Bugfix: the "return" directive did not change the response code when
       returning errors if the request was redirected by the "error_page"
       directive.

    *) Bugfix: standard error pages and responses of the
       ngx_http_autoindex_module module used the "bgcolor" attribute, and
       might be displayed incorrectly when using custom color settings in
       browsers.
       Thanks to Nova DasSarma.

    *) Change: the logging level of the "no suitable key share" and "no
       suitable signature algorithm" SSL errors has been lowered from "crit"
       to "info".


Changes with nginx 1.15.3                                        28 Aug 2018

    *) Feature: now TLSv1.3 can be used with BoringSSL.

    *) Feature: the "ssl_early_data" directive, currently available with
       BoringSSL.

    *) Feature: the "keepalive_timeout" and "keepalive_requests" directives
       in the "upstream" block.

    *) Bugfix: the ngx_http_dav_module did not truncate destination file
       when copying a file over an existing one with the COPY method.

    *) Bugfix: the ngx_http_dav_module used zero access rights on the
       destination file and did not preserve file modification time when
       moving a file between different file systems with the MOVE method.

    *) Bugfix: the ngx_http_dav_module used default access rights when
       copying a file with the COPY method.

    *) Workaround: some clients might not work when using HTTP/2; the bug
       had appeared in 1.13.5.

    *) Bugfix: nginx could not be built with LibreSSL 2.8.0.


Changes with nginx 1.15.2                                        24 Jul 2018

    *) Feature: the $ssl_preread_protocol variable in the
       ngx_stream_ssl_preread_module.

    *) Feature: now when using the "reset_timedout_connection" directive
       nginx will reset connections being closed with the 444 code.

    *) Change: a logging level of the "http request", "https proxy request",
       "unsupported protocol", and "version too low" SSL errors has been
       lowered from "crit" to "info".

    *) Bugfix: DNS requests were not resent if initial sending of a request
       failed.

    *) Bugfix: the "reuseport" parameter of the "listen" directive was
       ignored if the number of worker processes was specified after the
       "listen" directive.

    *) Bugfix: when using OpenSSL 1.1.0 or newer it was not possible to
       switch off "ssl_prefer_server_ciphers" in a virtual server if it was
       switched on in the default server.

    *) Bugfix: SSL session reuse with upstream servers did not work with the
       TLS 1.3 protocol.


Changes with nginx 1.15.1                                        03 Jul 2018

    *) Feature: the "random" directive inside the "upstream" block.

    *) Feature: improved performance when using the "hash" and "ip_hash"
       directives with the "zone" directive.

    *) Feature: the "reuseport" parameter of the "listen" directive now uses
       SO_REUSEPORT_LB on FreeBSD 12.

    *) Bugfix: HTTP/2 server push did not work if SSL was terminated by a
       proxy server in front of nginx.

    *) Bugfix: the "tcp_nopush" directive was always used on backend
       connections.

    *) Bugfix: sending a disk-buffered request body to a gRPC backend might
       fail.


Changes with nginx 1.15.0                                        05 Jun 2018

    *) Change: the "ssl" directive is deprecated; the "ssl" parameter of the
       "listen" directive should be used instead.

    *) Change: now nginx detects missing SSL certificates during
       configuration testing when using the "ssl" parameter of the "listen"
       directive.

    *) Feature: now the stream module can handle multiple incoming UDP
       datagrams from a client within a single session.

    *) Bugfix: it was possible to specify an incorrect response code in the
       "proxy_cache_valid" directive.

    *) Bugfix: nginx could not be built by gcc 8.1.

    *) Bugfix: logging to syslog stopped on local IP address changes.

    *) Bugfix: nginx could not be built by clang with CUDA SDK installed;
       the bug had appeared in 1.13.8.

    *) Bugfix: "getsockopt(TCP_FASTOPEN) ... failed" messages might appear
       in logs during binary upgrade when using unix domain listen sockets
       on FreeBSD.

    *) Bugfix: nginx could not be built on Fedora 28 Linux.

    *) Bugfix: request processing rate might exceed configured rate when
       using the "limit_req" directive.

    *) Bugfix: in handling of client addresses when using unix domain listen
       sockets to work with datagrams on Linux.

    *) Bugfix: in memory allocation error handling.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

Changes with nginx 1.15.6 06 Nov 2018

*) Security: when using HTTP/2 a client might cause excessive memory

consumption (CVE-2018-16843) and CPU usage (CVE-2018-16844).

*) Security: processing of a specially crafted mp4 file with the

ngx_http_mp4_module might result in worker process memory disclosure

(CVE-2018-16845).

*) Feature: the "proxy_socket_keepalive", "fastcgi_socket_keepalive",

"grpc_socket_keepalive", "memcached_socket_keepalive",

"scgi_socket_keepalive", and "uwsgi_socket_keepalive" directives.

*) Bugfix: if nginx was built with OpenSSL 1.1.0 and used with OpenSSL

1.1.1, the TLS 1.3 protocol was always enabled.

*) Bugfix: working with gRPC backends might result in excessive memory

consumption.

Changes with nginx 1.15.5 02 Oct 2018

*) Bugfix: a segmentation fault might occur in a worker process when

using OpenSSL 1.1.0h or newer; the bug had appeared in 1.15.4.

*) Bugfix: of minor potential bugs.

Changes with nginx 1.15.4 25 Sep 2018

*) Feature: now the "ssl_early_data" directive can be used with OpenSSL.

*) Bugfix: in the ngx_http_uwsgi_module.

Thanks to Chris Caputo.

*) Bugfix: connections with some gRPC backends might not be cached when

using the "keepalive" directive.

*) Bugfix: a socket leak might occur when using the "error_page"

directive to redirect early request processing errors, notably errors

with code 400.

*) Bugfix: the "return" directive did not change the response code when

returning errors if the request was redirected by the "error_page"

directive.

*) Bugfix: standard error pages and responses of the

ngx_http_autoindex_module module used the "bgcolor" attribute, and

might be displayed incorrectly when using custom color settings in

browsers.

Thanks to Nova DasSarma.

*) Change: the logging level of the "no suitable key share" and "no

suitable signature algorithm" SSL errors has been lowered from "crit"

to "info".

Changes with nginx 1.15.3 28 Aug 2018

*) Feature: now TLSv1.3 can be used with BoringSSL.

*) Feature: the "ssl_early_data" directive, currently available with

BoringSSL.

*) Feature: the "keepalive_timeout" and "keepalive_requests" directives

in the "upstream" block.

*) Bugfix: the ngx_http_dav_module did not truncate destination file

when copying a file over an existing one with the COPY method.

*) Bugfix: the ngx_http_dav_module used zero access rights on the

destination file and did not preserve file modification time when

moving a file between different file systems with the MOVE method.

*) Bugfix: the ngx_http_dav_module used default access rights when

copying a file with the COPY method.

*) Workaround: some clients might not work when using HTTP/2; the bug

had appeared in 1.13.5.

*) Bugfix: nginx could not be built with LibreSSL 2.8.0.

Changes with nginx 1.15.2 24 Jul 2018

*) Feature: the $ssl_preread_protocol variable in the

ngx_stream_ssl_preread_module.

*) Feature: now when using the "reset_timedout_connection" directive

nginx will reset connections being closed with the 444 code.

*) Change: a logging level of the "http request", "https proxy request",

"unsupported protocol", and "version too low" SSL errors has been

lowered from "crit" to "info".

*) Bugfix: DNS requests were not resent if initial sending of a request

failed.

*) Bugfix: the "reuseport" parameter of the "listen" directive was

ignored if the number of worker processes was specified after the

"listen" directive.

*) Bugfix: when using OpenSSL 1.1.0 or newer it was not possible to

switch off "ssl_prefer_server_ciphers" in a virtual server if it was

switched on in the default server.

*) Bugfix: SSL session reuse with upstream servers did not work with the

TLS 1.3 protocol.

Changes with nginx 1.15.1 03 Jul 2018

*) Feature: the "random" directive inside the "upstream" block.

*) Feature: improved performance when using the "hash" and "ip_hash"

directives with the "zone" directive.

*) Feature: the "reuseport" parameter of the "listen" directive now uses

SO_REUSEPORT_LB on FreeBSD 12.

*) Bugfix: HTTP/2 server push did not work if SSL was terminated by a

proxy server in front of nginx.

*) Bugfix: the "tcp_nopush" directive was always used on backend

connections.

*) Bugfix: sending a disk-buffered request body to a gRPC backend might

fail.

Changes with nginx 1.15.0 05 Jun 2018

*) Change: the "ssl" directive is deprecated; the "ssl" parameter of the

"listen" directive should be used instead.

*) Change: now nginx detects missing SSL certificates during

configuration testing when using the "ssl" parameter of the "listen"

directive.

*) Feature: now the stream module can handle multiple incoming UDP

datagrams from a client within a single session.

*) Bugfix: it was possible to specify an incorrect response code in the

"proxy_cache_valid" directive.

*) Bugfix: nginx could not be built by gcc 8.1.

*) Bugfix: logging to syslog stopped on local IP address changes.

*) Bugfix: nginx could not be built by clang with CUDA SDK installed;

the bug had appeared in 1.13.8.

*) Bugfix: "getsockopt(TCP_FASTOPEN) ... failed" messages might appear

in logs during binary upgrade when using unix domain listen sockets

on FreeBSD.

*) Bugfix: nginx could not be built on Fedora 28 Linux.

*) Bugfix: request processing rate might exceed configured rate when

using the "limit_req" directive.

*) Bugfix: in handling of client addresses when using unix domain listen

sockets to work with datagrams on Linux.

*) Bugfix: in memory allocation error handling.

Development branch changes are made every few weeks and stable branch changes are made less often.

Updating Nginx

Normally you update Nginx bu running an update and upgrade

apt-get update && apt-get upgrade

1	apt-get update && apt-get upgrade

Restart Nginx for good measure

/etc/init.d/nginx restart

1	/etc/init.d/nginx restart

Checking NGINX Version

nginx -v
nginx version: nginx/1.14.1

1 2	nginx -v nginx version: nginx/1.14.1

Changing your repository to the development branch

I changed ot the development branch by running

sudo add-apt-repository ppa:nginx/development

1	sudo add-apt-repository ppa:nginx/development

Update and upgrade Nginx

apt-get update && apt-get upgrade

1	apt-get update && apt-get upgrade

Restart Nginx for good measure

/etc/init.d/nginx restart

1	/etc/init.d/nginx restart

Checking NGINX Version

nginx -v
nginx version: nginx/1.16.6

1 2	nginx -v nginx version: nginx/1.16.6

Removing the stable Nginx repository

Run this command to remove the stable branch of Nginx

sudo add-apt-repository -r ppa:nginx/stable

1	sudo add-apt-repository -r ppa:nginx/stable

Check to see if the development branch is listed

grep -r --include '*.list' '^deb ' /etc/apt/sources.list* |grep nginx
/etc/apt/sources.list.d/nginx-ubuntu-development-bionic.list:deb http://ppa.launchpad.net/nginx/development/ubuntu bionic main

1 2	grep -r --include '.list' '^deb ' /etc/apt/sources.list \|grep nginx /etc/apt/sources.list.d/nginx-ubuntu-development-bionic.list:deb http://ppa.launchpad.net/nginx/development/ubuntu bionic main

Good luck and I hope this guide helps someone

Ask a question or recommend an article

Revision History

v1.0 Initial post

Adding two sub domains (one pointing to a new UpCloud VM and the other pointing to an NGINX subsite) on Ubuntu 18.04

June 27, 2018 by Simon

Here is how I added two subdomains (one pointing to a new UpCloud VM and the other pointing to an NGINX subsite) on Ubuntu 18.04

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

UpCloud performance is great.

Buy a domain name from Namecheap here.

Goal(s)

Setup 2x subdomains on https://fearby.com

– Sub Domain #1: https://test.fearby.com (pointing to a dedicated UpCloud VM in Singapore for testing).

– Sub Domain #2: https://audit.fearby.com (pointing to a sub-website on the NGINX/VM that runs https://fearby.com )

Let’s set up the first Sub Domain (dedicated VM) and SSL

Backup

Do backup your server first.

I created a second server ($5 month or $0.07c hour 1,024MB Memory, 25GB Disk, 1024 GB Month Data Transfer) at UpCloud. If you don’t already have an account at UpCloud use this link to signup and get $25 free credit ( https://www.upcloud.com/register/?promo=D84793 ). Read my blog post on why UpCloud is awesome and how I moved my domain to UpCloud.

Once I spun up a second server I obtained the IPv4 and IPv6 IP addresses of the new “test” VM from the UpCloud dashboard.

IPV4 IP: 94.237.65.54
IPV6 IP: 2a04:3543:1000:2310:24b7:7cff:fe92:468c

1 2	IPV4 IP: 94.237.65.54 IPV6 IP: 2a04:3543:1000:2310:24b7:7cff:fe92:468c

DNS

These DNS records were already in place with my DNS provider (Cloudflare).

A fearby.com 209.50.48.88
AAAA fearby.com 2605:7380:1000:1310:24b7:7cff:fe92:0d64

1 2	A fearby.com 209.50.48.88 AAAA fearby.com 2605:7380:1000:1310:24b7:7cff:fe92:0d64

I added these DNS records for the subdomains.

I added a new A NAME record for the new shared NGINX subdomain (for https://audit.fearby.com), this subdomain will be a sub-website that is running off the same server as https://fearby.com

A audit 209.50.48.88
AAAA audit 2605:7380:1000:1310:24b7:7cff:fe92:0d64

1 2	A audit 209.50.48.88 AAAA audit 2605:7380:1000:1310:24b7:7cff:fe92:0d64

I added another set of records for the new dedicated VM subdomain (for https://test.fearby.com)

A test 94.237.65.54
AAAA test 2a04:3543:1000:2310:24b7:7cff:fe92:468c

1 2	A test 94.237.65.54 AAAA test 2a04:3543:1000:2310:24b7:7cff:fe92:468c

I waited for DNS to replicate around the globe by watching https://www.whatsmydns.net/

Setup a Firewall

On the new dedicated https://test.fearby.com VM, I installed the ufw firewall.

sudo apt-get install ufw

1	sudo apt-get install ufw

I configured the firewall to allow minimum ports (and added whitelisted IP for port 22 and added UpCloud DNS servers). I will lock this down some more later.

TIP: If your ISP does not offer a dedicated IP try a VPN. I use https://cyberghostvpn.com on OSX and Android.

Firewall rules.

sudo ufw status numbered

     To                         Action      From
     --                         ------      ----
[ 1] 22                         ALLOW IN    x.x.x.x
[ 2] 80                         ALLOW IN    Anywhere
[ 3] 443                        ALLOW IN    Anywhere
[ 4] 53                         ALLOW IN    93.237.127.9
[ 5] 53                         ALLOW IN    93.237.40.9
[ 6] 25                         DENY IN     Anywhere
[ 7] 80 (v6)                    ALLOW IN    Anywhere (v6)
[ 8] 443 (v6)                   ALLOW IN    Anywhere (v6)
[ 9] 53                         ALLOW IN    2a04:3540:53::1
[10] 53                         ALLOW IN    2a04:3544:53::1
[11] 22                         ALLOW IN    x.x.x.x.x.x.x.x.x
[12] 25 (v6)                    DENY IN     Anywhere (v6)

sudo ufw status numbered

To Action From

-- ------ ----

[ 1] 22 ALLOW IN x.x.x.x

[ 2] 80 ALLOW IN Anywhere

[ 3] 443 ALLOW IN Anywhere

[ 4] 53 ALLOW IN 93.237.127.9

[ 5] 53 ALLOW IN 93.237.40.9

[ 6] 25 DENY IN Anywhere

[ 7] 80 (v6) ALLOW IN Anywhere (v6)

[ 8] 443 (v6) ALLOW IN Anywhere (v6)

[ 9] 53 ALLOW IN 2a04:3540:53::1

[10] 53 ALLOW IN 2a04:3544:53::1

[11] 22 ALLOW IN x.x.x.x.x.x.x.x.x

[12] 25 (v6) DENY IN Anywhere (v6)

I enabled the firewall.

sudo ufw enable

1	sudo ufw enable

Install NGINX (on https://test.fearby.com)

On the new dedicated https://test.fearby.com VM I…

Created a new www root

mkdir /www-root

1	mkdir /www-root

Set permissions

sudo chown -R www-data:www-data /www-root

1	sudo chown -R www-data:www-data /www-root

Installed NGINX

sudo apt-get update
sudo apt-get install nginx

1 2	sudo apt-get update sudo apt-get install nginx

I created a placeholder webpage

sudo nano /www-root/index.html

1	sudo nano /www-root/index.html

Configured the root value in /etc/nginx/sites-available/default

Created a symbolic link of the nginx config

sudo ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default

1	sudo ln -s /etc/nginx/sites-available/default /etc/nginx/sites-enabled/default

Lets Encrypt SSL

I have previously setup Lets encrypt on Ubuntu 16.04 but not 18.04. Certbot had info on setting up Lets Encrypt for 14.x 16.x and 17.x but not 18.x

Full credit for the SSL steps goes to @Linuxize ( tips on setting up Lets Encrypt on Ubuntu 18.04 ). Check out https://linuxize.com/

I installed Lets Encrypt certbot

sudo apt update
sudo apt install certbot

1 2	sudo apt update sudo apt install certbot

I created a new Diffie–Hellman key

mkdir -p /etc/ssl/certs/
sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

1 2	mkdir -p /etc/ssl/certs/ sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Map requests to http://test.fearby.com/.well-known/acme-challenge to /var/lib/letsencrypt/.well-known ( Read the linuxize post for detailed steps ).

mkdir -p /var/lib/letsencrypt/.well-known
chgrp www-data /var/lib/letsencrypt
chmod g+s /var/lib/letsencrypt

mkdir -p /var/lib/letsencrypt/.well-known

chgrp www-data /var/lib/letsencrypt

chmod g+s /var/lib/letsencrypt

Create a /etc/nginx/snippets/letsencrypt.conf on http://test.fearby.com and enforce the redirect.

location ^~ /.well-known/acme-challenge/ {
  allow all;
  root /var/lib/letsencrypt/;
  default_type "text/plain";
  try_files $uri =404;
}

location ^~ /.well-known/acme-challenge/ {

allow all;

root /var/lib/letsencrypt/;

default_type "text/plain";

try_files $uri =404;

}

Create a /etc/nginx/snippets/ssl.conf file on http://test.fearby.com

ssl_dhparam /etc/ssl/certs/dhparam.pem;

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;

ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 30s;

add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;

ssl_dhparam /etc/ssl/certs/dhparam.pem;

ssl_session_timeout 1d;

ssl_session_cache shared:SSL:50m;

ssl_session_tickets off;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';

ssl_prefer_server_ciphers on;

ssl_stapling on;

ssl_stapling_verify on;

resolver 8.8.8.8 8.8.4.4 valid=300s;

resolver_timeout 30s;

add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";

add_header X-Frame-Options SAMEORIGIN;

add_header X-Content-Type-Options nosniff;

Let’s get a certificate

sudo certbot certonly --agree-tos --email your@email.com --webroot -w /var/lib/letsencrypt/ -d test.fearby.com

1	sudo certbot certonly --agree-tos --email your@email.com --webroot -w /var/lib/letsencrypt/ -d test.fearby.com

Certificates have been created 🙂

ls -al /etc/letsencrypt/live/test.fearby.com/
total 12
drwxr-xr-x 2 user user 4096 Jun 26 11:30 .
drwx------ 3 user user 4096 Jun 26 11:30 ..
-rw-r--r-- 1 user user  543 Jun 26 11:30 README
lrwxrwxrwx 1 user user   39 Jun 26 11:30 cert.pem -> ../../archive/test.fearby.com/cert1.pem
lrwxrwxrwx 1 user user   40 Jun 26 11:30 chain.pem -> ../../archive/test.fearby.com/chain1.pem
lrwxrwxrwx 1 user user   44 Jun 26 11:30 fullchain.pem -> ../../archive/test.fearby.com/fullchain1.pem
lrwxrwxrwx 1 user user   42 Jun 26 11:30 privkey.pem -> ../../archive/test.fearby.com/privkey1.pem

ls -al /etc/letsencrypt/live/test.fearby.com/

total 12

drwxr-xr-x 2 user user 4096 Jun 26 11:30 .

drwx------ 3 user user 4096 Jun 26 11:30 ..

-rw-r--r-- 1 user user 543 Jun 26 11:30 README

lrwxrwxrwx 1 user user 39 Jun 26 11:30 cert.pem -> ../../archive/test.fearby.com/cert1.pem

lrwxrwxrwx 1 user user 40 Jun 26 11:30 chain.pem -> ../../archive/test.fearby.com/chain1.pem

lrwxrwxrwx 1 user user 44 Jun 26 11:30 fullchain.pem -> ../../archive/test.fearby.com/fullchain1.pem

lrwxrwxrwx 1 user user 42 Jun 26 11:30 privkey.pem -> ../../archive/test.fearby.com/privkey1.pem

Now lets edit “/etc/nginx/sites-available/default” on https://test.fearby.com VM and add the cert paths.

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        if ($scheme != "https") {
                return 301 https://$host$request_uri;
        }

        ssl_certificate /etc/letsencrypt/live/test.fearby.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/test.fearby.com/privkey.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/test.fearby.com/chain.pem;

        include snippets/ssl.conf;

        #ssl_stapling on; # Requires nginx >= 1.3.7
        # add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";

        root /www-root/;

        include snippets/letsencrypt.conf;

        index index.html;

        server_name test.fearby.com;

        location / {
                try_files $uri $uri/ =404;
        }
}

server {

listen 80 default_server;

listen [::]:80 default_server;

listen 443 ssl http2;

listen [::]:443 ssl http2;

if ($scheme != "https") {

return 301 https://$host$request_uri;

}

ssl_certificate /etc/letsencrypt/live/test.fearby.com/fullchain.pem;

ssl_certificate_key /etc/letsencrypt/live/test.fearby.com/privkey.pem;

ssl_trusted_certificate /etc/letsencrypt/live/test.fearby.com/chain.pem;

include snippets/ssl.conf;

#ssl_stapling on; # Requires nginx >= 1.3.7

# add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

add_header X-Frame-Options DENY;

add_header X-Content-Type-Options nosniff;

add_header X-XSS-Protection "1; mode=block";

root /www-root/;

include snippets/letsencrypt.conf;

index index.html;

server_name test.fearby.com;

location / {

try_files $uri $uri/ =404;

}

Reload NGINX

sudo systemctl reload nginx

1	sudo systemctl reload nginx

sudo nginx -t
sudo nginx -s reload
sudo systemctl reload nginx

sudo nginx -t

sudo nginx -s reload

sudo systemctl reload nginx

Now let’s setup the second subdomain (subsite off https://fearby.com) and SSL

I already have NGINX on https://fearby.com set up a second site.

DNS

We have already setup a DNS record for https://audit.fearby.com (above)

Firewall

Already configured at https://fearby.com

SSL

Because I had an existing Comodo certificate on https://fearby.com I am going to repeat the steps above to generate a new certificate but save the NGINX config to /etc/nginx/sites-available/audit.fearby.com (this activates the second site)

TIP: Follow the Linuxize guide here (for creating ssl.conf, letsencrypt.conf etc config files), Do a backup and restore if need be.

I created a new Diffie–Hellman key

mkdir -p /etc/ssl/certs/
sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

1 2	mkdir -p /etc/ssl/certs/ sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Let’s get a certificate

sudo certbot certonly --agree-tos --email your@email.com --webroot -w /var/lib/letsencrypt/ -d audit.fearby.com

1	sudo certbot certonly --agree-tos --email your@email.com --webroot -w /var/lib/letsencrypt/ -d audit.fearby.com

Configure NGINX

Map requests to http://audit.fearby.com/.well-known/acme-challenge to /var/lib/letsencrypt/.well-known ( Read the linuxize post for detailed steps ).

mkdir -p /var/lib/letsencrypt/.well-known
chgrp www-data /var/lib/letsencrypt
chmod g+s /var/lib/letsencrypt

mkdir -p /var/lib/letsencrypt/.well-known

chgrp www-data /var/lib/letsencrypt

chmod g+s /var/lib/letsencrypt

I created a new NGINX site ( /etc/nginx/sites-available/audit.fearby.com )

#proxy_cache_path /tmp/nginx-cache keys_zone=one:10m;#
server {
        root /www-audit-root;

        # Listen Ports
        listen 80;
        listen [::]:80;
        listen 443 ssl http2;
        listen [::]:443 ssl http2;

        # Default File
        index index.html index.php index.htm;

        # Server Name
        server_name audit.fearby.com;

        include snippets/letsencrypt.conf;

        location / {
                try_files $uri $uri/ =404;
        }

        ssl_certificate /etc/letsencrypt/live/audit.fearby.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/audit.fearby.com/privkey.pem;
        ssl_trusted_certificate /etc/letsencrypt/live/audit.fearby.com/chain.pem;

        ssl_dhparam /etc/ssl/certs/auditdhparam.pem;

        ssl_session_timeout 1d;
        #ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA38$

        ssl_prefer_server_ciphers on;

        ssl_stapling on;
        ssl_stapling_verify on;

        #resolver 8.8.8.8 8.8.4.4 valid=300s;
        #resolver_timeout 30s;

        add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";
        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;

        if ($scheme != "https") {
                return 301 https://$host$request_uri;
        }
}

#proxy_cache_path /tmp/nginx-cache keys_zone=one:10m;#

server {

root /www-audit-root;

# Listen Ports

listen 80;

listen [::]:80;

listen 443 ssl http2;

listen [::]:443 ssl http2;

# Default File

index index.html index.php index.htm;

# Server Name

server_name audit.fearby.com;

include snippets/letsencrypt.conf;

location / {

try_files $uri $uri/ =404;

}

ssl_certificate /etc/letsencrypt/live/audit.fearby.com/fullchain.pem;

ssl_certificate_key /etc/letsencrypt/live/audit.fearby.com/privkey.pem;

ssl_trusted_certificate /etc/letsencrypt/live/audit.fearby.com/chain.pem;

ssl_dhparam /etc/ssl/certs/auditdhparam.pem;

ssl_session_timeout 1d;

#ssl_session_cache shared:SSL:50m;

ssl_session_tickets off;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_prefer_server_ciphers on;

ssl_stapling on;

ssl_stapling_verify on;

#resolver 8.8.8.8 8.8.4.4 valid=300s;

#resolver_timeout 30s;

add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";

add_header X-Frame-Options SAMEORIGIN;

add_header X-Content-Type-Options nosniff;

if ($scheme != "https") {

return 301 https://$host$request_uri;

}

I created a symbolic link of the config file

sudo ln -s /etc/nginx/sites-available/audit.fearby.com /etc/nginx/sites-enabled/audit.fearby.com

1	sudo ln -s /etc/nginx/sites-available/audit.fearby.com /etc/nginx/sites-enabled/audit.fearby.com

Reload NGINX

sudo systemctl reload nginx

1	sudo systemctl reload nginx

sudo nginx -t
sudo nginx -s reload
sudo systemctl reload nginx

sudo nginx -t

sudo nginx -s reload

sudo systemctl reload nginx

How to test the certificate renewal

sudo certbot renew --dry-run

1	sudo certbot renew --dry-run

Automate the renewal in crontab (every 12 hours)

I set this crontab entry up on https://fearby.com and https://test.fearby.com

crontab -e
0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew --renew-hook "systemctl reload nginx"

1 2	crontab -e 0 /12 * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew --renew-hook "systemctl reload nginx"

Conclusion

Yes, I haVe 2 subdomains (1x dedicated VM and the other is a sub-website off an existing server) with SSL certificates.

Ping Results

ping -c 4 fearby.com
PING fearby.com (209.50.48.88): 56 data bytes
64 bytes from 209.50.48.88: icmp_seq=0 ttl=44 time=220.000 ms
64 bytes from 209.50.48.88: icmp_seq=1 ttl=44 time=290.602 ms
64 bytes from 209.50.48.88: icmp_seq=2 ttl=44 time=311.938 ms
64 bytes from 209.50.48.88: icmp_seq=3 ttl=44 time=330.841 ms

--- fearby.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 220.000/288.345/330.841/41.948 ms

ping -c 4 test.fearby.com
PING test.fearby.com (94.237.65.54): 56 data bytes
64 bytes from 94.237.65.54: icmp_seq=0 ttl=44 time=333.590 ms
64 bytes from 94.237.65.54: icmp_seq=1 ttl=44 time=252.433 ms
64 bytes from 94.237.65.54: icmp_seq=2 ttl=44 time=271.153 ms
64 bytes from 94.237.65.54: icmp_seq=3 ttl=44 time=292.685 ms

--- test.fearby.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 252.433/287.465/333.590/30.200 ms

ping -c 4 audit.fearby.com
PING audit.fearby.com (209.50.48.88): 56 data bytes
64 bytes from 209.50.48.88: icmp_seq=0 ttl=44 time=281.662 ms
64 bytes from 209.50.48.88: icmp_seq=1 ttl=44 time=307.676 ms
64 bytes from 209.50.48.88: icmp_seq=2 ttl=44 time=227.985 ms
64 bytes from 209.50.48.88: icmp_seq=3 ttl=44 time=215.566 ms

--- audit.fearby.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 215.566/258.222/307.676/37.845 ms

ping -c 4 fearby.com

PING fearby.com (209.50.48.88): 56 data bytes

64 bytes from 209.50.48.88: icmp_seq=0 ttl=44 time=220.000 ms

64 bytes from 209.50.48.88: icmp_seq=1 ttl=44 time=290.602 ms

64 bytes from 209.50.48.88: icmp_seq=2 ttl=44 time=311.938 ms

64 bytes from 209.50.48.88: icmp_seq=3 ttl=44 time=330.841 ms

--- fearby.com ping statistics ---

4 packets transmitted, 4 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 220.000/288.345/330.841/41.948 ms

ping -c 4 test.fearby.com

PING test.fearby.com (94.237.65.54): 56 data bytes

64 bytes from 94.237.65.54: icmp_seq=0 ttl=44 time=333.590 ms

64 bytes from 94.237.65.54: icmp_seq=1 ttl=44 time=252.433 ms

64 bytes from 94.237.65.54: icmp_seq=2 ttl=44 time=271.153 ms

64 bytes from 94.237.65.54: icmp_seq=3 ttl=44 time=292.685 ms

--- test.fearby.com ping statistics ---

4 packets transmitted, 4 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 252.433/287.465/333.590/30.200 ms

ping -c 4 audit.fearby.com

PING audit.fearby.com (209.50.48.88): 56 data bytes

64 bytes from 209.50.48.88: icmp_seq=0 ttl=44 time=281.662 ms

64 bytes from 209.50.48.88: icmp_seq=1 ttl=44 time=307.676 ms

64 bytes from 209.50.48.88: icmp_seq=2 ttl=44 time=227.985 ms

64 bytes from 209.50.48.88: icmp_seq=3 ttl=44 time=215.566 ms

--- audit.fearby.com ping statistics ---

4 packets transmitted, 4 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 215.566/258.222/307.676/37.845 ms

Webpage Results

Troubleshooting

If you are having troubles generating the initial certificate check that you have not blocked port 80 and don’t have “Strict-Transport-Security” heavers enabled.

sudo certbot certonly --agree-tos --email your@email.com --webroot -w /var/lib/letsencrypt/ -d g
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for yoursubdomain.domain.com
Using the webroot path /var/lib/letsencrypt for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. yoursubdomain.domain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficzLlmg_w6Tc: q%!(EXTRA string=<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: yoursubdomain.domain.com
   Type:   unauthorized
   Detail: Invalid response from
   http://yoursubdomain.domain.com/.well-known/acme-challenge/_QA3jblEydx5mE8I8OdRsd2EdHIj4R-przLlmg_w6Tc:
   q%!(EXTRA string=<html>
   <head><title>404 Not Found</title></head>
   <body bgcolor="white">
   <center><h1>404 Not Found</h1></center>
   <hr><center>)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.

sudo certbot certonly --agree-tos --email your@email.com --webroot -w /var/lib/letsencrypt/ -d g

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Obtaining a new certificate

Performing the following challenges:

http-01 challenge for yoursubdomain.domain.com

Using the webroot path /var/lib/letsencrypt for all unmatched domains.

Waiting for verification...

Cleaning up challenges

Failed authorization procedure. yoursubdomain.domain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficzLlmg_w6Tc: q%!(EXTRA string=<html>

<head><title>404 Not Found</title></head>

<center><h1>404 Not Found</h1></center>

<hr><center>)

IMPORTANT NOTES:

- The following errors were reported by the server:

Domain: yoursubdomain.domain.com

Type: unauthorized

Detail: Invalid response from

http://yoursubdomain.domain.com/.well-known/acme-challenge/_QA3jblEydx5mE8I8OdRsd2EdHIj4R-przLlmg_w6Tc:

q%!(EXTRA string=<html>

<head><title>404 Not Found</title></head>

<center><h1>404 Not Found</h1></center>

<hr><center>)

To fix these errors, please make sure that your domain name was

entered correctly and the DNS A record(s) for that domain

contain(s) the right IP address.

I re-ran the certbot command but pointed to the real /www-root (not/var/lib/letsencrypt/)

Create a new

mkdir /www-root/.well-known/
mkdir /www-root/.well-known/acme-challenge/
sudo certbot certonly --agree-tos --email your@email.com --webroot -w /www-root -d yoursubdomain.domain.com

mkdir /www-root/.well-known/

mkdir /www-root/.well-known/acme-challenge/

sudo certbot certonly --agree-tos --email your@email.com --webroot -w /www-root -d yoursubdomain.domain.com

I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Revision History

v1.1 Troubleshooting

v1.0 Initial Post

How to use the UpCloud API to manage your UpCloud servers

June 17, 2018 by Simon

How to use the UpCloud API to manage your UpCloud servers.

If you have not read my previous posts I have now moved my blog etc to the awesome UpCloud host. Sign up using this link to get $25 free credit.

I recently compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here).

Here is my blog post on moving from Vultr to UpCloud.

Spoiler: UpCloud performance is great.

I have never had an UpCloud page load take longer than 2 seconds since moving.

UpCloud API

UpCloud has an API that we can opt into to using where we can manage servers. Read the official UpCloud API documentation here.

The API allows you to control:

Accounts
Pricing
Zones
Timezones
Plans
Servers
Storages
IP-Addresses
Firewall
Tags
etc

Create a sub-account to query the API

You should create a new user account (in the UpCloud dasbboard) just for API access. I created two accounts for use on my server and on my home laptop and my server (and set a limiting IP(s) that can access it).

Create a Sub Account for API Access

Click My Accounts,
Click User Accounts,
Click Change on your user and enable API connections.
TIP: Set up an IP rule to limit access to your API for security (I set up a VPN to get a static IP on my dynamic IP Internet host at home)).
Save the changes

TIP: Lock down the account to have the minimum permissions required.

e.g

Disable access to the control panel (Untick).
Allow API Connections (Tick) and specify an IP
Disable access to billing contact (Untick).
Disable access to billing section in the control panel (Untick).
Disable allowing of emails to billing contact (Untick).
Allow or Remove access to all server (or manually add access to desired servers)
Allow or Remove access to modify storage (or manually allow or remove access to desired storage)
etc

Save the account.

Now let’s make our first API call

I use OSX and I use the awesome Paw API testing tool from https://paw.cloud (This is not a plug, they are awesome). Postman is a popular API testing tool too. Any good programing language or CLI will allow you to send API requests.

First, let’s prepare the authorization string (this is a Base64 encoded combination of your username and password) read more here.

Head over to https://www.base64encode.org/
Click the Encode tab
Add your “username:password” (without the quotes).
Click Encode

A Base64 string will be outputted 🙂

e.g > eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk

fyi

You can encode also Encode and Decode Base64 from the Ubuntu Command line

Encode Base64 from the CLI Sample

echo -n 'yourapiusername:yoursupersecurepassword' | base64
eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk

1 2	echo -n 'yourapiusername:yoursupersecurepassword' \| base64 eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk

Decode Base64 from the CLI Sample

echo `echo eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk | base64 --decode`
yourapiusername:yoursupersecurepassword

1 2	echo `echo eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk \| base64 --decode` yourapiusername:yoursupersecurepassword

Now we can add an “Authorization Basic” token to the API request in Paw.

A quick test of the UpCloud Prices API endpoint https://api.upcloud.com/1.2/price reveals the API is working.

I can now see a full breakdown of my service prices in JSON 🙂

Query My Account

OK, Let’s see how much credit I have left by querying the https://api.upcloud.com/1.2/account, I duplicated the item in Paw and changed the URL to https://api.upcloud.com/1.2/account but no data returned?

I had to enable “Access to Billing section in Control Panel” for the user before this data returned from the API (make sense).

> HTTP/1.1 200 OK

Query (GET)

GET /1.2/account HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic *******************************************

GET /1.2/account HTTP/1.1

Host: api.upcloud.com

User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23

Authorization: Basic *******************************************

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:23:32 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 91
Server: Apache

{
   "account" : {
      "credits" : 2500.00,
      "username" : "yourapiusername"
   }
}

HTTP/1.1 200 OK

Date: Sun, 17 Jun 2018 04:23:32 GMT

Content-Type: application/json; charset=UTF-8

Connection: close

Content-Length: 91

Server: Apache

{

"account" : {

"credits" : 2500.00,

"username" : "yourapiusername"

}

“2500.00” = cents ($25)

Query All of Your Servers

Ok, Let’s get server information by querying https://api.upcloud.com/1.2/server

Query (GET)

GET /1.2/server HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic ##############base64hash##############

GET /1.2/server HTTP/1.1

Host: api.upcloud.com

User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23

Authorization: Basic ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:32:22 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 1154
Server: Apache

{
   "servers" : {
      "server" : [
         {
            "core_number" : "1",
            "hostname" : "server1nameredacted.com",
            "license" : 0,
            "memory_amount" : "2048",
            "plan" : "1xCPU-2GB",
            "plan_ipv4_bytes" : "3472464313",
            "plan_ipv6_bytes" : "166293599",
            "state" : "started",
            "tags" : {
               "tag" : [
                  "tag1"
               ]
            },
            "title" : "server1nameredacted.com",
            "uuid" : "########-####-####-####-############",
            "zone" : "us-chi1"
         },
         {
            "core_number" : "1",
            "hostname" : "server2nameredacted.com",
            "license" : 0,
            "memory_amount" : "1024",
            "plan" : "1xCPU-1GB",
            "plan_ipv4_bytes" : "198911",
            "plan_ipv6_bytes" : "19742",
            "state" : "started",
            "tags" : {
               "tag" : [
                  "tag2"
               ]
            },
            "title" : "server1nameredacted.com",
            "uuid" : "########-####-####-####-############",
            "zone" : "us-chi1"
         }
      ]
   }
}

HTTP/1.1 200 OK

Date: Sun, 17 Jun 2018 04:32:22 GMT

Content-Type: application/json; charset=UTF-8

Connection: close

Content-Length: 1154

Server: Apache

{

"servers" : {

"server" : [

{

"core_number" : "1",

"hostname" : "server1nameredacted.com",

"license" : 0,

"memory_amount" : "2048",

"plan" : "1xCPU-2GB",

"plan_ipv4_bytes" : "3472464313",

"plan_ipv6_bytes" : "166293599",

"state" : "started",

"tags" : {

"tag" : [

"tag1"

]

"title" : "server1nameredacted.com",

"uuid" : "########-####-####-####-############",

"zone" : "us-chi1"

{

"core_number" : "1",

"hostname" : "server2nameredacted.com",

"license" : 0,

"memory_amount" : "1024",

"plan" : "1xCPU-1GB",

"plan_ipv4_bytes" : "198911",

"plan_ipv6_bytes" : "19742",

"state" : "started",

"tags" : {

"tag" : [

"tag2"

]

"title" : "server1nameredacted.com",

"uuid" : "########-####-####-####-############",

"zone" : "us-chi1"

}

]

}

Query Server Information

I have redated the UUID’s for my servers but once you know them you can query them by hitting https://api.upcloud.com/1.2/server/########-####-####-####-############

Query (GET)

GET /1.2/server/########-####-####-####-############ HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic ##############base64hash##############

GET /1.2/server/########-####-####-####-############ HTTP/1.1

Host: api.upcloud.com

User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23

Authorization: Basic ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:45:14 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 1656
Server: Apache

{
   "server" : {
      "boot_order" : "cdrom,disk",
      "core_number" : "1",
      "firewall" : "on",
      "host" : redacted,
      "hostname" : "server1nameredacted.com",
      "ip_addresses" : {
         "ip_address" : [
            {
               "access" : "private",
               "address" : "##.#.#.###",
               "family" : "IPv4"
            },
            {
               "access" : "public",
               "address" : "###.###.###.###",
               "family" : "IPv4",
               "part_of_plan" : "yes"
            },
            {
               "access" : "public",
               "address" : "####:####:####:####:####:####:########",
               "family" : "IPv6"
            }
         ]
      },
      "license" : 0,
      "memory_amount" : "2048",
      "nic_model" : "virtio",
      "plan" : "1xCPU-2GB",
      "plan_ipv4_bytes" : "3519033266",
      "plan_ipv6_bytes" : "168200052",
      "state" : "started",
      "storage_devices" : {
         "storage_device" : [
            {
               "address" : "virtio:0",
               "boot_disk" : "0",
               "part_of_plan" : "yes",
               "storage" : "########-####-####-####-############",
               "storage_size" : 50,
               "storage_title" : "system",
               "type" : "disk"
            }
         ]
      },
      "tags" : {
         "tag" : [
            "fearby"
         ]
      },
      "timezone" : "Australia/Sydney",
      "title" : "server1nameredacted.com",
      "uuid" : "########-####-####-####-############",
      "video_model" : "cirrus",
      "vnc" : "off",
      "vnc_password" : "#########################",
      "zone" : "us-chi1"
   }
}

HTTP/1.1 200 OK

Date: Sun, 17 Jun 2018 04:45:14 GMT

Content-Type: application/json; charset=UTF-8

Connection: close

Content-Length: 1656

Server: Apache

{

"server" : {

"boot_order" : "cdrom,disk",

"core_number" : "1",

"firewall" : "on",

"host" : redacted,

"hostname" : "server1nameredacted.com",

"ip_addresses" : {

"ip_address" : [

{

"access" : "private",

"address" : "##.#.#.###",

"family" : "IPv4"

{

"access" : "public",

"address" : "###.###.###.###",

"family" : "IPv4",

"part_of_plan" : "yes"

{

"access" : "public",

"address" : "####:####:####:####:####:####:########",

"family" : "IPv6"

}

]

"license" : 0,

"memory_amount" : "2048",

"nic_model" : "virtio",

"plan" : "1xCPU-2GB",

"plan_ipv4_bytes" : "3519033266",

"plan_ipv6_bytes" : "168200052",

"state" : "started",

"storage_devices" : {

"storage_device" : [

{

"address" : "virtio:0",

"boot_disk" : "0",

"part_of_plan" : "yes",

"storage" : "########-####-####-####-############",

"storage_size" : 50,

"storage_title" : "system",

"type" : "disk"

}

]

"tags" : {

"tag" : [

"fearby"

]

"timezone" : "Australia/Sydney",

"title" : "server1nameredacted.com",

"uuid" : "########-####-####-####-############",

"video_model" : "cirrus",

"vnc" : "off",

"vnc_password" : "#########################",

"zone" : "us-chi1"

}

The servers name, IPv4 and IPV6 network adapters are listed, CPU(s), Memory, Disk Sized and UUID’s are all visible 🙂

Surprisingly the VNC password is visible (enabling access to the root console).

TIP: Ensure your API account is safe and secure.

Query Storage Information

Now, Let’s query the storage with the GUID from above by querying https://api.upcloud.com/1.2/storage/########-####-####-####-############

Query (GET)

GET /1.2/storage/########-####-####-####-############ HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic  ##############base64hash##############

GET /1.2/storage/########-####-####-####-############ HTTP/1.1

Host: api.upcloud.com

User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23

Authorization: Basic ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:53:36 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 559
Server: Apache

{
   "storage" : {
      "access" : "private",
      "backup_rule" : {},
      "backups" : {
         "backup" : [
            "########-####-####-####-############"
         ]
      },
      "license" : 0,
      "part_of_plan" : "yes",
      "servers" : {
         "server" : [
            "########-####-####-####-############"
         ]
      },
      "size" : 50,
      "state" : "online",
      "tier" : "maxiops",
      "title" : "system",
      "type" : "normal",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

HTTP/1.1 200 OK

Date: Sun, 17 Jun 2018 04:53:36 GMT

Content-Type: application/json; charset=UTF-8

Connection: close

Content-Length: 559

Server: Apache

{

"storage" : {

"access" : "private",

"backup_rule" : {},

"backups" : {

"backup" : [

"########-####-####-####-############"

]

"license" : 0,

"part_of_plan" : "yes",

"servers" : {

"server" : [

"########-####-####-####-############"

]

"size" : 50,

"state" : "online",

"tier" : "maxiops",

"title" : "system",

"type" : "normal",

"uuid" : "########-####-####-####-############",

"zone" : "us-chi1"

}

I can see information about the storage’s assigned server and backups 🙂

Query Backup Information

Backup storage can be queried with the same storge API endpoint https://api.upcloud.com/1.2/storage/########-####-####-####-############

Query (GET)

GET /1.2/storage/014fd483-ea90-4055-b445-bf2011951999 HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic ##############base64hash##############

GET /1.2/storage/014fd483-ea90-4055-b445-bf2011951999 HTTP/1.1

Host: api.upcloud.com

User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23

Authorization: Basic ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 05:01:11 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 412
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-16T04:47:56Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "online",
      "title" : "On-Demand Backup",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

HTTP/1.1 200 OK

Date: Sun, 17 Jun 2018 05:01:11 GMT

Content-Type: application/json; charset=UTF-8

Connection: close

Content-Length: 412

Server: Apache

{

"storage" : {

"access" : "private",

"created" : "2018-06-16T04:47:56Z",

"license" : 0,

"origin" : "########-####-####-####-############",

"servers" : {

"server" : []

"size" : 50,

"state" : "online",

"title" : "On-Demand Backup",

"type" : "backup",

"uuid" : "########-####-####-####-############",

"zone" : "us-chi1"

}

Rename Backup

One thing that I would like to be able to do is to rename on-demand backups in the UpCloud dashboard (this is not a feature yet) but I can rename manual backup’s in the API though 🙂

Boring “On-Demand Backup” label.

I tried sending JSON to https://api.upcloud.com/1.2/storage/########-####-####-####-############ to rename a backup but kept getting an error?

JSON

{
> “storage”: {
> “title”: “Latest manual backup , Working NGINX, PHP, MySQL w Tweaks”,
> “size”: “50”
> }
> }

Result

> “error_code” : “CONTENT_TYPE_INVALID”,
> “error_message” : “The Content-Type header has an invalid value.”

I googled and found an old manual for UpClouds API (official support here).

I added these missing content type headers (108 was the length in chars of the payload)

> Content-Type: application/json; Charset=UTF-8'
> Content-Length: 108

1 2	> Content-Type: application/json; Charset=UTF-8' > Content-Length: 108

Still no go?

I think the content length value is wrong, more here.

I fixed it, it turned out I had a semicolon in the Content-Type value. The JSON RFC always assumes that Content-Type is UTF8 encoded (more here).

This Fails

Content-Type: application/json; charset=utf-8

1	Content-Type: application/json; charset=utf-8

This Works

Content-Type: application/json

1	Content-Type: application/json

Now I can rename my Backup (storage). I manually calculated the length of the JSON payload and added a “Content-Length” header and value.

Query (PUT)

PUT /1.2/storage/########-####-####-####-############ HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Content-Type: application/json
Content-Length: 113
Authorization: Basic ##############base64hash##############

{"storage":{"size":"50","title":"Latest manual backup , Working NGINX, PHP, MySQL w Tweaks"}}

PUT /1.2/storage/########-####-####-####-############ HTTP/1.1

Host: api.upcloud.com

User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23

Content-Type: application/json

Content-Length: 113

Authorization: Basic ##############base64hash##############

{"storage":{"size":"50","title":"Latest manual backup , Working NGINX, PHP, MySQL w Tweaks"}}

Output

HTTP/1.1 202 ACCEPTED
Date: Sun, 17 Jun 2018 05:47:02 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 453
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-16T04:47:56Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "online",
      "title" : "Latest manual backup , Working NGINX, PHP, MySQL w Tweaks",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

HTTP/1.1 202 ACCEPTED

Date: Sun, 17 Jun 2018 05:47:02 GMT

Content-Type: application/json; charset=UTF-8

Connection: close

Content-Length: 453

Server: Apache

{

"storage" : {

"access" : "private",

"created" : "2018-06-16T04:47:56Z",

"license" : 0,

"origin" : "########-####-####-####-############",

"servers" : {

"server" : []

"size" : 50,

"state" : "online",

"title" : "Latest manual backup , Working NGINX, PHP, MySQL w Tweaks",

"type" : "backup",

"uuid" : "########-####-####-####-############",

"zone" : "us-chi1"

}

Success 🙂

Create a Backup

Backups can be performed with a “/backup” added to the end of the query string.

Query (POST)

POST /1.2/storage/########-####-####-####-############/backup HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Content-Type: application/json
Content-Length: 100
Authorization: Basic ##############base64hash##############

{
  "storage": {
    "title": "Sunday 17th Latest backup , Working NGINX, PHP, MySQL w Tweaks"
  }
}

POST /1.2/storage/########-####-####-####-############/backup HTTP/1.1

Host: api.upcloud.com

User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23

Content-Type: application/json

Content-Length: 100

Authorization: Basic ##############base64hash##############

{

"storage": {

"title": "Sunday 17th Latest backup , Working NGINX, PHP, MySQL w Tweaks"

}

Output

HTTP/1.1 201 CREATED
Date: Sun, 17 Jun 2018 06:17:35 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 487
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-17T06:17:35Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "progress" : "0",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "maintenance",
      "title" : "Sunday 17th Latest backup , Working NGINX, PHP, MySQL w Tweaks",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

HTTP/1.1 201 CREATED

Date: Sun, 17 Jun 2018 06:17:35 GMT

Content-Type: application/json; charset=UTF-8

Connection: close

Content-Length: 487

Server: Apache

{

"storage" : {

"access" : "private",

"created" : "2018-06-17T06:17:35Z",

"license" : 0,

"origin" : "########-####-####-####-############",

"progress" : "0",

"servers" : {

"server" : []

"size" : 50,

"state" : "maintenance",

"title" : "Sunday 17th Latest backup , Working NGINX, PHP, MySQL w Tweaks",

"type" : "backup",

"uuid" : "########-####-####-####-############",

"zone" : "us-chi1"

}

Success (UpCloud GUI)

Conclusion

UpCloud does have great API docs.

I can easily integrate this into bash scripts to manage my servers via API and a future Java app for managing servers.

Paw does give CURL output to allow me to copy working API’s for use in BASH 🙂

More to come

BASH Script to Deploy and configure a server on UpCloud via Initalization scripts (or manual) (1 week)
JAVA App to manage your server (3 months)

If you are signing up for UpCloud please consider using my referral code and get $25 credit for free.

Read my setup guide here.

https://www.upcloud.com/register/?promo=D84793

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

V1.1 updated typo

v1.0 Initial Post.

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

June 5, 2018 by Simon

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

Read Part 1, Part 2, Part 3 or Part 4

I ran the MySQL benchmark preparation command again (no problem this time).

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=###################### prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=###################### prepare

sysbench 0.4.12: multi-threaded system evaluation benchmark

Creating table 'sbtest'...

Creating 1000000 records in table 'sbtest'...

Test table and records created

Now I can benchmark MySQL on my main server.

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=################################# --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

1	sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=################################# --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

RAW Output

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            336210
        write:                           0
        other:                           48030
        total:                           384240
    transactions:                        24015  (400.09 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 336210 (5601.24 per sec.)
    other operations:                    48030  (800.18 per sec.)

Test execution summary:
    total time:                          60.0242s
    total number of events:              24015
    total time taken by event execution: 480.0242
    per-request statistics:
         min:                                  1.79ms
         avg:                                 19.99ms
         max:                                141.00ms
         approx.  95 percentile:              37.49ms

Threads fairness:
    events (avg/stddev):           3001.8750/27.36
    execution time (avg/stddev):   60.0030/0.01

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 8

Doing OLTP test.

Running mixed OLTP test

Doing read-only test

Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)

Using "BEGIN" for starting transactions

Using auto_inc on the id column

Threads started!

Time limit exceeded, exiting...

(last message repeated 7 times)

Done.

OLTP test statistics:

queries performed:

write: 0

other: 48030

total: 384240

transactions: 24015 (400.09 per sec.)

deadlocks: 0 (0.00 per sec.)

read/write requests: 336210 (5601.24 per sec.)

other operations: 48030 (800.18 per sec.)

Test execution summary:

total time: 60.0242s

total number of events: 24015

total time taken by event execution: 480.0242

per-request statistics:

min: 1.79ms

avg: 19.99ms

max: 141.00ms

approx. 95 percentile: 37.49ms

Threads fairness:

events (avg/stddev): 3001.8750/27.36

execution time (avg/stddev): 60.0030/0.01

Results

queries performed (in 60 seconds):

read: 336210
other: 48030
total: 384240

I decided to add an index to see if I can speed this query up (read the MySQL index page here). I added an index (in Adminer) on the columns “Id” and “pad” for the sbtest table in the test database

I restarted the MySQL process

mysql restart
[ ok ] Restarting mysql (via systemctl): mysql.service.

1 2	mysql restart [ ok ] Restarting mysql (via systemctl): mysql.service.

I ran the same benchmark again.

Raw Output

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=########################## --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            426538
        write:                           0
        other:                           60934
        total:                           487472
    transactions:                        30467  (507.69 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 426538 (7107.67 per sec.)
    other operations:                    60934  (1015.38 per sec.)

Test execution summary:
    total time:                          60.0110s
    total number of events:              30467
    total time taken by event execution: 479.9124
    per-request statistics:
         min:                                  5.75ms
         avg:                                 15.75ms
         max:                                138.57ms
         approx.  95 percentile:              25.10ms

Threads fairness:
    events (avg/stddev):           3808.3750/8.70
    execution time (avg/stddev):   59.9891/0.00

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=########################## --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 8

Doing OLTP test.

Running mixed OLTP test

Doing read-only test

Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)

Using "BEGIN" for starting transactions

Using auto_inc on the id column

Threads started!

Time limit exceeded, exiting...

(last message repeated 7 times)

Done.

OLTP test statistics:

queries performed:

write: 0

other: 60934

total: 487472

transactions: 30467 (507.69 per sec.)

deadlocks: 0 (0.00 per sec.)

read/write requests: 426538 (7107.67 per sec.)

other operations: 60934 (1015.38 per sec.)

Test execution summary:

total time: 60.0110s

total number of events: 30467

total time taken by event execution: 479.9124

per-request statistics:

min: 5.75ms

avg: 15.75ms

max: 138.57ms

approx. 95 percentile: 25.10ms

Threads fairness:

events (avg/stddev): 3808.3750/8.70

execution time (avg/stddev): 59.9891/0.00

Results

The quick index added 20% extra throughput on queries 🙂

Don’t forget to delete your test database

DROP DATABASE `test`;

1	DROP DATABASE `test`;

Viewing MySQL Index Usage (on the “test” database)

Query to show Index stats for a table ‘test’

SELECT
 OBJECT_SCHEMA as 'Database', OBJECT_NAME as 'Table', 
 INDEX_NAME as 'Index', 
 COUNT_STAR, 
 SUM_TIMER_WAIT,  MIN_TIMER_WAIT, AVG_TIMER_WAIT, MAX_TIMER_WAIT, 
 COUNT_READ, 
 SUM_TIMER_READ, MIN_TIMER_READ, AVG_TIMER_READ, MAX_TIMER_READ,  
 COUNT_FETCH, SUM_TIMER_FETCH, MIN_TIMER_FETCH, AVG_TIMER_FETCH, MAX_TIMER_FETCH
FROM 
 performance_schema.table_io_waits_summary_by_index_usage
WHERE 
 object_schema = 'test'

SELECT

OBJECT_SCHEMA as 'Database', OBJECT_NAME as 'Table',

INDEX_NAME as 'Index',

COUNT_STAR,

SUM_TIMER_WAIT, MIN_TIMER_WAIT, AVG_TIMER_WAIT, MAX_TIMER_WAIT,

COUNT_READ,

SUM_TIMER_READ, MIN_TIMER_READ, AVG_TIMER_READ, MAX_TIMER_READ,

COUNT_FETCH, SUM_TIMER_FETCH, MIN_TIMER_FETCH, AVG_TIMER_FETCH, MAX_TIMER_FETCH

FROM

performance_schema.table_io_waits_summary_by_index_usage

WHERE

object_schema = 'test'

I can see the MySQL PRIMARY index is getting used 🙂

Read more in viewable query stats (columns) here.

Other System Information Tools

Show processor information

cat /proc/cpuinfo

1	cat /proc/cpuinfo

Output

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 61
model name      : Virtual CPU a7769a6388d5
stepping        : 2
microcode       : 0x1
cpu MHz         : 2394.454
cache size      : 16384 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4788.90
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 61

model name : Virtual CPU a7769a6388d5

stepping : 2

microcode : 0x1

cpu MHz : 2394.454

cache size : 16384 KB

physical id : 0

siblings : 1

core id : 0

cpu cores : 1

apicid : 0

initial apicid : 0

fpu : yes

fpu_exception : yes

cpuid level : 13

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat

bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass

bogomips : 4788.90

clflush size : 64

cache_alignment : 64

address sizes : 40 bits physical, 48 bits virtual

power management:

Memory Information

You can assign 512MB, 1GB, 2GB or more memory to a server on Vultr, Read my guide here on upgrading resources for Vultr VM’s here.

Only upgrade your server’s memory when server processes demand it, there is no need to pay for extra idle memory. Read my older guides on upgrading Digital Ocean and AWS servers.

I use the htop utility to monitor memory and processes. The memory usage will depend on how you have configured your server to use connection pools in code, MySQL or services. Also what memory demands do you get in pean bandwidth times?

You can check your server memory details on Ubuntu with this command

cat /proc/meminfo

1	cat /proc/meminfo

Output

MemTotal:        2048104 kB
MemFree:           96176 kB
MemAvailable:     693072 kB
Buffers:          183476 kB
Cached:           526124 kB
SwapCached:            0 kB
Active:          1467220 kB
Inactive:         243228 kB
Active(anon):    1070464 kB
Inactive(anon):    27004 kB
Active(file):     396756 kB
Inactive(file):   216224 kB
Unevictable:        3652 kB
Mlocked:            3652 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                64 kB
Writeback:             0 kB
AnonPages:       1004504 kB
Mapped:           114664 kB
Shmem:             94192 kB
Slab:             192692 kB
SReclaimable:     171892 kB
SUnreclaim:        20800 kB
KernelStack:        3072 kB
PageTables:        20528 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1024052 kB
Committed_AS:    2424332 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    247808 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       67440 kB
DirectMap2M:     2029568 kB

MemTotal: 2048104 kB

MemFree: 96176 kB

MemAvailable: 693072 kB

Buffers: 183476 kB

Cached: 526124 kB

SwapCached: 0 kB

Active: 1467220 kB

Inactive: 243228 kB

Active(anon): 1070464 kB

Inactive(anon): 27004 kB

Active(file): 396756 kB

Inactive(file): 216224 kB

Unevictable: 3652 kB

Mlocked: 3652 kB

SwapTotal: 0 kB

SwapFree: 0 kB

Dirty: 64 kB

Writeback: 0 kB

AnonPages: 1004504 kB

Mapped: 114664 kB

Shmem: 94192 kB

Slab: 192692 kB

SReclaimable: 171892 kB

SUnreclaim: 20800 kB

KernelStack: 3072 kB

PageTables: 20528 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

WritebackTmp: 0 kB

CommitLimit: 1024052 kB

Committed_AS: 2424332 kB

VmallocTotal: 34359738367 kB

VmallocUsed: 0 kB

VmallocChunk: 0 kB

HardwareCorrupted: 0 kB

AnonHugePages: 247808 kB

CmaTotal: 0 kB

CmaFree: 0 kB

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

DirectMap4k: 67440 kB

DirectMap2M: 2029568 kB

Use Memory or Disk (Swap)

You can configure the use of Memory over Disk by configuring your/etc/sysctl.conf file (setting value “vm.swappiness”)

You can check your swap file settings by running the following command

cat /proc/sys/vm/swappiness
1

1 2	cat /proc/sys/vm/swappiness 1

Or By running

sysctl vm.swappiness
vm.swappiness = 1

1 2	sysctl vm.swappiness vm.swappiness = 1

Set a new swap file value by editing /etc/sysctl.conf

sudo nano /etc/sysctl.conf

1	sudo nano /etc/sysctl.conf

Set the following to use more ram over the swap disk.

vm.swappiness = 1

1	vm.swappiness = 1

Read about swappiness values here: https://en.wikipedia.org/wiki/Swappiness

Service Performance

Performance (and allocated resources) depends on the demands of your operating system and installed software

What operating system do you have?

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 16.04.4 LTS

Release: 16.04

Codename: xenial

View NGINX Status, how much memory does it use?

/etc/init.d/nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:25 AEST; 1 weeks 3 days ago
     Docs: man:nginx(8)
 Main PID: #### (nginx)
    Tasks: 3
   Memory: 58.9M
      CPU: 33min 11.515s
   CGroup: /system.slice/nginx.service
           ├─#### nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─#### nginx: worker process
           └─#### nginx: cache manager process

/etc/init.d/nginx status

● nginx.service - A high performance web server and a reverse proxy server

Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)

Active: active (running) since Fri 2018-05-25 21:28:25 AEST; 1 weeks 3 days ago

Docs: man:nginx(8)

Main PID: #### (nginx)

Tasks: 3

Memory: 58.9M

CPU: 33min 11.515s

CGroup: /system.slice/nginx.service

├─#### nginx: master process /usr/sbin/nginx -g daemon on; master_process on;

├─#### nginx: worker process

└─#### nginx: cache manager process

PHP (and Child Worker) status how much memory does it use and how many child workers do you have? Read my add PHP child workers post here (and update to PHP 7.2 here)

sudo service php7.2-fpm status
● php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:26 AEST; 1 weeks 3 days ago
     Docs: man:php-fpm7.2(8)
 Main PID: #### (php-fpm7.2)
   Status: "Processes active: 0, idle: 20, Requests: 75911, slow: 0, Traffic: 0.1req/sec"
    Tasks: 21
   Memory: 694.2M
      CPU: 20h 49min 45.132s
   CGroup: /system.slice/php7.2-fpm.service
           ├─ #### php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
           ├─ #### php-fpm: pool www-acc
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-acc
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           └─ #### php-fpm: pool www-usr

sudo service php7.2-fpm status

● php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager

Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; vendor preset: enabled)

Active: active (running) since Fri 2018-05-25 21:28:26 AEST; 1 weeks 3 days ago

Docs: man:php-fpm7.2(8)

Main PID: #### (php-fpm7.2)

Status: "Processes active: 0, idle: 20, Requests: 75911, slow: 0, Traffic: 0.1req/sec"

Tasks: 21

Memory: 694.2M

CPU: 20h 49min 45.132s

CGroup: /system.slice/php7.2-fpm.service

├─ #### php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)

├─ #### php-fpm: pool www-acc

├─ #### php-fpm: pool www-usr

├─ #### php-fpm: pool www-acc

├─ #### php-fpm: pool www-usr

└─ #### php-fpm: pool www-usr

MySQL Status

sudo service mysql status
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:27 AEST; 1 weeks 3 days ago
 Main PID: ##### (mysqld)
    Tasks: 35
   Memory: 405.9M
      CPU: 2h 17min 31.822s
   CGroup: /system.slice/mysql.service
           └─#### /usr/sbin/mysqld

sudo service mysql status

● mysql.service - MySQL Community Server

Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)

Active: active (running) since Fri 2018-05-25 21:28:27 AEST; 1 weeks 3 days ago

Main PID: ##### (mysqld)

Tasks: 35

Memory: 405.9M

CPU: 2h 17min 31.822s

CGroup: /system.slice/mysql.service

└─#### /usr/sbin/mysqld

Shared VM Hosts

One of the biggest impacts (after server latency) for your server is not the disk performance but the number of hosts/websites on the server who are also using the disk and server resources.

I have 80 other web servers on my server (based on a reverse lookup).

I may move to a dedicated box when I can afford it.

Security

Above all else ensure that security is number 1 priority and make performance second priority.

Scan your site with Zap, Qualys and Kali Linux. Performance means nothing if you are hacked.

Simulated concurrent users

You can use Siege to test the maximum concurrent users accessing your site before the server starts to drop connections.

FYI: If you use Cloudflare (you should) this may not work as it will block connections.

Install Siege

sudo apt-get install siege

1	sudo apt-get install siege

Test your server with 10 concurrent serves for 1 minute

siege -t1m c10 'https://yourserver.com/'

1	siege -t1m c10 'https://yourserver.com/'

Results

siege -t1m c10 'https://yourserver.com/'
** SIEGE 3.0.8
** Preparing 15 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:                    417 hits
Availability:                 100.00 %
Elapsed time:                  59.01 secs
Data transferred:               8.24 MB
Response time:                  1.62 secs
Transaction rate:               7.07 trans/sec
Throughput:                     0.14 MB/sec
Concurrency:                   11.46
Successful transactions:         417
Failed transactions:               0
Longest transaction:            2.26
Shortest transaction:           1.49

siege -t1m c10 'https://yourserver.com/'

** SIEGE 3.0.8

** Preparing 15 concurrent users for battle.

The server is now under siege...

Lifting the server siege... done.

Transactions: 417 hits

Availability: 100.00 %

Elapsed time: 59.01 secs

Data transferred: 8.24 MB

Response time: 1.62 secs

Transaction rate: 7.07 trans/sec

Throughput: 0.14 MB/sec

Concurrency: 11.46

Successful transactions: 417

Failed transactions: 0

Longest transaction: 2.26

Shortest transaction: 1.49

Keep upping the connections (from 10 above) to a limit where connections start dropping.

I tried 25 then 50 concurrent users hitting a server on Digital Ocean and it did not fail.

Conclusion

Choose a server near your customers
Change hosts if one is faster and cheaper
Measure or benchmark your server (and compare over time).
Use Cloudflare

Create your own server today

Create your own server on Vultr here.
Create your own server on Digital Ocean here.
Create your own server on UpCloud here.

And remember you can install the Runcloud server management dashboard here.

I hope this guide helps someone.

< Previous

Read Part 1, Part 2, Part 3 or Part 4

Moving an Ubuntu 16.04 VM on Vultr from one data centre to another via snapshots

April 17, 2018 by Simon

This guide will show how you can move an Ubuntu VM server domain between Vultr data centres via snapshots.

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Sometimes you need to move a sever between locations and/or upgrade the server (to have more memory t install WordPress).

Moving an existing Vultr server

If you don’t have an Ubuntu server click here (follow this guide).

Login to Vultr and specify a source server, click Snapshots and click Take Snapshot.

Wait for the snapshot to finish (It may take 1 hour).

Great, the snapshot is done.

Now I can create a new server (in a different data centre).

Deploy New Instance

Choose a location (Australia is at capacity, so I’ll deploy to Silicon Valley then move again in a few weeks), choose the snapshot to restore, choose a plan, I enabled IPV6/Auto Backups and Private Networking.

TIP: The password for the server will be the same as the source server so write it down.

Click Deploy Now

After a few minutes, you can see the new servers IP address, you can log in to your domain name provider (in my case Namecheap) and update the target IPV4 and IPV6 address.

You can find IPV4 and IPV6 addresses by opening your server, clicking settings then IPxV4 or IPV6.

You will need to update Vultr DNS settings (login to Vultr, Click Servers, Click DNS then edit your existing Domain DNS entry). Add you’re new serves IP addresses.

Update: I added an IPV6/AAAA record too.

Wait for DNS Replication

Goto https://www.whatsmydns.net/ and check the global DNS propagation for your new domain’s server.

If you are happy that the server has been migrated (snapshot restored) and that the domain DNS is pointing to your new server you can delete the old server in the Vultr server list.

Post-Migrate Actions

Setup Daily backups.
Review firewall settings (guide here).
Optional: Install MySQL
Optional: Install PHP
Optional: Install PHP Pooled Connections
Optional: Install WordPress
Optional: Install WordPress CDN
Optional: Configure Cloudflare
etc

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.1 Vultr Link

v1.0 Initial post

Using the Qualys FreeScan Scanner to test your website for online vulnerabilities

March 23, 2018 by Simon

It is possible to deploy a server in minutes to hours but it can take days to secure. What tools can you use to help identify what to secure on your website?

I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line, installing a Free SSL certificate and setting up SSL security.

Security Tools

https://asafaweb.com/ is a good tool for quick scanning
Kali Linux has a number of security tools you can use.
You can run a system audit Lynis Audit.
Checking your site for vulnerabilities with Zap.
Run a Gravity Scan malware and supply chain scan
Use Qualys SSL scan to test your SSL certificate: https://www.ssllabs.com/ssltest/

Qualys SSL Labs SSL Tester is the best tool for checking an SSL certificate strength

Most people don’t know Qualys also has another free (limited to 10 scans) vulnerability scanner for websites.

Goto https://freescan.qualys.com/ and click Start your free account.

Complete the signup form

Now check your email to login and confirm your email account

Login now from the email.

Create a password (why the 25 char max Qualys?)

Enter your website URL and click Scan

The scan can take hours

While the scan was being performed I noticed that Qualys offers alerts (I’ll check this out later): https://www.qualys.com/research/security-alerts/

Yes, the scan can take hours, take a walk or read other posts here.

The scan is almost complete

Yay, my latest scan revealed 0 High, 0 Medium and 0 Low-risk vulnerabilities.

It did report 23 informational alerts like “Firewall Detected“.

Threat Report Results

Patch Report Results

This report was empty (probably because I don’t run Windows)

Threat Report Results

The OWASP report contained partial scan results (maybe the full report is available to pro users)

Previous Scan Results

The Qualys dashboard will show all past scans.

My first scan showed a Low priority issue with the /wp-login.php page as the input fields did not have “autocomplete=”off””, I fixed this by adding “autocomplete=”off”” the removing the page (safer).

The second scan found two issues with cookies (possibly ad banner cookies) and 2 subfolders that I created in past development exercises. I deleted the two sub-folders that were not needed.

The third scan was clean.

Here is a scan of a static website of a friends server (static can be less secure if the server underneath is old or unpatched).

Happy scanning. I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.1 Static Web Server Scan

v1.0 Initial post

Setting up the Debian Kali Linux distro to perform penetration testing of your systems

March 7, 2018 by Simon

This post will show you how to setup the Kali Linux distro to perform penetration testing of your systems

I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Securing your systems is very important (don’t stop) and keep learning (securing ubuntu in the cloud, securing checklist, run a Lynis system audit etc)

snip from: https://www.kali.org/about-us/

“Kali Linux is an open source project that is maintained and funded by Offensive Security, a provider of world-class information security training and penetration testing services. In addition to Kali Linux, Offensive Security also maintains the Exploit Database and the free online course, Metasploit Unleashed.”

Download Kali

I downloaded the torrent version (as the HTTP version kept stopping (even on 50/20 NBN).

After the download finished I checked the SHA sum to verify it’a integrity

cd /Users/username/Downloads/kali-linux-2018.1-amd64
shasum -a 256 ./kali-linux-2018.1-amd64.iso 
ed88466834ceeba65f426235ec191fb3580f71d50364ac5131daec1bf976b317  ./kali-linux-2018.1-amd64.iso

cd /Users/username/Downloads/kali-linux-2018.1-amd64

shasum -a 256 ./kali-linux-2018.1-amd64.iso

ed88466834ceeba65f426235ec191fb3580f71d50364ac5131daec1bf976b317 ./kali-linux-2018.1-amd64.iso

A least it matched the known (or hacked) hash here.

Installing Parallels in a VM on OSX

I use Parallels 11 on OSX to set up a VM os Demina Kali, you can use VirtualBox, VMWare etc.

Hardware: 2x CPU, 2048MB Ram, 32MB Graphics, 64GB Disk.

I selected Graphical Install (English, Australia, American English, host: kali, network: hyrule, New South Wales, Partition: Guided – entire disk, Default, Default, Default, Continue, Yes, Network Mirror: Yes, No Proxy, Installed GRUB bootloader on VM HD.

Post Install

Install Parallel Tools

Official Guide: https://kb.parallels.com/en/123968

I opened the VM then selected the Actions then Install Parallels Tools, this mounted /media/cdrom/, I copied all contents to /temp/

As recommended by the Parallels instal bash script I updated headers.

apt install linux-headers-4.14.0-kali1-amd64

1	apt install linux-headers-4.14.0-kali1-amd64

Then the following from https://kb.parallels.com/en/123968

apt-get clean
apt-get update
apt-get upgrade -y
apt-get dist-upgrade -y
apt-get install dkms kpartx printer-driver-postscript-hp

apt-get clean

apt-get update

apt-get upgrade -y

apt-get dist-upgrade -y

apt-get install dkms kpartx printer-driver-postscript-hp

Parallels will not install, I think I need to upgrade to parallel 12 or 12 as the printer driver detection is not detecting (even though it is installed).

Installing Google Chrome

I used the video below

I have to run chrome with

/usr/bin/gogole-chrome-stable %U --no-sandbox --user-data=dir &

1	/usr/bin/gogole-chrome-stable %U --no-sandbox --user-data=dir &

It works.

Running your first remote vulnerability scan in Kali

I found this video useful in helping me scan and check my systems for exploits

Simple exploit search in Armitage (metasploit)

A quick scan of my server revealed three ports open and (22, 80 and 443). Port 80 redirects to 443 and port 22 is firewalled. I have WordPress and exploits I rued failed to work thanks to patching (always stay ahead of patching and updating of software and the OS.

Without knowing what I was doing I was able to check my WordPress against known exploits.

If you open the Check Exploits menu at the end of the Attacks menu you can do a bulk exploit check.

WP Scan

Kali also comes with a WordPress scanner

wpscan --url https://fearby.com

1	wpscan --url https://fearby.com

This will try and output everything from your web server and WordPress plugins.

/xmlrpc.php was found and I was advised to deny access to that file in NGINX. xmlrpc.php is ok but can be used in denial of service attacks.

location = /xmlrpc.php {
	deny all;
	access_log off;
	log_not_found off;
}

location = /xmlrpc.php {

deny all;

access_log off;

log_not_found off;

}

I had a hit for a vulnerability in a Youtube Embed plugin but I had a patched version.

TIP: Check your WordPress often.

More to come (Draft Post).

OWASP scanner
WPSCAN
Ethical Hacker modules
Cybrary training
Sent tips to @FearbySoftware

Tips

Don’t have unwanted ports open, securely installed software, Use unattended security updates in Ubuntu, update WordPress frequently and limit plugins and also consider running more verbose audit tools like Lynis.

More Reading

Read my OWASP Zap guide on application testing and Cloudflare guide.

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.2 added More Reading links.

v1.1 Added bulk exploit check.

v1.0 Initial post

Securing Ubuntu in the cloud

August 9, 2017 by Simon

It is easy to deploy servers to the cloud within a few minutes, you can have a cloud-based server that you (or others can use). ubuntu has a great guide on setting up basic security issues but what do you need to do.

If you do not secure your server expect it to be hacked into. Below are tips on securing your cloud server.

First, read more on scanning your server with Lynis security scan.

Always use up to date software

Always use update software, malicious users can detect what software you use with sites like shodan.io (or use port scan tools) and then look for weaknesses from well-published lists (e.g WordPress, Windows, MySQL, node, LifeRay, Oracle etc). People can even use Google to search for login pages or sites with passwords in HTML (yes that simple). Once a system is identified by a malicious user they can send automated bots to break into your site (trying millions of passwords a day) or use tools to bypass existing defences (Security researcher Troy Hunt found out it’s child’s play).

Portscan sites like https://mxtoolbox.com/SuperTool.aspx?action=scan are good for knowing what you have exposed.

You can also use local programs like nmap to view open ports

Instal nmap

sudo apt-get install nmap

1	sudo apt-get install nmap

Find open ports

nmap -v -sT localhost

Starting Nmap 7.01 ( https://nmap.org ) at 2017-08-08 23:57 AEST
Initiating Connect Scan at 23:57
Scanning localhost (127.0.0.1) [1000 ports]
Discovered open port 80/tcp on 127.0.0.1
Discovered open port 3306/tcp on 127.0.0.1
Discovered open port 22/tcp on 127.0.0.1
Discovered open port 9101/tcp on 127.0.0.1
Discovered open port 9102/tcp on 127.0.0.1
Discovered open port 9103/tcp on 127.0.0.1
Completed Connect Scan at 23:57, 0.05s elapsed (1000 total ports)
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00020s latency).
Not shown: 994 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
3306/tcp open  mysql
9101/tcp open  jetdirect
9102/tcp open  jetdirect
9103/tcp open  jetdirect

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds
           Raw packets sent: 0 (0B) | Rcvd: 0 (0B)

nmap -v -sT localhost

Starting Nmap 7.01 ( https://nmap.org ) at 2017-08-08 23:57 AEST

Initiating Connect Scan at 23:57

Scanning localhost (127.0.0.1) [1000 ports]

Discovered open port 80/tcp on 127.0.0.1

Discovered open port 3306/tcp on 127.0.0.1

Discovered open port 22/tcp on 127.0.0.1

Discovered open port 9101/tcp on 127.0.0.1

Discovered open port 9102/tcp on 127.0.0.1

Discovered open port 9103/tcp on 127.0.0.1

Completed Connect Scan at 23:57, 0.05s elapsed (1000 total ports)

Nmap scan report for localhost (127.0.0.1)

Host is up (0.00020s latency).

Not shown: 994 closed ports

PORT STATE SERVICE

22/tcp open ssh

80/tcp open http

3306/tcp open mysql

9101/tcp open jetdirect

9102/tcp open jetdirect

9103/tcp open jetdirect

Read data files from: /usr/bin/../share/nmap

Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds

Raw packets sent: 0 (0B) | Rcvd: 0 (0B)

Limit ssh connections

Type	Protocol	Port Range	Source	Comment
HTTP	TCP	80	0.0.0.0/0	Opens a web server port for later
All ICMP	ALL	N/A	0.0.0.0/0	Allows you to ping
All traffic	ALL	All	0.0.0.0/0	Not advisable long term but OK for testing today.
SSH	TCP	22	0.0.0.0/0	Not advisable, try and limit this to known IP’s only.
HTTPS	TCP	443	0.0.0.0/0	Opens a secure web server port for later

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

"If you're not still learning, you're already dying."
- Ryan Holiday - Ego is the Enemy

Follow me on Twitter: @FearbySoftware

View All Posts.

Linux

Updating NGINX to the development branch to get more frequent updates and features over the stable branch

Adding two sub domains (one pointing to a new UpCloud VM and the other pointing to an NGINX subsite) on Ubuntu 18.04

How to use the UpCloud API to manage your UpCloud servers

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

Moving an Ubuntu 16.04 VM on Vultr from one data centre to another via snapshots

Using the Qualys FreeScan Scanner to test your website for online vulnerabilities

Setting up the Debian Kali Linux distro to perform penetration testing of your systems