Code, InfoSec and Server Stuff

Views are my own and not my employer's

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

"If you're not still learning, you're already dying."
- Ryan Holiday - Ego is the Enemy

Follow me on Twitter: @FearbySoftware

View All Posts.

VM

No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

by

No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Advertisement:



Aside

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Domain names for just 88 cents!

Now on with the post.

Murphy’s Law

I recently had an issue where I set up a website for a friend. I invested 6 hours into setting up..

I setup…

  • Debian OS
  • NGINX Web Server
  • MySQL Database
  • PHP 7.2
  • PHP-FPM Child Workers
  • HTTPS Certificate
  • Security (Firewall/Headers/SSH, WordPress, Plugins etc).
  • Installed WordPress and Plugins
  • Setup DNSSEC
  • Etc

I had tested GTMetrix scores = less than 1 second.  Security headers were tested and I was happy with the site.

The server and backups were automatically deleted after 7 days while I was away from my keyboard because I assumed the account was valid and had credits.

Lesson Learned

  • Always have a backup (of the server, setup/ www, MySQL etc).
  • Script setups (Ansible, Puppet or Scripts) to sate time redeploying if need be.
  • Backups are not always available.
  • Do have setup documented (Check)
  • Do have a disaster plan

I have guides on setting up a server on UpCloud, AWS, Vultr, Digital Ocean but setting up can be rather repetitive so how can you prevent resetting up servers?

Why Plan for the Worst

  • Companies disappear.
  • Some hosts go down.
  • Some hosts have weird trial modes and internal process that could take your site down.
  • Human error?
  • Murphy’s Law

How I will prevent this in future

  1. I am building a Java desktop app for Windows/OSX/Linux app that will deploy and set up on UpCloud/Vultr/Digital Ocean providers and allow for 1 click deploy and backup and restore.
  2. I am going to re-establish replication between servers with RSync etc.
  3. I am going to start to automate installs and environments.
  4. I am going to set up a hot (ready to go) Green/Blue mirrored environments (www and DB server) on different providers in case of emergency. Then I can set the active live servers with DNS (blog posts soon).
  5. Consider a server farm (same provider or different providers)

 

I hope this guide helps someone.

Please consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.0 Initial Post

Setup a dedicated Debian subdomain (VM), Install MySQL 14 and connect to it from a WordPress on a different VM

by

This is how I set up a dedicated Debian subdomain (VM), Installed MySQL 14 and connected to it from a WordPress installation on a different VM

Advertisement:



Aside

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving fearby.com from Vultr to UpCloud.

Buy a domain name here

Domain names for just 88 cents!

Now on with the post.

Fearby.com

I will be honest, fearby.com is my play server where I can code, learn about InfoSec and share (It’s also my stroke rehab blog).

There is no faster way to learn than actually doing. The problem is my “doing” usually breaks the live site from time to time (sorry).

I really need to set up a testing environment (DEV-TEST-LIVE or GREEN-BLUE) server(s). GREEN-BLUE has advantages as I can always have a hot spare ready. All I need to do is toggle DNS and I can set the GREEN or BLUE server as the live server.

But first  I need to separate my database from my current fearby.com server and setup up a new web server. Having a Green and Blue server that uses one database server will help with near real-time production website switches.

Dedicated Database Server

I read the following ( Should MySQL and Web Server share the same server? ) at Percona Database Performance Blog. Having a separate database server should not negatively impact performance (It may even help improve speeds).

Deploy a Debian VM (not Ubuntu)

I decided to set up a Debian server instead of Ubuntu (mostly because of the good focus on stability and focus on security within Debian).

I logged into the UpCloud dashboard on my mobile phone and deployed a Debian server in 5 mins.  I will be using my existing how to setup Ubuntu on UpCloud guide (even though this is Debian).

TIP: Sign up to UpCloud using this link to get $25 free UpCloud VM credit.

Deploy Debian Sevrer

Deploy a Debian server setup steps:

  1. Login to UpCloud and go to Create server.
  2. Name your Server (use a fully qualified domain name)
  3. Add a description.
  4. Choose your data centre (Chicago for me)
  5. Choose the server specs (1x CPU, 50GB Disk, 2GB Memory, 2TB Traffic for me)
  6. Name the Primary disk
  7. Choose an operating system (Debian for me)
  8. Select an SSH Key
  9. Choose misc settings
  10. Click Deploy server

After 5 mins your server should be deployed.

Advertisement:



After Deploy

Setup DNS

Login to your DNS provider and create DNS records to the new IP’s (IPv4 and IPv6) provided by UpCloud. It took DNS 12 hours to replicate to my in Australia.

Add a DNS record with your domain registra A NAMe = IPV4 and AAAA Name = IPv6

Setup a Firewall (at UpCloud)

I would recommend you set up a firewall at UpCloud as soon as possible (don’t forget to add the recommended UpCloud DNS IP’s and any whitelisted IP’s your firewall).

Block everything and only allow

  • Port 22: Allow known IP(s) of your ISP or VPN.
  • Port 53: Allow known UpCloud DNS servers
  • Port 80 (ALL)
  • Port 443 (ALL)
  • Port 3306 Allow your WordPress site and known IP(s) of your ISP or VPN.

Read my post on setting up a whitelisted IP on an UpCloud VM… as it is a good idea.

UpCloud thankfully has a copy firewall feature that is very handy.

Copy Firewall rules option at UpCloud

After I set up the firewall I SSH’ed into my server (I use vSSH on OSX buy you could use PUTTY).

I updated the Debian system with the following  command

Get the MySQL Package

Visit http://repo.mysql.com/ and get the URL of the latest apt-config repo deb file (e.g “mysql-apt-config_0.8.9-1_all.deb”). Make a temp folder.

Download the MySQL deb Package

Install the package

Update the system again

Install MySQL on Debian

Secure MySQL

Advertisement:



You can secure the MySQL server deployment (set options as needed)

Install NGINX

I installed NGINX to allow Adminer MySQL GUI to be used

I ran these commands to install NGINX.

I edited my NGINX configuration as required.

  • Set a web server root
  • Set desired headers
  • Optimized NGINX (see past guides here, here and here)

I reloaded NGINX

Advertisement:




Install PHP

I followed this guide to install PHP on Debian.

Install PHP FPM

Increase Upload Limits

You may need to temporarily increase upload limits in NGINX and PHP before you can restore a WordPress database. My feabry.com blog is about 87MB.

Add “client_max_body_size 100M;” to “/etc/nginx/nginx.conf”

Add the following to “/etc/php/7.2/fpm/php.ini”

  • post_max_size = 100M
  • upload_max_filesize = 100M

Restore a backup of my MySQL database in MySQL

You can now use Adminer to restore your blog to MySQL. Read my post here on Adminer here. I used Adminer to move my WordPress site from CPanel to a self-managed server a year ago.

First login to your source server and export your desired database then login to the target server and import the database.

Firewall Check

Don’t forget to allow your WordPress site’s 2x Public IP’s and 1x Private IP to access port 3306 in your UpCloud Firewall.

How to check open ports on your current server

Set MySQL Permissions

Open MySQL

I ran these statements to grant the user logging in on the nominate IP’s access to MySQL.

Reload permissions in MySQL

Allow access to the Debian machine from known IP’s

Edit “/etc/host.allow”

Additions (known safe IP’s that need access to this MySQL remotely).

Tell MySQL to listen on

Edit “/etc/mysql/my.cnf”

Added..

I guess you could change the port to something random???

Restart MySQL

Install a second local firewall on Debian

Advertisement:



Install ufw

Do add the IP of your desired server or VPN to access SSH

Do add the IP of your desired server or VPN to access WWW

Now add the known IP’s (e.g any web servers public (IPv4/IPv6) or Private IP’s) that you wish to grant access to MySQL (e.g the source website that used to have MySQL)

Do add UpCloud DNS Servers to your firewall

Add all other rules as needed (if you stuff up and lock your self out you can login to the server with the Console on UpCloud)

Restart the ufw firewall

Prevent MySQL starting on the source server

Now we can shut down MySQL on the source server (leave it there just in case).

Edit “/etc/init/mysql.conf”

Comment out the line that contains “start on ” and save the file

and run

Reboot

Stop and Disable NGINX on the new DB server

We don’t need NGINX running now the database has been imported with Adminer.

Stop and prevent NGINX from starting up on startup.

Check to see if MySQL is Disabled

Yep

Test access to the database server in PHP code

Add to dbtest.php

Check for open ports.

You can install nmap on another server and scan for open ports

Advertisement:



Install nmap

Scan a server for open ports with nmap

You should see this on a server that has access to see port 3306 (port 3306 should not be visible by non-whitelisted IP’s).  Port 3shouldoudl not be seen via everyone.

You should see something like this on a server that has access to see port 80/443 (a web server)

I’d recommend you use a service like https://pentest-tools.com/network-vulnerability-scanning/tcp-port-scanner-online-nmap# to check for open ports.  https://hackertarget.com/tcp-port-scan/ is a great tool too.

https://www.infobyip.com/tcpportchecker.php is also a free port checker that you can use to verify individual closed ports.

Screeshot of https://www.infobyip.com/tcpportchecker.php

Hardening MySQL and Debian

Read: https://www.debian.org/doc/manuals/securing-debian-howto/ap-checklist.en.html

Configuring WordPress use the dedicated Debian VM

On the source server that used to have MySQL edit your wp-config.php file for WordPress.

Remove

add (read the update below, I changed the DNS IP to the Private IP to have free traffic)

Restart NGINX

Restart PHP-FPM

Conclusion

Nice, I seem to have shaved off 0.3 seconds in load times (25% improvement)

1sec gtmtrix load time

Update: Using a Private IP or Public IP between WordPress and MySQL servers

After I released this blog post (version 1.0 with no help from UpCloud) UpCloud contacted me and said the following.

I will have updated my references in this post and replace the public IP address (that is linked to DNS record for db.fearby.com) and instead use the private ip address (e.g 10.x.x.x), your servers private IP address is listed against the public IPv$ and IPv6 address.

I checked that the local ufw firewall did indeed allow the private IP access to MySQL.

On my new Debian MySQL server, I edited the file /etc/mysql/my.cnf and changed the IP to the private IP and not the public IP.

Now it looked like

(10.x.x.x  my Debian servers private IP)

On my WordPress instance, I edited the file  /www-root/wp-config.php

I added the new private host

(10.x.x.x  my Debian servers private IP)

 

Alos on Debian/MySQL ensure you have granted access to the private IP of the WordPress server

Edit /etc/host.allow

Add

Restart MySQL

TIP: Enable UpCloud Backups

Do setup automatic backups (and or take manual backups). Backups are an extra charge but are essential IMHO.

UpCloud backups

Troubleshooting

If you can’t access MySQL log back into MySQL

and run

Reboot

Lower Upload Limits

Don’t forget to lower file upload sizes in NGINX and PHP (e.g 2M) now that the database has been restored.

 

I hope this guide helps someone.

TIP: Sign up to UpCloud using this link to get $25 free UpCloud VM credit.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Advertisement:



Revision History

v1.6 Changed Public IP use to private IP use to ensure we are not charged when the serves sage goes over the quota

v1.5 Fixed 03 type (should have been 0.3)

v1.4 added disable nginx info

v1.3 added https://www.infobyip.com/tcpportchecker.php

v1.1 added https://hackertarget.com/tcp-port-scan/

v1.0 Initial Post

Set up a whitelisted IP on an UpCloud VM and WordPress using a VPN to get a static IP address

by

This is how I set up a whitelisted IP on an UpCloud VM and WordPress using a VPN to get a static IP address

Advertisement:



If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Before you begin

Take a backup of WordPress files + database and take a snapshot of your VM (see my UpCloud VM guide here).

Having a ready backup IS a good idea.

Screenshot of https://my.upcloud.com/server/details/

Why Whitelist

Whitelisting is not bulletproof but it is an important link in the security chain. Security is only as good or bad as the strength of your weakest link.

Using updated software, applying patches, using HTTPS, using a reliable host in a reliable location, using good passwords are equally important as IP filtering. Whitelisting IP’s goes a long way to ensuring you have least access privileges on connections.

Remember to scan your site with OWASP Zap, Qualys and Kali Linux too.

What IP’s are you going to Whitelist?

Q1) Does your ISP offer a static IP address (or a dynamic IP)?

My ISP does NOT provide a static IP by default (I can pay $20 a month for one (that’s too expensive)).

You can check your public IP by loading http://icanhazip.com/ (this will return your public IPV4 address).

Load https://ipv6.icanhazip.com/ to view your IPV6 IP (if you have one)

Q2) Do you need to whitelist IP addresses while on the go (Mobile)? If so I would recommend you whitelist a VPN’s IP or IP range.

Advertisement:



Recently I had Apache web server auto install and knock out my NGINX web server and I needed to login on a mobile device to investigate,  Luckily I whitelist my VPN’s IP and logged in from my mobile device and resolved the issue.

Use a VPN to get a static IP

If you don’t have a static IP or you want to connect to your site on the go (Mobile) you can setup a VPN and use their static IP

I was using http://cyberghostvpn.com/ to have a static IP but a server failure in Sydney caused my defined whitelisted IP to disappear so I change to https://protonvpn.com/ (as Cybergost were unable to provide known IP’s of VPN servers).

TIP: Don’t just whitelist one server, whitelist a few as you never know when a server will go down.

Here is a screenshot of the 1st VPN I tried (Cyberghost), Cyberghost VPN is connected to a specified server (Dallas).

Cyberghost VN screenshot connected to Dallas

I swithced to ProtonVPN.

Here is a screenshot of ProtonVPN connected to a Switzerland server. Read more about Proton VPN here.

Screnshot of Protonvpn

I set Proton VPN to auto start and connect to my desired server

Screenshot of Proton VNS startup settings

Proton VPN offered me a 7-day PLUS trial (All Countries, 5 devices, highest speed, secure core etc) after I started using the free version (3 countries, 1 device, speed low). I assume everyone gets the same PLUS trail offer.

You can view Proton plans and pricing here.

Ok, now that we know how to get a static IP, let’s configure some firewalls.

Network Firewall at UpCloud

I use the awesome UpCloud to hold my domains (read more about UpCloud performance here). You can log in to your UpCloud Dashboard and load the server list, click your server and then click Firewall and define firewalls.

Firewall: Open IPv4/IPv6 ports for:

  1. ICMP
  2. 53 (DNS)
  3. 80 (HTTP)
  4. 443 (HTTPS)

Only allow access to port 22 from whitelisted IP’s (or IP ranges)

Screenshot of UpCloud firewall screen at https://my.upcloud.com/server/details/

I like to set separate firewall rules for IPV4 and IPV6, for TCP or UDP and I limit rules to certain IP range and port.

Ubuntu Firewall

I also like to run a ufw firewall (more information on ufw) on my Ubuntu server (read this guide on securing Ubuntu in the cloud and running a Lynis audit).

Manually setup firewall rules in ufw.

Don’t forget t restart your firewall

Run a  local nmap scan to find open ports

Don’t be concerned if you see open ports from a local nmap scan (e.g port 22 or 3306), these are locally open.  We need to scan externally to see if these ports are opened.

Scan your site with an external nmap tool like pen-test-tools or here.

Screenshot of a public nmap scanYou should not have non-web based service ports freely open externally (web-based ports e.g 80 and 443 are ok)

Port 22 access should be whitelisted to select IP’s only. You should not have any database ports open externally.

Whitelisting WordPress Access

Download WordFence plugin for WordPress from https://www.wordfence.com/

Read more on downloading WordPress plugins from the command line here. Read my past Wordfence post here.

Once Wordfence is installed open the WordFence All Options screen  (/wp-admin/admin.php?page=WordfenceOptions).

Now you can add your static IP (or IP ranges) to the WordFence whitelist.

Picture of WordFence whitelist

Setup auto block for any non whitelisted Itryingng to login to /wp-login.php

I permanently ban any IP accessing my login page (there are many).

What to do with rejected IP connections?

Wordfence will block connections to WordPress. I’d suggest you setup fail2ban to block other unwanted connections at network level too.

Advertisement:



Conclusion

You should now have a VM that will allow port 22 access by whitelisted IP’s and a WordPress that only allows logins from whitelisted IP’s.

Cons

  • If you forget to start your VPN you can’t log in to your VM via port 22 or log in to WordPress (excellent, this is by design).

Pros

  • Secure (need I say more)

 

I hope this guide helps someone.

Please consider using my UpCloud referral code and get $25 UpCloud VM credit for free when you signup to create a new VM.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.2 Added Proton plans link

v1.1 Added auto block WordFence option

v1.0 Initial Post

Adding two sub domains (one pointing to a new UpCloud VM and the other pointing to an NGINX subsite) on Ubuntu 18.04

by

Here is how I added two subdomains (one pointing to a new UpCloud VM and the other pointing to an NGINX subsite) on Ubuntu 18.04

Advertisement:



If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

UpCloud performance is great.

Upcloud Site Speed in GTMetrix

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Goal(s)

Setup 2x subdomains on https://fearby.com

– Sub Domain #1: https://test.fearby.com (pointing to a dedicated UpCloud VM in Singapore for testing).

– Sub Domain #2: https://audit.fearby.com (pointing to a sub-website on the NGINX/VM that runs https://fearby.com )

Let’s set up the first Sub Domain (dedicated VM) and SSL

Backup

Do backup your server first.

VM

I created a second server ($5 month or $0.07c hour 1,024MB Memory, 25GB Disk, 1024 GB Month Data Transfer) at UpCloud. If you don’t already have an account at UpCloud use this link to signup and get $25 free credit ( https://www.upcloud.com/register/?promo=D84793 ). Read my blog post on why UpCloud is awesome and how I moved my domain to UpCloud.

Once I spun up a second server I obtained the IPv4 and IPv6 IP addresses of the new “test” VM from the UpCloud dashboard.

DNS

These DNS records were already in place with my DNS provider (Cloudflare).

I added these DNS records for the subdomains.

I added a new A NAME record for the new shared NGINX subdomain (for https://audit.fearby.com), this subdomain will be a sub-website that is running off the same server as https://fearby.com

I added another set of records for the new dedicated VM  subdomain (for https://test.fearby.com)

I waited for DNS to replicate around the globe by watching https://www.whatsmydns.net/

Setup a Firewall

On the new dedicated https://test.fearby.com VM, I installed the ufw firewall.

I configured the firewall to allow minimum ports (and added whitelisted IP for port 22 and added UpCloud DNS servers). I will lock this down some more later.

TIP: If your ISP does not offer a dedicated IP try a VPN. I use https://cyberghostvpn.com on OSX and Android.

Firewall rules.

I enabled the firewall.

Install NGINX (on https://test.fearby.com)

On the new dedicated https://test.fearby.com VM I…

Created a new www root

Set permissions

Installed NGINX

I created a placeholder webpage

Configured the root value in /etc/nginx/sites-available/default

Created a symbolic link of the nginx config

Lets Encrypt SSL

I have previously setup Lets encrypt on Ubuntu 16.04 but not 18.04. Certbot had info on setting up Lets Encrypt for 14.x 16.x and 17.x but not 18.x

Full credit for the SSL steps goes to @Linuxize ( tips on setting up Lets Encrypt on Ubuntu 18.04 ). Check out https://linuxize.com/

I installed Lets Encrypt certbot

I created a new Diffie–Hellman key

Map requests to http://test.fearby.com/.well-known/acme-challenge to /var/lib/letsencrypt/.well-known ( Read the linuxize post for detailed steps ).

Create a /etc/nginx/snippets/letsencrypt.conf on http://test.fearby.com and enforce the redirect.

Create a /etc/nginx/snippets/ssl.conf file on http://test.fearby.com

Let’s get a certificate

Certificates have been created 🙂

Now lets edit “/etc/nginx/sites-available/default” on https://test.fearby.com VM and add the cert paths.

Advertisement:



Reload NGINX

or

Now let’s setup the second subdomain (subsite off https://fearby.com) and SSL

VM

I already have NGINX on https://fearby.com set up a second site.

DNS

We have already setup a DNS record for https://audit.fearby.com (above)

Firewall

Already configured at https://fearby.com

SSL

Because I had an existing Comodo certificate on https://fearby.com I am going to repeat the steps above to generate a new certificate but save the NGINX config to /etc/nginx/sites-available/audit.fearby.com (this activates the second site)

TIP: Follow the Linuxize guide here (for creating ssl.conf, letsencrypt.conf etc config files), Do a backup and restore if need be.

I created a new Diffie–Hellman key

Let’s get a certificate

Configure NGINX

Map requests to http://audit.fearby.com/.well-known/acme-challenge to /var/lib/letsencrypt/.well-known ( Read the linuxize post for detailed steps ).

I created a new NGINX site ( /etc/nginx/sites-available/audit.fearby.com )

I created a symbolic link of the config file

Reload NGINX

or

How to test the certificate renewal

Automate the renewal in crontab (every 12 hours)

I set this crontab entry up on https://fearby.com and https://test.fearby.com

Conclusion

Yes, I haVe 2 subdomains (1x dedicated VM and the other is a sub-website off an existing server) with SSL certificates.

Ping Results

Webpage Results

Screenshow showing the main site and 2 subdomains in a web browser

Troubleshooting

If you are having troubles generating the initial certificate check that you have not blocked port 80 and don’t have “Strict-Transport-Security” heavers enabled.

I re-ran the certbot command but pointed to the real /www-root (not/var/lib/letsencrypt/)

Create a new

 

 

Advertisement:



I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.1 Troubleshooting

v1.0 Initial Post

How to use the UpCloud API to manage your UpCloud servers

by

How to use the UpCloud API to manage your UpCloud servers.

Advertisement:



If you have not read my previous posts I have now moved my blog etc to the awesome UpCloud host. Sign up using this link to get $25 free credit.

I recently compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here).

Here is my blog post on moving from Vultr to UpCloud.

Spoiler: UpCloud performance is great.

Upcloud Site Speed in GTMetrix

I have never had an UpCloud page load take longer than 2 seconds since moving.

UpCloud API

UpCloud has an API that we can opt into to using where we can manage servers. Read the official UpCloud API documentation here.

The API allows you to control:

  • Accounts
  • Pricing
  • Zones
  • Timezones
  • Plans
  • Servers
  • Storages
  • IP-Addresses
  • Firewall
  • Tags
  • etc

Create a sub-account to query the API

You should create a new user account (in the UpCloud dasbboard) just for API access. I created two accounts for use on my server and on my home laptop and my server (and set a limiting IP(s) that can access it).

Create a Sub Account for API Access

Login to your UpCloud account (create an account here and get $25 free credit),

  1. Click My Accounts,
  2. Click User Accounts,
  3. Click Change on your user and enable API connections.
  4. TIP: Set up an IP rule to limit access to your API for security (I set up a VPN to get a static IP on my dynamic IP Internet host at home)).
  5. Save the changes

Enable API Connections

TIP: Lock down the account to have the minimum permissions required.

e.g

  • Disable access to the control panel (Untick).
  • Allow API Connections (Tick) and specify an IP
  • Disable access to billing contact (Untick).
  • Disable access to billing section in the control panel (Untick).
  • Disable allowing of emails to billing contact (Untick).
  • Allow or Remove access to all server (or manually add access to desired servers)
  • Allow or Remove access to modify storage (or manually allow or remove access to desired storage)
  • etc

Lock down the account to the minimum needed

Save the account.

Now let’s make our first API call

I use OSX and I use the awesome Paw API testing tool from https://paw.cloud (This is not a plug, they are awesome). Postman is a popular API testing tool too. Any good programing language or CLI will allow you to send API requests.

First, let’s prepare the authorization string (this is a Base64 encoded combination of your username and password) read more here.

  1. Head over to https://www.base64encode.org/
  2. Click the Encode tab
  3. Add your “username:password” (without the quotes).
  4. Click Encode

A Base64 string will be outputted 🙂

e.g > eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk

fyi

You can encode also Encode and Decode Base64 from the Ubuntu Command line

Encode Base64 from the CLI Sample

Decode Base64 from the CLI Sample

Now we can add an “Authorization Basic” token to the API request in Paw.

Authorization Header added with my base64 token.

A quick test of the UpCloud Prices API endpoint https://api.upcloud.com/1.2/price reveals the API is working.

Add Authorization Token

I can now see a full breakdown of my service prices in JSON 🙂

Query My Account

OK, Let’s see how much credit I have left by querying the https://api.upcloud.com/1.2/account, I duplicated the item in Paw and changed the URL to https://api.upcloud.com/1.2/account but no data returned?

I had to enable “Access to Billing section in Control Panel” for the user before this data returned from the API (make sense).

> HTTP/1.1 200 OK

Query (GET)

Output

“2500.00” = cents ($25)

Query All of Your Servers

Ok, Let’s get server information by querying https://api.upcloud.com/1.2/server

Query (GET)

Output

Query Server Information

I have redated the UUID’s for my servers but once you know them you can query them by hitting https://api.upcloud.com/1.2/server/########-####-####-####-############

Query (GET)

Output

The servers name, IPv4 and IPV6 network adapters are listed, CPU(s), Memory, Disk Sized and UUID’s are all visible 🙂

Surprisingly the VNC password is visible (enabling access to the root console).

TIP: Ensure your API account is safe and secure.

Query Storage Information

Now, Let’s query the storage with the GUID from above by querying https://api.upcloud.com/1.2/storage/########-####-####-####-############

Query (GET)

Output

I can see information about the storage’s assigned server and backups 🙂

Query Backup Information

Backup storage can be queried with the same storge API endpoint https://api.upcloud.com/1.2/storage/########-####-####-####-############

Query (GET)

Output

Rename Backup

One thing that I would like to be able to do is to rename on-demand backups in the UpCloud dashboard (this is not a feature yet) but I can rename manual backup’s in the API though 🙂

Boring “On-Demand Backup” label.

Rename Backups Not possible in the GUI

I tried sending JSON to https://api.upcloud.com/1.2/storage/########-####-####-####-############ to rename a backup but kept getting an error?

JSON

{
> “storage”: {
> “title”: “Latest manual backup , Working NGINX, PHP, MySQL w Tweaks”,
> “size”: “50”
> }
> }

Result

> “error_code” : “CONTENT_TYPE_INVALID”,
> “error_message” : “The Content-Type header has an invalid value.”

I googled and found an old manual for UpClouds API (official support here).

I added these missing content type headers (108 was the length in chars of the payload)

Still no go?

I think the content length value is wrong, more here.

I fixed it, it turned out I had a semicolon in the Content-Type value. The JSON RFC always assumes that Content-Type is UTF8 encoded (more here).

This Fails

This Works

Now I can rename my Backup (storage). I manually calculated the length of the JSON payload and added a “Content-Length” header and value.

Query (PUT)

Output

Success 🙂

Backup Renamed

Create a Backup

Backups can be performed with a “/backup” added to the end of the query string.

Query (POST)

Output

Success (UpCloud GUI)

Conclusion

UpCloud does have great API docs.

I can easily integrate this into bash scripts to manage my servers via API and a future Java app for managing servers.

Paw does give CURL output to allow me to copy working API’s for use in BASH 🙂

More to come

  1. BASH Script to Deploy and configure a server on UpCloud via Initalization scripts (or manual) (1 week)
  2. JAVA App to manage your server (3 months)

If you are signing up for UpCloud please consider using my referral code and get $25 credit for free.

Read my setup guide here.

https://www.upcloud.com/register/?promo=D84793

I hope this guide helps someone.

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

V1.1 updated typo

v1.0 Initial Post.

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 3 of 4

by

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 3 of 4

Advertisement:



Read Part 1, Part 2, Part 3 or Part 4

I used these commands to generate bonnie++ reports from the data in part 2

Image of results here

Bonnie Results

Network Performace

IMHO Network Latency is the biggest impact on server performance, Read my old post on scalability on a budget here. I am in Australia an having a server in Singapore was too far away and latency was terrible.

Here is a non-scientific example of pinging a Vultr, Digital Ocean and UpCloud server in three different locations (and Google).

Ping Test

Test Ping Results

Latency matters, run a https://www.webpagetest.org/ scan over your site to see why.

Adding https added almost 0.7 seconds to https communications in the past on Digital Ocean (a few thousand kilometres away). The longer the latency the longer HTTPS handshakes take.

SSL

Deploying a server to Singapore (in my experience) is bad if your visitors are in Australia. But deploying to other regions may be lower in cost though. It’s a trade off.

Server Location

Deploy servers as close as you can to your customers is the best tip for performance.

Deploy serves close to your customers

Also, consider setting up Image Optimization and Image CDN plugins (guide here) in WordPress and using Cloudflare (guide here)

Benchmarking with SysBench

Install CPU Benchmark

CPU Benchmark (Vultr/Sydney)

Result

39.15 seconds

CPU Benchmark (Digital Ocean/London)

Result

33.43 sec

CPU Benchmark (UpCloud/Singapore)

Result

23.77 sec

Surprisingly, 1st place in prime generation goes to UpCloud, then Digital Ocean then Vultr.  UpCloud has some good processors.

Processors:

  • UpCLoud (Singapore): Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz
  • Digital Ocean (London): Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
  • Vultr (Sydney): Virtual CPU a7769a6388d5 (Masked/Hidden CPU @ 2.40GHz)

(Lower is better)

prime benchmark results

(oops, typo in the chart should say Vultr)

Benchmark the file IO

Confirm free space

Install Sysbench

I had 10GB free on all servers (Vultr, Digitial Ocean and UpCloud) so I created a 10GB test file.

Now I can run the benchmark and use the pre-created text file.

SysBench description from the Ubuntu manpage.

“SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. The idea of this benchmark suite is to quickly get an impression about system performance without setting up complex database benchmarks or even without installing a database at all.”

SysBench Results (Vultr/Sydney)

SysBench Results (Digital Ocean/London)

SysBench Results (UpCloud/Singapore)

Comparison

Sysbench Results table

sysbench fileio results (text)

Read

  • Vultr (Sydney): 385,920
  • Digital Ocean (London): 944,280
  • UpCloud (Singapore): 944,320

Write

  • Vultr (Sydney): 823,266
  • Digital Ocean (London): 629,520
  • UpCloud (Singapore): 662,880

Other

  • Vultr (Sydney): 1,466,466
  • Digital Ocean (London): 3,588,232
  • UpCloud (Singapore): 2,121,090

Total Read Gb

  • Vultr (Sydney): 5.8887 Gb
  • Digital Ocean (London): 14.409 Gb
  • UpCloud (Singapore): 15.172 Gb

Total Written Gb

  • Vultr (Sydney): 3.9258 Gb
  • Digital Ocean (London): 9.6057 Gb
  • UpCloud (Singapore): 10.115 Gb

Total Transferred Gb

  • Vultr (Sydney): 9.8145 Gb
  • Digital Ocean (London): 24.014 Gb
  • UpCloud (Singapore): 25.287 Gb

Now I can remove test file io benchmark file

Confirm the test file has been deleted

Bonus: Benchmark MySQL (on my main server (Vultr) not on Digital Ocean and UpCLoud)

I tried to run a command

To fix the error I created a test table with Adminer (guide here).

Create Test Table

< Previous – Next >

Read Part 1, Part 2, Part 3 or Part 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

by

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

Advertisement:



Read Part 1, Part 2, Part 3 or Part 4

Measure Disk Performance with Bonnie++

Installing Bonnie++ on Ubuntu

Read this. post on using Bonnie++

Benchmark disk IO with DD and Bonnie++

Starting Bonnie++

Bonnie++ Readme.

Disk io with bonnie++ on Vultr/Sydney

Disk io with bonnie++ on Digital Ocean/London

Disk io with bonnie++ on UpCloud/Singapore