and

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

June 5, 2018 by Simon

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

Read Part 1, Part 2, Part 3 or Part 4

I ran the MySQL benchmark preparation command again (no problem this time).

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=###################### prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Creating table 'sbtest'...
Creating 1000000 records in table 'sbtest'...

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=###################### prepare

sysbench 0.4.12: multi-threaded system evaluation benchmark

Creating table 'sbtest'...

Creating 1000000 records in table 'sbtest'...

Test table and records created

Now I can benchmark MySQL on my main server.

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=################################# --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

1	sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=################################# --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

RAW Output

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            336210
        write:                           0
        other:                           48030
        total:                           384240
    transactions:                        24015  (400.09 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 336210 (5601.24 per sec.)
    other operations:                    48030  (800.18 per sec.)

Test execution summary:
    total time:                          60.0242s
    total number of events:              24015
    total time taken by event execution: 480.0242
    per-request statistics:
         min:                                  1.79ms
         avg:                                 19.99ms
         max:                                141.00ms
         approx.  95 percentile:              37.49ms

Threads fairness:
    events (avg/stddev):           3001.8750/27.36
    execution time (avg/stddev):   60.0030/0.01

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 8

Doing OLTP test.

Running mixed OLTP test

Doing read-only test

Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)

Using "BEGIN" for starting transactions

Using auto_inc on the id column

Threads started!

Time limit exceeded, exiting...

(last message repeated 7 times)

Done.

OLTP test statistics:

queries performed:

write: 0

other: 48030

total: 384240

transactions: 24015 (400.09 per sec.)

deadlocks: 0 (0.00 per sec.)

read/write requests: 336210 (5601.24 per sec.)

other operations: 48030 (800.18 per sec.)

Test execution summary:

total time: 60.0242s

total number of events: 24015

total time taken by event execution: 480.0242

per-request statistics:

min: 1.79ms

avg: 19.99ms

max: 141.00ms

approx. 95 percentile: 37.49ms

Threads fairness:

events (avg/stddev): 3001.8750/27.36

execution time (avg/stddev): 60.0030/0.01

Results

queries performed (in 60 seconds):

read: 336210
other: 48030
total: 384240

I decided to add an index to see if I can speed this query up (read the MySQL index page here). I added an index (in Adminer) on the columns “Id” and “pad” for the sbtest table in the test database

I restarted the MySQL process

mysql restart
[ ok ] Restarting mysql (via systemctl): mysql.service.

1 2	mysql restart [ ok ] Restarting mysql (via systemctl): mysql.service.

I ran the same benchmark again.

Raw Output

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=########################## --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            426538
        write:                           0
        other:                           60934
        total:                           487472
    transactions:                        30467  (507.69 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 426538 (7107.67 per sec.)
    other operations:                    60934  (1015.38 per sec.)

Test execution summary:
    total time:                          60.0110s
    total number of events:              30467
    total time taken by event execution: 479.9124
    per-request statistics:
         min:                                  5.75ms
         avg:                                 15.75ms
         max:                                138.57ms
         approx.  95 percentile:              25.10ms

Threads fairness:
    events (avg/stddev):           3808.3750/8.70
    execution time (avg/stddev):   59.9891/0.00

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=########################## --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 8

Doing OLTP test.

Running mixed OLTP test

Doing read-only test

Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)

Using "BEGIN" for starting transactions

Using auto_inc on the id column

Threads started!

Time limit exceeded, exiting...

(last message repeated 7 times)

Done.

OLTP test statistics:

queries performed:

write: 0

other: 60934

total: 487472

transactions: 30467 (507.69 per sec.)

deadlocks: 0 (0.00 per sec.)

read/write requests: 426538 (7107.67 per sec.)

other operations: 60934 (1015.38 per sec.)

Test execution summary:

total time: 60.0110s

total number of events: 30467

total time taken by event execution: 479.9124

per-request statistics:

min: 5.75ms

avg: 15.75ms

max: 138.57ms

approx. 95 percentile: 25.10ms

Threads fairness:

events (avg/stddev): 3808.3750/8.70

execution time (avg/stddev): 59.9891/0.00

Results

The quick index added 20% extra throughput on queries 🙂

Don’t forget to delete your test database

DROP DATABASE `test`;

1	DROP DATABASE `test`;

Viewing MySQL Index Usage (on the “test” database)

Query to show Index stats for a table ‘test’

SELECT
 OBJECT_SCHEMA as 'Database', OBJECT_NAME as 'Table', 
 INDEX_NAME as 'Index', 
 COUNT_STAR, 
 SUM_TIMER_WAIT,  MIN_TIMER_WAIT, AVG_TIMER_WAIT, MAX_TIMER_WAIT, 
 COUNT_READ, 
 SUM_TIMER_READ, MIN_TIMER_READ, AVG_TIMER_READ, MAX_TIMER_READ,  
 COUNT_FETCH, SUM_TIMER_FETCH, MIN_TIMER_FETCH, AVG_TIMER_FETCH, MAX_TIMER_FETCH
FROM 
 performance_schema.table_io_waits_summary_by_index_usage
WHERE 
 object_schema = 'test'

SELECT

OBJECT_SCHEMA as 'Database', OBJECT_NAME as 'Table',

INDEX_NAME as 'Index',

COUNT_STAR,

SUM_TIMER_WAIT, MIN_TIMER_WAIT, AVG_TIMER_WAIT, MAX_TIMER_WAIT,

COUNT_READ,

SUM_TIMER_READ, MIN_TIMER_READ, AVG_TIMER_READ, MAX_TIMER_READ,

COUNT_FETCH, SUM_TIMER_FETCH, MIN_TIMER_FETCH, AVG_TIMER_FETCH, MAX_TIMER_FETCH

FROM

performance_schema.table_io_waits_summary_by_index_usage

WHERE

object_schema = 'test'

I can see the MySQL PRIMARY index is getting used 🙂

Read more in viewable query stats (columns) here.

Other System Information Tools

Show processor information

cat /proc/cpuinfo

1	cat /proc/cpuinfo

Output

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 61
model name      : Virtual CPU a7769a6388d5
stepping        : 2
microcode       : 0x1
cpu MHz         : 2394.454
cache size      : 16384 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4788.90
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 61

model name : Virtual CPU a7769a6388d5

stepping : 2

microcode : 0x1

cpu MHz : 2394.454

cache size : 16384 KB

physical id : 0

siblings : 1

core id : 0

cpu cores : 1

apicid : 0

initial apicid : 0

fpu : yes

fpu_exception : yes

cpuid level : 13

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single kaiser fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat

bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass

bogomips : 4788.90

clflush size : 64

cache_alignment : 64

address sizes : 40 bits physical, 48 bits virtual

power management:

Memory Information

You can assign 512MB, 1GB, 2GB or more memory to a server on Vultr, Read my guide here on upgrading resources for Vultr VM’s here.

Only upgrade your server’s memory when server processes demand it, there is no need to pay for extra idle memory. Read my older guides on upgrading Digital Ocean and AWS servers.

I use the htop utility to monitor memory and processes. The memory usage will depend on how you have configured your server to use connection pools in code, MySQL or services. Also what memory demands do you get in pean bandwidth times?

You can check your server memory details on Ubuntu with this command

cat /proc/meminfo

1	cat /proc/meminfo

Output

MemTotal:        2048104 kB
MemFree:           96176 kB
MemAvailable:     693072 kB
Buffers:          183476 kB
Cached:           526124 kB
SwapCached:            0 kB
Active:          1467220 kB
Inactive:         243228 kB
Active(anon):    1070464 kB
Inactive(anon):    27004 kB
Active(file):     396756 kB
Inactive(file):   216224 kB
Unevictable:        3652 kB
Mlocked:            3652 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                64 kB
Writeback:             0 kB
AnonPages:       1004504 kB
Mapped:           114664 kB
Shmem:             94192 kB
Slab:             192692 kB
SReclaimable:     171892 kB
SUnreclaim:        20800 kB
KernelStack:        3072 kB
PageTables:        20528 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1024052 kB
Committed_AS:    2424332 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    247808 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       67440 kB
DirectMap2M:     2029568 kB

MemTotal: 2048104 kB

MemFree: 96176 kB

MemAvailable: 693072 kB

Buffers: 183476 kB

Cached: 526124 kB

SwapCached: 0 kB

Active: 1467220 kB

Inactive: 243228 kB

Active(anon): 1070464 kB

Inactive(anon): 27004 kB

Active(file): 396756 kB

Inactive(file): 216224 kB

Unevictable: 3652 kB

Mlocked: 3652 kB

SwapTotal: 0 kB

SwapFree: 0 kB

Dirty: 64 kB

Writeback: 0 kB

AnonPages: 1004504 kB

Mapped: 114664 kB

Shmem: 94192 kB

Slab: 192692 kB

SReclaimable: 171892 kB

SUnreclaim: 20800 kB

KernelStack: 3072 kB

PageTables: 20528 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

WritebackTmp: 0 kB

CommitLimit: 1024052 kB

Committed_AS: 2424332 kB

VmallocTotal: 34359738367 kB

VmallocUsed: 0 kB

VmallocChunk: 0 kB

HardwareCorrupted: 0 kB

AnonHugePages: 247808 kB

CmaTotal: 0 kB

CmaFree: 0 kB

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

DirectMap4k: 67440 kB

DirectMap2M: 2029568 kB

Use Memory or Disk (Swap)

You can configure the use of Memory over Disk by configuring your/etc/sysctl.conf file (setting value “vm.swappiness”)

You can check your swap file settings by running the following command

cat /proc/sys/vm/swappiness
1

1 2	cat /proc/sys/vm/swappiness 1

Or By running

sysctl vm.swappiness
vm.swappiness = 1

1 2	sysctl vm.swappiness vm.swappiness = 1

Set a new swap file value by editing /etc/sysctl.conf

sudo nano /etc/sysctl.conf

1	sudo nano /etc/sysctl.conf

Set the following to use more ram over the swap disk.

vm.swappiness = 1

1	vm.swappiness = 1

Read about swappiness values here: https://en.wikipedia.org/wiki/Swappiness

Service Performance

Performance (and allocated resources) depends on the demands of your operating system and installed software

What operating system do you have?

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.4 LTS
Release:        16.04
Codename:       xenial

lsb_release -a

No LSB modules are available.

Distributor ID: Ubuntu

Description: Ubuntu 16.04.4 LTS

Release: 16.04

Codename: xenial

View NGINX Status, how much memory does it use?

/etc/init.d/nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:25 AEST; 1 weeks 3 days ago
     Docs: man:nginx(8)
 Main PID: #### (nginx)
    Tasks: 3
   Memory: 58.9M
      CPU: 33min 11.515s
   CGroup: /system.slice/nginx.service
           ├─#### nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─#### nginx: worker process
           └─#### nginx: cache manager process

/etc/init.d/nginx status

● nginx.service - A high performance web server and a reverse proxy server

Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)

Active: active (running) since Fri 2018-05-25 21:28:25 AEST; 1 weeks 3 days ago

Docs: man:nginx(8)

Main PID: #### (nginx)

Tasks: 3

Memory: 58.9M

CPU: 33min 11.515s

CGroup: /system.slice/nginx.service

├─#### nginx: master process /usr/sbin/nginx -g daemon on; master_process on;

├─#### nginx: worker process

└─#### nginx: cache manager process

PHP (and Child Worker) status how much memory does it use and how many child workers do you have? Read my add PHP child workers post here (and update to PHP 7.2 here)

sudo service php7.2-fpm status
● php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:26 AEST; 1 weeks 3 days ago
     Docs: man:php-fpm7.2(8)
 Main PID: #### (php-fpm7.2)
   Status: "Processes active: 0, idle: 20, Requests: 75911, slow: 0, Traffic: 0.1req/sec"
    Tasks: 21
   Memory: 694.2M
      CPU: 20h 49min 45.132s
   CGroup: /system.slice/php7.2-fpm.service
           ├─ #### php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
           ├─ #### php-fpm: pool www-acc
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-acc
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           ├─ #### php-fpm: pool www-usr
           └─ #### php-fpm: pool www-usr

sudo service php7.2-fpm status

● php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager

Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; vendor preset: enabled)

Active: active (running) since Fri 2018-05-25 21:28:26 AEST; 1 weeks 3 days ago

Docs: man:php-fpm7.2(8)

Main PID: #### (php-fpm7.2)

Status: "Processes active: 0, idle: 20, Requests: 75911, slow: 0, Traffic: 0.1req/sec"

Tasks: 21

Memory: 694.2M

CPU: 20h 49min 45.132s

CGroup: /system.slice/php7.2-fpm.service

├─ #### php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)

├─ #### php-fpm: pool www-acc

├─ #### php-fpm: pool www-usr

├─ #### php-fpm: pool www-acc

├─ #### php-fpm: pool www-usr

└─ #### php-fpm: pool www-usr

MySQL Status

sudo service mysql status
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-05-25 21:28:27 AEST; 1 weeks 3 days ago
 Main PID: ##### (mysqld)
    Tasks: 35
   Memory: 405.9M
      CPU: 2h 17min 31.822s
   CGroup: /system.slice/mysql.service
           └─#### /usr/sbin/mysqld

sudo service mysql status

● mysql.service - MySQL Community Server

Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)

Active: active (running) since Fri 2018-05-25 21:28:27 AEST; 1 weeks 3 days ago

Main PID: ##### (mysqld)

Tasks: 35

Memory: 405.9M

CPU: 2h 17min 31.822s

CGroup: /system.slice/mysql.service

└─#### /usr/sbin/mysqld

Shared VM Hosts

One of the biggest impacts (after server latency) for your server is not the disk performance but the number of hosts/websites on the server who are also using the disk and server resources.

I have 80 other web servers on my server (based on a reverse lookup).

I may move to a dedicated box when I can afford it.

Security

Above all else ensure that security is number 1 priority and make performance second priority.

Scan your site with Zap, Qualys and Kali Linux. Performance means nothing if you are hacked.

Simulated concurrent users

You can use Siege to test the maximum concurrent users accessing your site before the server starts to drop connections.

FYI: If you use Cloudflare (you should) this may not work as it will block connections.

Install Siege

sudo apt-get install siege

1	sudo apt-get install siege

Test your server with 10 concurrent serves for 1 minute

siege -t1m c10 'https://yourserver.com/'

1	siege -t1m c10 'https://yourserver.com/'

Results

siege -t1m c10 'https://yourserver.com/'
** SIEGE 3.0.8
** Preparing 15 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:                    417 hits
Availability:                 100.00 %
Elapsed time:                  59.01 secs
Data transferred:               8.24 MB
Response time:                  1.62 secs
Transaction rate:               7.07 trans/sec
Throughput:                     0.14 MB/sec
Concurrency:                   11.46
Successful transactions:         417
Failed transactions:               0
Longest transaction:            2.26
Shortest transaction:           1.49

siege -t1m c10 'https://yourserver.com/'

** SIEGE 3.0.8

** Preparing 15 concurrent users for battle.

The server is now under siege...

Lifting the server siege... done.

Transactions: 417 hits

Availability: 100.00 %

Elapsed time: 59.01 secs

Data transferred: 8.24 MB

Response time: 1.62 secs

Transaction rate: 7.07 trans/sec

Throughput: 0.14 MB/sec

Concurrency: 11.46

Successful transactions: 417

Failed transactions: 0

Longest transaction: 2.26

Shortest transaction: 1.49

Keep upping the connections (from 10 above) to a limit where connections start dropping.

I tried 25 then 50 concurrent users hitting a server on Digital Ocean and it did not fail.

Conclusion

Choose a server near your customers
Change hosts if one is faster and cheaper
Measure or benchmark your server (and compare over time).
Use Cloudflare

Create your own server today

Create your own server on Vultr here.
Create your own server on Digital Ocean here.
Create your own server on UpCloud here.

And remember you can install the Runcloud server management dashboard here.

I hope this guide helps someone.

< Previous

Read Part 1, Part 2, Part 3 or Part 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 3 of 4

June 5, 2018 by Simon

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 3 of 4

Read Part 1, Part 2, Part 3 or Part 4

I used these commands to generate bonnie++ reports from the data in part 2

echo "<h1>Bonnie Results</h1>" > /www-data/bonnie.html
echo "<h2>Vultr (Sydney)</h2>" >> /www-data/bonnie.html
echo "1.97,1.97,servername,1,1528177870,4G,,656,99,308954,68,113706,33,1200,92,188671,30,10237,251,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,26067us,119ms,179ms,29139us,26069us,16118us,1463us,703us,880us,263us,119us,593us" | bon_csv2html >> /www-data/bonnie.html
echo "<h2>Digital Ocean (London)</h2>" >> /www-data/bonnie.html
echo "1.97,1.97,servername,1,1528186398,4G,,699,99,778636,74,610414,60,1556,99,1405337,59,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,17678us,10099us,17014us,7027us,3067us,2366us,1243us,376us,611us,108us,59us,181us" | bon_csv2html >> /www-data/bonnie.html
echo "<h2>UpCloud (Singapore)</h2>" >> /www-data/bonnie.html
echo "1.97,1.97,servername,1,1528226703,4G,,1014,99,407179,24,366622,32,2137,99,451886,17,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,11297us,54232us,16443us,4949us,44883us,1595us,264us,340us,561us,138us,66us,327us" | bon_csv2html >> /www-data/bonnie.html

echo "<h1>Bonnie Results</h1>" > /www-data/bonnie.html

echo "<h2>Vultr (Sydney)</h2>" >> /www-data/bonnie.html

echo "1.97,1.97,servername,1,1528177870,4G,,656,99,308954,68,113706,33,1200,92,188671,30,10237,251,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,26067us,119ms,179ms,29139us,26069us,16118us,1463us,703us,880us,263us,119us,593us" | bon_csv2html >> /www-data/bonnie.html

echo "<h2>Digital Ocean (London)</h2>" >> /www-data/bonnie.html

echo "1.97,1.97,servername,1,1528186398,4G,,699,99,778636,74,610414,60,1556,99,1405337,59,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,17678us,10099us,17014us,7027us,3067us,2366us,1243us,376us,611us,108us,59us,181us" | bon_csv2html >> /www-data/bonnie.html

echo "<h2>UpCloud (Singapore)</h2>" >> /www-data/bonnie.html

echo "1.97,1.97,servername,1,1528226703,4G,,1014,99,407179,24,366622,32,2137,99,451886,17,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,11297us,54232us,16443us,4949us,44883us,1595us,264us,340us,561us,138us,66us,327us" | bon_csv2html >> /www-data/bonnie.html

Image of results here

Network Performace

IMHO Network Latency is the biggest impact on server performance, Read my old post on scalability on a budget here. I am in Australia an having a server in Singapore was too far away and latency was terrible.

Here is a non-scientific example of pinging a Vultr, Digital Ocean and UpCloud server in three different locations (and Google).

Test Ping Results

Vultr 132ms Ping Average (Sydney)
Digital Ocean 322ms Ping Average (London)
UpCloud 180ms Ping Average (Singapore)

Latency matters, run a https://www.webpagetest.org/ scan over your site to see why.

Adding https added almost 0.7 seconds to https communications in the past on Digital Ocean (a few thousand kilometres away). The longer the latency the longer HTTPS handshakes take.

Deploying a server to Singapore (in my experience) is bad if your visitors are in Australia. But deploying to other regions may be lower in cost though. It’s a trade off.

Deploy servers as close as you can to your customers is the best tip for performance.

Also, consider setting up Image Optimization and Image CDN plugins (guide here) in WordPress and using Cloudflare (guide here)

Benchmarking with SysBench

Install CPU Benchmark

sudo apt-get install sysbench

1	sudo apt-get install sysbench

CPU Benchmark (Vultr/Sydney)

Result

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          39.1700s
    total number of events:              10000
    total time taken by event execution: 39.1586
    per-request statistics:
         min:                                  2.90ms
         avg:                                  3.92ms
         max:                                 20.44ms
         approx.  95 percentile:               7.43ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   39.1586/0.00

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Doing CPU performance benchmark

Threads started!

Done.

Maximum prime number checked in CPU test: 20000

Test execution summary:

total time: 39.1700s

total number of events: 10000

total time taken by event execution: 39.1586

per-request statistics:

min: 2.90ms

avg: 3.92ms

max: 20.44ms

approx. 95 percentile: 7.43ms

Threads fairness:

events (avg/stddev): 10000.0000/0.00

execution time (avg/stddev): 39.1586/0.00

39.15 seconds

CPU Benchmark (Digital Ocean/London)

sysbench --test=cpu --cpu-max-prime=20000 run

1	sysbench --test=cpu --cpu-max-prime=20000 run

Result

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          33.4382s
    total number of events:              10000
    total time taken by event execution: 33.4352
    per-request statistics:
         min:                                  3.24ms
         avg:                                  3.34ms
         max:                                  6.45ms
         approx.  95 percentile:               3.45ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   33.4352/0.00

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Doing CPU performance benchmark

Threads started!

Done.

Maximum prime number checked in CPU test: 20000

Test execution summary:

total time: 33.4382s

total number of events: 10000

total time taken by event execution: 33.4352

per-request statistics:

min: 3.24ms

avg: 3.34ms

max: 6.45ms

approx. 95 percentile: 3.45ms

Threads fairness:

events (avg/stddev): 10000.0000/0.00

execution time (avg/stddev): 33.4352/0.00

33.43 sec

CPU Benchmark (UpCloud/Singapore)

sysbench --test=cpu --cpu-max-prime=20000 run

1	sysbench --test=cpu --cpu-max-prime=20000 run

Result

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1



Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          23.7809s
    total number of events:              10000
    total time taken by event execution: 23.7780
    per-request statistics:
         min:                                  2.35ms
         avg:                                  2.38ms
         max:                                  6.92ms
         approx.  95 percentile:               2.46ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   23.7780/0.00

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Doing CPU performance benchmark

Threads started!

Done.

Maximum prime number checked in CPU test: 20000

Test execution summary:

total time: 23.7809s

total number of events: 10000

total time taken by event execution: 23.7780

per-request statistics:

min: 2.35ms

avg: 2.38ms

max: 6.92ms

approx. 95 percentile: 2.46ms

Threads fairness:

events (avg/stddev): 10000.0000/0.00

execution time (avg/stddev): 23.7780/0.00

23.77 sec

Surprisingly, 1st place in prime generation goes to UpCloud, then Digital Ocean then Vultr. UpCloud has some good processors.

Processors:

UpCLoud (Singapore): Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz
Digital Ocean (London): Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
Vultr (Sydney): Virtual CPU a7769a6388d5 (Masked/Hidden CPU @ 2.40GHz)

(Lower is better)

(oops, typo in the chart should say Vultr)

Benchmark the file IO

Confirm free space

df -h /

df -h /

Install Sysbench

sudo apt-get install sysbench

1	sudo apt-get install sysbench

I had 10GB free on all servers (Vultr, Digitial Ocean and UpCloud) so I created a 10GB test file.

sysbench --test=fileio --file-total-size=10G prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

128 files, 81920Kb each, 10240Mb total
Creating files for the test...

sysbench --test=fileio --file-total-size=10G prepare

sysbench 0.4.12: multi-threaded system evaluation benchmark

128 files, 81920Kb each, 10240Mb total

Creating files for the test...

Now I can run the benchmark and use the pre-created text file.

sysbench --test=fileio --file-total-size=10G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run

1	sysbench --test=fileio --file-total-size=10G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run

SysBench description from the Ubuntu manpage.

“SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. The idea of this benchmark suite is to quickly get an impression about system performance without setting up complex database benchmarks or even without installing a database at all.”

SysBench Results (Vultr/Sydney)

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  385920 Read, 257280 Write, 823266 Other = 1466466 Total
Read 5.8887Gb  Written 3.9258Gb  Total transferred 9.8145Gb  (33.5Mb/sec)
 2143.98 Requests/sec executed

Test execution summary:
    total time:                          300.0026s
    total number of events:              643200
    total time taken by event execution: 182.4249
    per-request statistics:
         min:                                  0.01ms
         avg:                                  0.28ms
         max:                                 18.12ms
         approx.  95 percentile:               0.55ms

Threads fairness:
    events (avg/stddev):           643200.0000/0.00
    execution time (avg/stddev):   182.4249/0.00

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Initializing random number generator from timer.

Extra file open flags: 0

128 files, 80Mb each

10Gb total file size

Block size 16Kb

Number of random requests for random IO: 0

Read/Write ratio for combined random IO test: 1.50

Periodic FSYNC enabled, calling fsync() each 100 requests.

Calling fsync() at the end of test, Enabled.

Using synchronous I/O mode

Doing random r/w test

Threads started!

Time limit exceeded, exiting...

Done.

Operations performed: 385920 Read, 257280 Write, 823266 Other = 1466466 Total

Read 5.8887Gb Written 3.9258Gb Total transferred 9.8145Gb (33.5Mb/sec)

2143.98 Requests/sec executed

Test execution summary:

total time: 300.0026s

total number of events: 643200

total time taken by event execution: 182.4249

per-request statistics:

min: 0.01ms

avg: 0.28ms

max: 18.12ms

approx. 95 percentile: 0.55ms

Threads fairness:

events (avg/stddev): 643200.0000/0.00

execution time (avg/stddev): 182.4249/0.00

SysBench Results (Digital Ocean/London)

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  944280 Read, 629520 Write, 2014432 Other = 3588232 Total
Read 14.409Gb  Written 9.6057Gb  Total transferred 24.014Gb  (81.968Mb/sec)
 5245.96 Requests/sec executed

Test execution summary:
    total time:                          300.0024s
    total number of events:              1573800
    total time taken by event execution: 160.5558
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.10ms
         max:                                 18.62ms
         approx.  95 percentile:               0.34ms

Threads fairness:
    events (avg/stddev):           1573800.0000/0.00
    execution time (avg/stddev):   160.5558/0.00

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Initializing random number generator from timer.

Extra file open flags: 0

128 files, 80Mb each

10Gb total file size

Block size 16Kb

Number of random requests for random IO: 0

Read/Write ratio for combined random IO test: 1.50

Periodic FSYNC enabled, calling fsync() each 100 requests.

Calling fsync() at the end of test, Enabled.

Using synchronous I/O mode

Doing random r/w test

Threads started!

Time limit exceeded, exiting...

Done.

Operations performed: 944280 Read, 629520 Write, 2014432 Other = 3588232 Total

Read 14.409Gb Written 9.6057Gb Total transferred 24.014Gb (81.968Mb/sec)

5245.96 Requests/sec executed

Test execution summary:

total time: 300.0024s

total number of events: 1573800

total time taken by event execution: 160.5558

per-request statistics:

min: 0.00ms

avg: 0.10ms

max: 18.62ms

approx. 95 percentile: 0.34ms

Threads fairness:

events (avg/stddev): 1573800.0000/0.00

execution time (avg/stddev): 160.5558/0.00

SysBench Results (UpCloud/Singapore)

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.


Extra file open flags: 0
128 files, 80Mb each
10Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.

Operations performed:  994320 Read, 662880 Write, 2121090 Other = 3778290 Total
Read 15.172Gb  Written 10.115Gb  Total transferred 25.287Gb  (86.312Mb/sec)
 5523.97 Requests/sec executed

Test execution summary:
    total time:                          300.0016s
    total number of events:              1657200
    total time taken by event execution: 107.4434
    per-request statistics:
         min:                                  0.00ms
         avg:                                  0.06ms
         max:                                 15.43ms
         approx.  95 percentile:               0.13ms

Threads fairness:
    events (avg/stddev):           1657200.0000/0.00
    execution time (avg/stddev):   107.4434/0.00

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:

Number of threads: 1

Initializing random number generator from timer.

Extra file open flags: 0

128 files, 80Mb each

10Gb total file size

Block size 16Kb

Number of random requests for random IO: 0

Read/Write ratio for combined random IO test: 1.50

Periodic FSYNC enabled, calling fsync() each 100 requests.

Calling fsync() at the end of test, Enabled.

Using synchronous I/O mode

Doing random r/w test

Threads started!

Time limit exceeded, exiting...

Done.

Operations performed: 994320 Read, 662880 Write, 2121090 Other = 3778290 Total

Read 15.172Gb Written 10.115Gb Total transferred 25.287Gb (86.312Mb/sec)

5523.97 Requests/sec executed

Test execution summary:

total time: 300.0016s

total number of events: 1657200

total time taken by event execution: 107.4434

per-request statistics:

min: 0.00ms

avg: 0.06ms

max: 15.43ms

approx. 95 percentile: 0.13ms

Threads fairness:

events (avg/stddev): 1657200.0000/0.00

execution time (avg/stddev): 107.4434/0.00

Comparison

sysbench fileio results (text)

Read

Vultr (Sydney): 385,920
Digital Ocean (London): 944,280
UpCloud (Singapore): 944,320

Write

Vultr (Sydney): 823,266
Digital Ocean (London): 629,520
UpCloud (Singapore): 662,880

Other

Vultr (Sydney): 1,466,466
Digital Ocean (London): 3,588,232
UpCloud (Singapore): 2,121,090

Total Read Gb

Vultr (Sydney): 5.8887 Gb
Digital Ocean (London): 14.409 Gb
UpCloud (Singapore): 15.172 Gb

Total Written Gb

Vultr (Sydney): 3.9258 Gb
Digital Ocean (London): 9.6057 Gb
UpCloud (Singapore): 10.115 Gb

Total Transferred Gb

Vultr (Sydney): 9.8145 Gb
Digital Ocean (London): 24.014 Gb
UpCloud (Singapore): 25.287 Gb

Now I can remove test file io benchmark file

sysbench --test=fileio --file-total-size=2=10G cleanup
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Removing test files...

sysbench --test=fileio --file-total-size=2=10G cleanup

sysbench 0.4.12: multi-threaded system evaluation benchmark

Removing test files...

Confirm the test file has been deleted

df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G   16G   23G  41% /

df -h /

Filesystem Size Used Avail Use% Mounted on

/dev/vda1 40G 16G 23G 41% /

Bonus: Benchmark MySQL (on my main server (Vultr) not on Digital Ocean and UpCLoud)

I tried to run a command

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=#################################### prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

FATAL: unable to connect to MySQL server, aborting...
FATAL: error 1049: Unknown database 'test'
FATAL: failed to connect to database server!

sysbench --test=oltp --oltp-table-size=1000000 --db-driver=mysql --mysql-db=test --mysql-user=root --mysql-password=#################################### prepare

sysbench 0.4.12: multi-threaded system evaluation benchmark

FATAL: unable to connect to MySQL server, aborting...

FATAL: error 1049: Unknown database 'test'

FATAL: failed to connect to database server!

To fix the error I created a test table with Adminer (guide here).

< Previous – Next >

Read Part 1, Part 2, Part 3 or Part 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

June 5, 2018 by Simon

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

Read Part 1, Part 2, Part 3 or Part 4

Measure Disk Performance with Bonnie++

Installing Bonnie++ on Ubuntu

apt-get install bonnie++

1	apt-get install bonnie++

Read this. post on using Bonnie++

Benchmark disk IO with DD and Bonnie++

Starting Bonnie++

bonnie++ -d /tmp -r 2048 -u username

1	bonnie++ -d /tmp -r 2048 -u username

Bonnie++ Readme.

Disk io with bonnie++ on Vultr/Sydney

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
servername 4G 656 99 308954 68 113706 33 1200 92 188671 30 10237 251
Latency 26067us 119ms 179ms 29139us 26069us 16118us
Version 1.97 ------Sequential Create------ --------Random Create--------
servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 1463us 703us 880us 263us 119us 593us
1.97,1.97,servername,1,1528177870,4G,,656,99,308954,68,113706,33,1200,92,188671,30,10237,251,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,26067us,119ms,179ms,29139us,26069us,16118us,1463us,703us,880us,263us,119us,593us

Writing a byte at a time...done

Writing intelligently...done

Rewriting...done

Reading a byte at a time...done

Reading intelligently...done

start 'em...done...done...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.97 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

servername 4G 656 99 308954 68 113706 33 1200 92 188671 30 10237 251

Latency 26067us 119ms 179ms 29139us 26069us 16118us

Version 1.97 ------Sequential Create------ --------Random Create--------

servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

Latency 1463us 703us 880us 263us 119us 593us

1.97,1.97,servername,1,1528177870,4G,,656,99,308954,68,113706,33,1200,92,188671,30,10237,251,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,26067us,119ms,179ms,29139us,26069us,16118us,1463us,703us,880us,263us,119us,593us

Disk io with bonnie++ on Digital Ocean/London

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
servername 4G 699 99 778636 74 610414 60 1556 99 1405337 59 +++++ +++
Latency 17678us 10099us 17014us 7027us 3067us 2366us
Version 1.97 ------Sequential Create------ --------Random Create--------
servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 1243us 376us 611us 108us 59us 181us
1.97,1.97,servername,1,1528186398,4G,,699,99,778636,74,610414,60,1556,99,1405337,59,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,17678us,10099us,17014us,7027us,3067us,2366us,1243us,376us,611us,108us,59us,181us

Writing a byte at a time...done

Writing intelligently...done

Rewriting...done

Reading a byte at a time...done

Reading intelligently...done

start 'em...done...done...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.97 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

servername 4G 699 99 778636 74 610414 60 1556 99 1405337 59 +++++ +++

Latency 17678us 10099us 17014us 7027us 3067us 2366us

Version 1.97 ------Sequential Create------ --------Random Create--------

servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

Latency 1243us 376us 611us 108us 59us 181us

1.97,1.97,servername,1,1528186398,4G,,699,99,778636,74,610414,60,1556,99,1405337,59,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,17678us,10099us,17014us,7027us,3067us,2366us,1243us,376us,611us,108us,59us,181us

Disk io with bonnie++ on UpCloud/Singapore

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
servername 4G 1014 99 407179 24 366622 32 2137 99 451886 17 +++++ +++
Latency 11297us 54232us 16443us 4949us 44883us 1595us
Version 1.97 ------Sequential Create------ --------Random Create--------
servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 264us 340us 561us 138us 66us 327us
1.97,1.97,servername,1,1528226703,4G,,1014,99,407179,24,366622,32,2137,99,451886,17,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,11297us,54232us,16443us,4949us,44883us,1595us,264us,340us,561us,138us,66us,327us

Writing a byte at a time...done

Writing intelligently...done

Rewriting...done

Reading a byte at a time...done

Reading intelligently...done

start 'em...done...done...done...done...done...

Create files in sequential order...done.

Stat files in sequential order...done.

Delete files in sequential order...done.

Create files in random order...done.

Stat files in random order...done.

Delete files in random order...done.

Version 1.97 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

servername 4G 1014 99 407179 24 366622 32 2137 99 451886 17 +++++ +++

Latency 11297us 54232us 16443us 4949us 44883us 1595us

Version 1.97 ------Sequential Create------ --------Random Create--------

servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++

Latency 264us 340us 561us 138us 66us 327us

1.97,1.97,servername,1,1528226703,4G,,1014,99,407179,24,366622,32,2137,99,451886,17,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,11297us,54232us,16443us,4949us,44883us,1595us,264us,340us,561us,138us,66us,327us

Now read this site on how to make sense of this data

< Previous – Next >

Read Part 1, Part 2, Part 3 or Part 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4

June 2, 2018 by Simon

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4. Update: I moved my domain to UpCloud.

Update (June 2018): I moved my domain to UpCloud (they are that awesome). Use this link to signup and get $25 free credit. Read the steps I took to move my domain to UpCloud here.

Comparing Digital Ocean/Vultr and UpCloud Disk IO

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean (all in the search of extra performance) but how do you know when a server performance is ok apart from running GT Metrix and other external site benchmarking tools.

This post is split up as it was too long.

Read Part 1, Part 2, Part 3 or Part 4

Spoiler: It all depends on where your server is located and what you do with it (Tweaks will improve the performance).

P.S This is NOT a paid endorsement or conclusive test (just a quick benchmark/review).

What does your server do?

You need to know what your server does 24/7 and what resources the services need.

I use htop to view real-time and historical usage data for each process.

Tweaking Advice

A friend gave me good advice re-tweaking a cheap host to get good performance

yeah but you are trying to get speed out of budget hosting. Good, fast, cheap, pick 2.
— Kerry Hoath (@khoath) June 2, 2018

I am not a fan of just throwing more money at a host and expecting better performance. Host have unique features and cons., there is no shortage of hosts or host cons.

How can you run synthetic benchmarks to determine comparable performance metrics?

WARNING: Comparing synthetic benchmarks can be far removed from real-world speeds. Benchmark results below were from 3 different servers I have on 3 different hosts in three different locations (the only thing the same was the use of Ubuntu 16.04 $5/m servers). These results are not scientific and should not be used to compare host providers. Benchmark runs were one-off (not averages over multiple timezones/days).

Disk Performance

Speaking of disk performance I noticed this the other day on the RunCloud blog. Faster than SSD (UpCloud)?

UpCloud Faster-than-SSD Cloud Hosting Server (Promo Code Inside)

Runcloud is a server management console that can interface with your domains (read my old review here). I don’t use Runcloud but it is great for those who need a GUI to help manage VM via a dashboard. However, I prefer to know what is going on under the hood. I have investigated webmin in the past though.

Let’s do a quick IO benchmark test between UpCloud, Digital Ocean and Vultr on similarly low end $5/m servers,

Good advice on command line benchmarking tools from a friend.

depends on what sort of load you want to simulate. iozone is old but reliable. bonny might give you more figures you want.
— Kerry Hoath (@khoath) June 2, 2018

Installing iozone to test disk performance

I searched for a post on using iozone (Thanks thegeekstuff). I will be reviewing the “Writer report” and “Reader report”. Read more about iozone here.

View the iozone page for how to break down results.

(image snip from http://www.iozone.org/)

Install iozone on Ubuntu

sudo apt-get install iozone3

1	sudo apt-get install iozone3

Run an iozone disk test and output the results to a spreadsheet.

iozone -a -b iozone.xls

1	iozone -a -b iozone.xls

Now let’s run a Read/Write test on Vultr/Digital Ocean and UpCloud. Multiple runs were not performed, this is not a scientific test (just a simple benchmark test (as is, ignoring sever load and local infrastructure/timezone load)).

iozone Benchmark results for Vultr “Read” (Sydney)

	“4”	“8”	“16”	“32”	“64”	“128”	“256”	“512”	“1024”	“2048”	“4096”	“8192”	“16384”
64	2133730	3363612	4274062	4564786	6421025
128	2248149	3536566	4135958	7082197	4135958	11720614
256	1884399	2699161	3879045	3667079	5971678	5687020	5687020
512	3140488	3736016	3684733	4262523	4610256	2638816	5067142	5684095
1024	1617808	1939207	3411938	3999762	4048778	4614246	3083680	5885083	6609617
2048	1926510	2569678	4423683	4997618	3937075	459605	2896324	3542524	4971585	4707314
4096	1701683	2151300	4209920	5001700	4751325	4869845	5389246	3647681	4928521	6207035	4347346
8192	2063424	2329346	3203763	2937280	3221485	3232699	3626431	3650706	3789200	4110603	3715045	4350542
16384	1738553	2778362	3397613	3679205	3693442	3171501	3524291	3393586	3004024	3552531	3456574	2693845	2488861
32768	0	0	0	0	2952894	3537153	3574875	3768155	4719613	3890280	3394995	2735222	2542914
65536	0	0	0	0	4057489	3610789	3619967	3800078	3275327	3591212	3607188	1770426	2826659
131072	0	0	0	0	3552270	1890742	5275167	3727339	3527607	1753893	3234736	2341111	1378601
262144	0	0	0	0	3798586	1302021	1491429	3712825	3228816	3757963	3715510	2592485	2481061
524288	0	0	0	0	2758756	2487923	3705741	1807328	2118309	3675988	3196367	3394330	2396842

iozone Benchmark results for Digital Ocean “Read” (London)

	4	“8”	“16”	“32”	“64”	“128”	“256”	“512”	“1024”	“2048”	“4096”	“8192”	“16384”
64	4564786	7100397	9006179	10402178	12902017
128	4717434	7082197	8548124	9795896	10567140	10779307
256	4840911	7073132	8271916	9868433	10148242	10651598	1E+07
512	4742616	6909408	8140399	9304292	9638369	10044089	1E+07	10044089
1024	4249053	5917516	6208343	7537599	9300377	10454984	7E+06	7113161	9946527
2048	3885431	6967792	6603549	6845629	10401883	9808036	9E+06	7903836	9308497	7817519
4096	2506983	5953231	6263611	6953144	7774379	6225028	6E+06	8081580	7683972	8081580	8240513
8192	3665114	4850463	5479317	6141364	6277120	6108608	6E+06	6569983	5732541	7166033	6633402	5479317
16384	3673501	4828584	5416182	6187150	6614761	6298872	6E+06	6430310	5984033	6402750	6046159	4791883	3405527
32768	0	0	0	0	4692542	6140929	6E+06	6295642	5231224	6545707	5781108	4513475	3702577
65536	0	0	0	0	6315430	5830131	6E+06	6444695	6219125	6473838	5338595	4248118	3679324
131072	0	0	0	0	6130002	6461496	6E+06	5958068	5983423	6387547	6138078	3994888	3602079
262144	0	0	0	0	6456746	6323727	6E+06	6504146	6390176	6486151	6433963	3955165	3654188
524288	0	0	0	0	1667337	6381456	6E+06	6445708	6448714	6421071	5981200	4155185	3770740

iozone Benchmark results for UpCloud “Read” (Singapore)

	“4”	“8”	“16”	“32”	“64”	“128”	“256”	“512”	“1024”	“2048”	“4096”	“8192”	“16384”
64	6421025	6421025	10821524	12902017	15972885
128	4889281	6406138	9129573	10779307	14200794	14200794
256	5320671	3879045	10758322	8815202	10245071	12812277	12228612
512	4305250	5115422	8844453	8234036	7091952	8394979	7540170	10235583
1024	4339202	4762630	5821271	6163794	6819511	4674510	6479979	8183918	10230845
2048	4204968	5319484	5800851	5816563	6243566	6378005	5953632	6851089	7940367	8229438
4096	4526013	5556581	4817948	5404504	7301864	5759634	5810280	6007355	6919538	8620945	6281934
8192	4298295	5019093	5927357	6036702	6781341	6082655	5855636	6527546	6553692	6792065	6466126	4437634
16384	4282172	5849558	6313919	6635840	6741958	6657054	6423097	5536622	6558575	6442970	4527032	3784777	3901898
32768	0	0	0	0	5825460	5423408	6504198	6665385	6365329	6426343	5263076	3718605	3705971
65536	0	0	0	0	6908075	6623116	6493259	6609738	6311805	6483610	5489674	4035982	3561526
131072	0	0	0	0	5650180	5718949	2465429	5391253	3495911	5784844	5367408	3733490	3582175
262144	0	0	0	0	6814627	6691250	6189661	5906786	6081645	5799913	5247919	4121250	3637601
524288	0	0	0	0	6404764	6309263	5673979	5751609	6288245	6305103	5978680	3911984	3767116

iozone Benchmark results for Vultr “Write” (Sydney)

	“4”	“8”	“16”	“32”	“64”	“128”	“256”	“512”	“1024”	“2048”	“4096”	“8192”	“16384”
64	289322	532815	507625	429630	566551
128	398921	465304	434078	417212	669577	821147
256	530031	613985	820398	474937	891956	815414	370025
512	387576	754083	709019	819085	702295	609421	924123	496091
1024	297233	448522	716089	923488	854073	817340	1203137	1072453	601636
2048	408697	634655	695383	1358134	549657	1295458	821154	797520	964207	258493
4096	236150	433804	1215774	1245025	820832	809958	1371339	914269	921083	1004682	1481431
8192	611113	666677	806286	715219	779825	824294	875947	870091	1046378	791192	1023592	453248
16384	435454	706149	718313	845499	893495	888068	812778	842885	820591	941120	839610	862672	406590
32768	0	0	0	0	465196	786067	938881	627294	890917	968147	872369	871329	842843
65536	0	0	0	0	515057	790172	937568	915601	897235	867197	907562	852002	743856
131072	0	0	0	0	501091	480492	813147	870886	880239	805333	684630	1117578	633185
262144	0	0	0	0	387126	323185	323656	473258	405744	369599	422554	468992	453563
524288	0	0	0	0	325588	380450	392965	451608	303255	355148	386250	432054	416512

iozone Benchmark results for Digital Ocean “Write” (London)

	“4”	“8”	“16”	“32”	“64”	“128”	“256”	“512”	“1024”	“2048”	“4096”	“8192”	“16384”
64	831569	566551	1279447	1363961	1392258
128	652488	1319723	1421023	990891	1663139	1561553
256	1185399	1152323	1534342	1598292	1826695	1707589	1514860
512	1166599	1296159	1399189	1620980	1620980	1361920	1589779	1672748
1024	1079190	1321200	1584972	1917562	1592612	1701108	1718120	1462960	1643814
2048	1210394	1470172	1621719	1550584	1796378	1643753	1713598	1759581	1649117	1488257
4096	916513	1287575	1574718	1406594	1742237	1734148	1652418	1583280	1599346	1661045	1533532
8192	1109745	1318748	1178567	1544201	1502340	1371492	1466747	1499521	1479759	1564878	1291292	1347609
16384	1106205	1282084	1374037	1503649	1429398	1461407	1496119	1578132	1547289	1333431	1203371	1198815	1501316
32768	0	0	0	0	1270914	1406589	1513114	1468226	1558303	1552038	1516336	1443280	1440360
65536	0	0	0	0	1319322	1327984	1311504	1411955	1266988	1359645	1386446	1347092	1368295
131072	0	0	0	0	1100658	1229326	1227197	1318631	1265552	1233306	1227747	1237896	1233502
262144	0	0	0	0	1167160	1064078	1155828	1185185	1086152	1193673	1080872	1062611	1141960
524288	0	0	0	0	977835	1124816	1052757	1219183	1128972	1140177	1091954	1141635	1132063

iozone Benchmark results for UpCloud “Write” (Singapore)

	“8”	“16”	“32”	“64”	“128”	“256”	“512”	“1024”	“2048”	“4096”	“8192”	“16384”
64	1143223	1255511	1562436	1452528	1279447
128	1451764	1406136	1543594	1504659	1852520	1749872
256	1642294	1829808	1970871	1855098	1802167	1952947	2000242
512	1537424	1854787	1801873	2294796	1983258	2124526	1895721	1417662
1024	1434138	1553442	1609925	1931359	2098375	2044438	1872419	1768345	1892218
2048	1562145	1901771	1817281	1848169	1967097	1296240	2267786	2081497	1915768	2007554
4096	1625372	1966378	1924741	1342092	1950306	2078175	1914873	1459656	1995152	2102849	1326855
8192	1444062	1808330	1956503	1924397	2127300	2042328	2135630	1986478	2062557	2061319	1337016	1812049
16384	1667066	1820248	1898495	2051339	2012530	2111080	2119806	1491217	2060875	1974254	1934789	1815823	1921911
32768	0	0	0	0	2057506	1454537	2075621	2070899	1869795	2052896	1892347	1855382	1873440
65536	0	0	0	0	2067127	2077673	2088994	2179809	2087471	2099108	1904723	1642505	1832204
131072	0	0	0	0	1234663	1824959	1304340	1775514	1287481	1560379	1631992	1085609	1675467
262144	0	0	0	0	685774	808487	823824	662524	681762	548308	814946	645663	732176
524288	0	0	0	0	547296	517384	503422	521173	538714	518429	528950	529593	512944

Here is my quick unscientific take on a one pass benchmark results above.

These results need some decoding.

Next >>

Read Part 1, Part 2, Part 3 or Part 4

Ask a question or recommend an article

Update (June 2018): I moved my domain to UpCloud (they are that awesome). Use this link to signup and get $25 free credit. Read the steps I took to move my domain to UpCloud here.

Revision History

v1.2 added the fact that I Moved to UpCloud.

v1.1 Re ran iozone -a -b iozone.xls on all servers.

v1.0 Initial post

Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare

April 14, 2018 by Simon

This post will show you how you can setup Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM) Signatures and Domain-based Message Authentication, Reporting, and Conformance (DMARC) on your GMail (G Suite) email to limit spam and increase security.

I have a number of guides on moving away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. I use Google G Suite to send and receive emails that are linked to my domain (even via the command line) using multiple domains (with aliases).

Buy how can you extend your email security and limit spam?

Enter..

Sender Policy Framework (SPF) for Authorizing Use of Domains in Email

Background: SPF summary from the RFC document from the Internet Engineering Task Force (IETF).

“Email on the Internet can be forged in a number of ways. In particular, existing protocols place no restriction on what a sending host can use as the “MAIL FROM” of a message or the domain given on the SMTP HELO/EHLO commands. This document describes version 1 of the Sender Policy Framework (SPF) protocol, whereby ADministrative Management Domains (ADMDs) can explicitly authorize the hosts that are allowed to use their domain names, and a receiving host can check such authorization.”

Google has a guide in setting up SPF records for your G Suite account.

Scan your site for SPF, DKIM and DMARC configuration(s).

Gmail has a test site where you can check your site SPF, DKIM and DMARC etc: https://toolbox.googleapps.com/apps/checkmx/

How to set up an SPF Record

I followed this guide to set up an SPF record on my G Suite account. I use Cloudflare for my DNS provider so I’ll make my DNS changes there.

Update: Google instructions were wrong, use a TXT record and not a SPF record.

Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare

April 14, 2018 by

Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

April 10, 2018 by Simon

This guide will help you install and setup the pm2 NodejJS process monitor PM2 from Keymetrics.io for free and manage your node apps performance and exceptions.

What is PM2?

PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks. This is the steps I used on Ubuntu 16.04. This is NOT a paid endorsement (just self-documenting).

Key Features of PM2

PM2 offers web-based monitoring dashboard, exception reporting, load balancer, CPU and memory monitoring, transaction tracer and much more for NodeJS apps.

What is PM2?

Official page: http://pm2.keymetrics.io/

More info https://www.npmjs.com/package/pm2

Install PM2

npm install pm2 -g

1	npm install pm2 -g

Install Output

npm install pm2 -g
/usr/bin/pm2 -> /usr/lib/node_modules/pm2/bin/pm2
/usr/bin/pm2-dev -> /usr/lib/node_modules/pm2/bin/pm2-dev
/usr/bin/pm2-docker -> /usr/lib/node_modules/pm2/bin/pm2-docker
/usr/bin/pm2-runtime -> /usr/lib/node_modules/pm2/bin/pm2-runtime
/usr/lib
└─┬ pm2@2.10.2
  ├─┬ async@2.6.0
  │ └── lodash@4.17.5
  ├── blessed@0.1.81
  ├─┬ chalk@1.1.3
  │ ├── ansi-styles@2.2.1
  │ ├── escape-string-regexp@1.0.5
  │ ├─┬ has-ansi@2.0.0
  │ │ └── ansi-regex@2.1.1
  │ ├── strip-ansi@3.0.1
  │ └── supports-color@2.0.0
  ├─┬ chokidar@2.0.3
  │ ├─┬ anymatch@2.0.0
  │ │ └─┬ micromatch@3.1.10
  │ │   ├── arr-diff@4.0.0
  │ │   ├─┬ define-property@2.0.2
  │ │   │ └─┬ is-descriptor@1.0.2
  │ │   │   ├── is-accessor-descriptor@1.0.0
  │ │   │   └── is-data-descriptor@1.0.0
  │ │   ├─┬ extend-shallow@3.0.2
  │ │   │ ├── assign-symbols@1.0.0
  │ │   │ └─┬ is-extendable@1.0.1
  │ │   │   └── is-plain-object@2.0.4
  │ │   ├─┬ extglob@2.0.4
  │ │   │ ├─┬ define-property@1.0.0
  │ │   │ │ └─┬ is-descriptor@1.0.2
  │ │   │ │   ├── is-accessor-descriptor@1.0.0
  │ │   │ │   └── is-data-descriptor@1.0.0
  │ │   │ ├─┬ expand-brackets@2.1.4
  │ │   │ │ ├── debug@2.6.9
  │ │   │ │ ├── define-property@0.2.5
  │ │   │ │ ├── extend-shallow@2.0.1
  │ │   │ │ └── posix-character-classes@0.1.1
  │ │   │ └── extend-shallow@2.0.1
  │ │   ├── fragment-cache@0.2.1
  │ │   ├── kind-of@6.0.2
  │ │   ├─┬ nanomatch@1.2.9
  │ │   │ ├─┬ is-odd@2.0.0
  │ │   │ │ └── is-number@4.0.0
  │ │   │ └── is-windows@1.0.2
  │ │   ├── object.pick@1.3.0
  │ │   └── regex-not@1.0.2
  │ ├── async-each@1.0.1
  │ ├─┬ braces@2.3.2
  │ │ ├── arr-flatten@1.1.0
  │ │ ├── array-unique@0.3.2
  │ │ ├─┬ extend-shallow@2.0.1
  │ │ │ └── is-extendable@0.1.1
  │ │ ├─┬ fill-range@4.0.0
  │ │ │ ├── extend-shallow@2.0.1
  │ │ │ ├─┬ is-number@3.0.0
  │ │ │ │ └─┬ kind-of@3.2.2
  │ │ │ │   └── is-buffer@1.1.6
  │ │ │ ├── repeat-string@1.6.1
  │ │ │ └── to-regex-range@2.1.1
  │ │ ├── isobject@3.0.1
  │ │ ├── repeat-element@1.1.2
  │ │ ├─┬ snapdragon@0.8.2
  │ │ │ ├─┬ base@0.11.2
  │ │ │ │ ├─┬ cache-base@1.0.1
  │ │ │ │ │ ├─┬ collection-visit@1.0.0
  │ │ │ │ │ │ ├── map-visit@1.0.0
  │ │ │ │ │ │ └── object-visit@1.0.1
  │ │ │ │ │ ├── get-value@2.0.6
  │ │ │ │ │ ├─┬ has-value@1.0.0
  │ │ │ │ │ │ └─┬ has-values@1.0.0
  │ │ │ │ │ │   └── kind-of@4.0.0
  │ │ │ │ │ ├─┬ set-value@2.0.0
  │ │ │ │ │ │ └── extend-shallow@2.0.1
  │ │ │ │ │ ├─┬ to-object-path@0.3.0
  │ │ │ │ │ │ └── kind-of@3.2.2
  │ │ │ │ │ ├─┬ union-value@1.0.0
  │ │ │ │ │ │ └─┬ set-value@0.4.3
  │ │ │ │ │ │   └── extend-shallow@2.0.1
  │ │ │ │ │ └─┬ unset-value@1.0.0
  │ │ │ │ │   └─┬ has-value@0.3.1
  │ │ │ │ │     ├── has-values@0.1.4
  │ │ │ │ │     └── isobject@2.1.0
  │ │ │ │ ├─┬ class-utils@0.3.6
  │ │ │ │ │ ├── arr-union@3.1.0
  │ │ │ │ │ ├── define-property@0.2.5
  │ │ │ │ │ └─┬ static-extend@0.1.2
  │ │ │ │ │   ├── define-property@0.2.5
  │ │ │ │ │   └─┬ object-copy@0.1.0
  │ │ │ │ │     ├── copy-descriptor@0.1.1
  │ │ │ │ │     ├── define-property@0.2.5
  │ │ │ │ │     └── kind-of@3.2.2
  │ │ │ │ ├── component-emitter@1.2.1
  │ │ │ │ ├─┬ define-property@1.0.0
  │ │ │ │ │ └─┬ is-descriptor@1.0.2
  │ │ │ │ │   ├── is-accessor-descriptor@1.0.0
  │ │ │ │ │   └── is-data-descriptor@1.0.0
  │ │ │ │ ├─┬ mixin-deep@1.3.1
  │ │ │ │ │ ├── for-in@1.0.2
  │ │ │ │ │ └── is-extendable@1.0.1
  │ │ │ │ └── pascalcase@0.1.1
  │ │ │ ├── debug@2.6.9
  │ │ │ ├─┬ define-property@0.2.5
  │ │ │ │ └─┬ is-descriptor@0.1.6
  │ │ │ │   ├─┬ is-accessor-descriptor@0.1.6
  │ │ │ │   │ └── kind-of@3.2.2
  │ │ │ │   ├─┬ is-data-descriptor@0.1.4
  │ │ │ │   │ └── kind-of@3.2.2
  │ │ │ │   └── kind-of@5.1.0
  │ │ │ ├── extend-shallow@2.0.1
  │ │ │ ├── map-cache@0.2.2
  │ │ │ ├── source-map@0.5.7
  │ │ │ ├─┬ source-map-resolve@0.5.1
  │ │ │ │ ├── atob@2.1.0
  │ │ │ │ ├── decode-uri-component@0.2.0
  │ │ │ │ ├── resolve-url@0.2.1
  │ │ │ │ ├── source-map-url@0.4.0
  │ │ │ │ └── urix@0.1.0
  │ │ │ └── use@3.1.0
  │ │ ├─┬ snapdragon-node@2.1.1
  │ │ │ ├─┬ define-property@1.0.0
  │ │ │ │ └─┬ is-descriptor@1.0.2
  │ │ │ │   ├── is-accessor-descriptor@1.0.0
  │ │ │ │   └── is-data-descriptor@1.0.0
  │ │ │ └─┬ snapdragon-util@3.0.1
  │ │ │   └── kind-of@3.2.2
  │ │ ├── split-string@3.1.0
  │ │ └─┬ to-regex@3.0.2
  │ │   └─┬ safe-regex@1.1.0
  │ │     └── ret@0.1.15
  │ ├─┬ glob-parent@3.1.0
  │ │ ├── is-glob@3.1.0
  │ │ └── path-dirname@1.0.2
  │ ├── inherits@2.0.3
  │ ├─┬ is-binary-path@1.0.1
  │ │ └── binary-extensions@1.11.0
  │ ├─┬ is-glob@4.0.0
  │ │ └── is-extglob@2.1.1
  │ ├─┬ normalize-path@2.1.1
  │ │ └── remove-trailing-separator@1.1.0
  │ ├── path-is-absolute@1.0.1
  │ ├─┬ readdirp@2.1.0
  │ │ ├── graceful-fs@4.1.11
  │ │ ├─┬ minimatch@3.0.4
  │ │ │ └─┬ brace-expansion@1.1.11
  │ │ │   ├── balanced-match@1.0.0
  │ │ │   └── concat-map@0.0.1
  │ │ ├─┬ readable-stream@2.3.6
  │ │ │ ├── core-util-is@1.0.2
  │ │ │ ├── isarray@1.0.0
  │ │ │ ├── process-nextick-args@2.0.0
  │ │ │ ├── safe-buffer@5.1.1
  │ │ │ ├── string_decoder@1.1.1
  │ │ │ └── util-deprecate@1.0.2
  │ │ └── set-immediate-shim@1.0.1
  │ └── upath@1.0.4
  ├── cli-table-redemption@1.0.1
  ├── commander@2.13.0
  ├─┬ cron@1.3.0
  │ └── moment-timezone@0.5.14
  ├─┬ debug@3.1.0
  │ └── ms@2.0.0
  ├── eventemitter2@1.0.5
  ├── fclone@1.0.11
  ├── gkt@1.0.0
  ├─┬ mkdirp@0.5.1
  │ └── minimist@0.0.8
  ├── moment@2.22.0
  ├─┬ needle@2.2.0
  │ ├── debug@2.6.9
  │ ├─┬ iconv-lite@0.4.21
  │ │ └── safer-buffer@2.1.2
  │ └── sax@1.2.4
  ├─┬ nssocket@0.6.0
  │ ├── eventemitter2@0.4.14
  │ └── lazy@1.0.11
  ├── pidusage@1.2.0
  ├─┬ pm2-axon@3.1.0
  │ ├── amp@0.3.1
  │ ├── amp-message@0.1.2
  │ └── escape-regexp@0.0.1
  ├── pm2-axon-rpc@0.5.0
  ├─┬ pm2-deploy@0.3.9
  │ ├── async@1.5.2
  │ └── tv4@1.3.0
  ├─┬ pm2-multimeter@0.1.2
  │ └── charm@0.1.2
  ├─┬ pmx@1.6.4
  │ ├── deep-metrics@0.0.1
  │ ├── json-stringify-safe@5.0.1
  │ └─┬ vxx@1.2.2
  │   ├─┬ continuation-local-storage@3.2.1
  │   │ ├── async-listener@0.6.9
  │   │ └── emitter-listener@1.1.1
  │   ├── debug@2.6.9
  │   ├── extend@3.0.1
  │   ├── is@3.2.1
  │   ├── lodash.findindex@4.6.0
  │   ├── lodash.isequal@4.5.0
  │   ├── lodash.merge@4.6.1
  │   ├── methods@1.1.2
  │   ├── shimmer@1.2.0
  │   └── uuid@3.2.1
  ├─┬ promptly@2.2.0
  │ └─┬ read@1.0.7
  │   └── mute-stream@0.0.7
  ├── semver@5.5.0
  ├─┬ shelljs@0.7.8
  │ ├─┬ glob@7.1.2
  │ │ ├── fs.realpath@1.0.0
  │ │ ├─┬ inflight@1.0.6
  │ │ │ └── wrappy@1.0.2
  │ │ └── once@1.4.0
  │ ├── interpret@1.1.0
  │ └─┬ rechoir@0.6.2
  │   └─┬ resolve@1.7.0
  │     └── path-parse@1.0.5
  ├─┬ source-map-support@0.5.4
  │ └── source-map@0.6.1
  ├── sprintf-js@1.1.1
  ├── v8-compile-cache@1.1.2
  ├─┬ vizion@0.2.13
  │ └── async@1.5.2
  └─┬ yamljs@0.3.0
    └─┬ argparse@1.0.10
      └── sprintf-js@1.0.3

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

npm install pm2 -g

/usr/bin/pm2 -> /usr/lib/node_modules/pm2/bin/pm2

/usr/bin/pm2-dev -> /usr/lib/node_modules/pm2/bin/pm2-dev

/usr/bin/pm2-docker -> /usr/lib/node_modules/pm2/bin/pm2-docker

/usr/bin/pm2-runtime -> /usr/lib/node_modules/pm2/bin/pm2-runtime

/usr/lib

└─┬ pm2@2.10.2

├─┬ async@2.6.0

│ └── lodash@4.17.5

├── blessed@0.1.81

├─┬ chalk@1.1.3

│ ├── ansi-styles@2.2.1

│ ├── escape-string-regexp@1.0.5

│ ├─┬ has-ansi@2.0.0

│ │ └── ansi-regex@2.1.1

│ ├── strip-ansi@3.0.1

│ └── supports-color@2.0.0

├─┬ chokidar@2.0.3

│ ├─┬ anymatch@2.0.0

│ │ └─┬ micromatch@3.1.10

│ │ ├── arr-diff@4.0.0

│ │ ├─┬ define-property@2.0.2

│ │ │ └─┬ is-descriptor@1.0.2

│ │ │ ├── is-accessor-descriptor@1.0.0

│ │ │ └── is-data-descriptor@1.0.0

│ │ ├─┬ extend-shallow@3.0.2

│ │ │ ├── assign-symbols@1.0.0

│ │ │ └─┬ is-extendable@1.0.1

│ │ │ └── is-plain-object@2.0.4

│ │ ├─┬ extglob@2.0.4

│ │ │ ├─┬ define-property@1.0.0

│ │ │ │ └─┬ is-descriptor@1.0.2

│ │ │ │ ├── is-accessor-descriptor@1.0.0

│ │ │ │ └── is-data-descriptor@1.0.0

│ │ │ ├─┬ expand-brackets@2.1.4

│ │ │ │ ├── debug@2.6.9

│ │ │ │ ├── define-property@0.2.5

│ │ │ │ ├── extend-shallow@2.0.1

│ │ │ │ └── posix-character-classes@0.1.1

│ │ │ └── extend-shallow@2.0.1

│ │ ├── fragment-cache@0.2.1

│ │ ├── kind-of@6.0.2

│ │ ├─┬ nanomatch@1.2.9

│ │ │ ├─┬ is-odd@2.0.0

│ │ │ │ └── is-number@4.0.0

│ │ │ └── is-windows@1.0.2

│ │ ├── object.pick@1.3.0

│ │ └── regex-not@1.0.2

│ ├── async-each@1.0.1

│ ├─┬ braces@2.3.2

│ │ ├── arr-flatten@1.1.0

│ │ ├── array-unique@0.3.2

│ │ ├─┬ extend-shallow@2.0.1

│ │ │ └── is-extendable@0.1.1

│ │ ├─┬ fill-range@4.0.0

│ │ │ ├── extend-shallow@2.0.1

│ │ │ ├─┬ is-number@3.0.0

│ │ │ │ └─┬ kind-of@3.2.2

│ │ │ │ └── is-buffer@1.1.6

│ │ │ ├── repeat-string@1.6.1

│ │ │ └── to-regex-range@2.1.1

│ │ ├── isobject@3.0.1

│ │ ├── repeat-element@1.1.2

│ │ ├─┬ snapdragon@0.8.2

│ │ │ ├─┬ base@0.11.2

│ │ │ │ ├─┬ cache-base@1.0.1

│ │ │ │ │ ├─┬ collection-visit@1.0.0

│ │ │ │ │ │ ├── map-visit@1.0.0

│ │ │ │ │ │ └── object-visit@1.0.1

│ │ │ │ │ ├── get-value@2.0.6

│ │ │ │ │ ├─┬ has-value@1.0.0

│ │ │ │ │ │ └─┬ has-values@1.0.0

│ │ │ │ │ │ └── kind-of@4.0.0

│ │ │ │ │ ├─┬ set-value@2.0.0

│ │ │ │ │ │ └── extend-shallow@2.0.1

│ │ │ │ │ ├─┬ to-object-path@0.3.0

│ │ │ │ │ │ └── kind-of@3.2.2

│ │ │ │ │ ├─┬ union-value@1.0.0

│ │ │ │ │ │ └─┬ set-value@0.4.3

│ │ │ │ │ │ └── extend-shallow@2.0.1

│ │ │ │ │ └─┬ unset-value@1.0.0

│ │ │ │ │ └─┬ has-value@0.3.1

│ │ │ │ │ ├── has-values@0.1.4

│ │ │ │ │ └── isobject@2.1.0

│ │ │ │ ├─┬ class-utils@0.3.6

│ │ │ │ │ ├── arr-union@3.1.0

│ │ │ │ │ ├── define-property@0.2.5

│ │ │ │ │ └─┬ static-extend@0.1.2

│ │ │ │ │ ├── define-property@0.2.5

│ │ │ │ │ └─┬ object-copy@0.1.0

│ │ │ │ │ ├── copy-descriptor@0.1.1

│ │ │ │ │ ├── define-property@0.2.5

│ │ │ │ │ └── kind-of@3.2.2

│ │ │ │ ├── component-emitter@1.2.1

│ │ │ │ ├─┬ define-property@1.0.0

│ │ │ │ │ └─┬ is-descriptor@1.0.2

│ │ │ │ │ ├── is-accessor-descriptor@1.0.0

│ │ │ │ │ └── is-data-descriptor@1.0.0

│ │ │ │ ├─┬ mixin-deep@1.3.1

│ │ │ │ │ ├── for-in@1.0.2

│ │ │ │ │ └── is-extendable@1.0.1

│ │ │ │ └── pascalcase@0.1.1

│ │ │ ├── debug@2.6.9

│ │ │ ├─┬ define-property@0.2.5

│ │ │ │ └─┬ is-descriptor@0.1.6

│ │ │ │ ├─┬ is-accessor-descriptor@0.1.6

│ │ │ │ │ └── kind-of@3.2.2

│ │ │ │ ├─┬ is-data-descriptor@0.1.4

│ │ │ │ │ └── kind-of@3.2.2

│ │ │ │ └── kind-of@5.1.0

│ │ │ ├── extend-shallow@2.0.1

│ │ │ ├── map-cache@0.2.2

│ │ │ ├── source-map@0.5.7

│ │ │ ├─┬ source-map-resolve@0.5.1

│ │ │ │ ├── atob@2.1.0

│ │ │ │ ├── decode-uri-component@0.2.0

│ │ │ │ ├── resolve-url@0.2.1

│ │ │ │ ├── source-map-url@0.4.0

│ │ │ │ └── urix@0.1.0

│ │ │ └── use@3.1.0

│ │ ├─┬ snapdragon-node@2.1.1

│ │ │ ├─┬ define-property@1.0.0

│ │ │ │ └─┬ is-descriptor@1.0.2

│ │ │ │ ├── is-accessor-descriptor@1.0.0

│ │ │ │ └── is-data-descriptor@1.0.0

│ │ │ └─┬ snapdragon-util@3.0.1

│ │ │ └── kind-of@3.2.2

│ │ ├── split-string@3.1.0

│ │ └─┬ to-regex@3.0.2

│ │ └─┬ safe-regex@1.1.0

│ │ └── ret@0.1.15

│ ├─┬ glob-parent@3.1.0

│ │ ├── is-glob@3.1.0

│ │ └── path-dirname@1.0.2

│ ├── inherits@2.0.3

│ ├─┬ is-binary-path@1.0.1

│ │ └── binary-extensions@1.11.0

│ ├─┬ is-glob@4.0.0

│ │ └── is-extglob@2.1.1

│ ├─┬ normalize-path@2.1.1

│ │ └── remove-trailing-separator@1.1.0

│ ├── path-is-absolute@1.0.1

│ ├─┬ readdirp@2.1.0

│ │ ├── graceful-fs@4.1.11

│ │ ├─┬ minimatch@3.0.4

│ │ │ └─┬ brace-expansion@1.1.11

│ │ │ ├── balanced-match@1.0.0

│ │ │ └── concat-map@0.0.1

│ │ ├─┬ readable-stream@2.3.6

│ │ │ ├── core-util-is@1.0.2

│ │ │ ├── isarray@1.0.0

│ │ │ ├── process-nextick-args@2.0.0

│ │ │ ├── safe-buffer@5.1.1

│ │ │ ├── string_decoder@1.1.1

│ │ │ └── util-deprecate@1.0.2

│ │ └── set-immediate-shim@1.0.1

│ └── upath@1.0.4

├── cli-table-redemption@1.0.1

├── commander@2.13.0

├─┬ cron@1.3.0

│ └── moment-timezone@0.5.14

├─┬ debug@3.1.0

│ └── ms@2.0.0

├── eventemitter2@1.0.5

├── fclone@1.0.11

├── gkt@1.0.0

├─┬ mkdirp@0.5.1

│ └── minimist@0.0.8

├── moment@2.22.0

├─┬ needle@2.2.0

│ ├── debug@2.6.9

│ ├─┬ iconv-lite@0.4.21

│ │ └── safer-buffer@2.1.2

│ └── sax@1.2.4

├─┬ nssocket@0.6.0

│ ├── eventemitter2@0.4.14

│ └── lazy@1.0.11

├── pidusage@1.2.0

├─┬ pm2-axon@3.1.0

│ ├── amp@0.3.1

│ ├── amp-message@0.1.2

│ └── escape-regexp@0.0.1

├── pm2-axon-rpc@0.5.0

├─┬ pm2-deploy@0.3.9

│ ├── async@1.5.2

│ └── tv4@1.3.0

├─┬ pm2-multimeter@0.1.2

│ └── charm@0.1.2

├─┬ pmx@1.6.4

│ ├── deep-metrics@0.0.1

│ ├── json-stringify-safe@5.0.1

│ └─┬ vxx@1.2.2

│ ├─┬ continuation-local-storage@3.2.1

│ │ ├── async-listener@0.6.9

│ │ └── emitter-listener@1.1.1

│ ├── debug@2.6.9

│ ├── extend@3.0.1

│ ├── is@3.2.1

│ ├── lodash.findindex@4.6.0

│ ├── lodash.isequal@4.5.0

│ ├── lodash.merge@4.6.1

│ ├── methods@1.1.2

│ ├── shimmer@1.2.0

│ └── uuid@3.2.1

├─┬ promptly@2.2.0

│ └─┬ read@1.0.7

│ └── mute-stream@0.0.7

├── semver@5.5.0

├─┬ shelljs@0.7.8

│ ├─┬ glob@7.1.2

│ │ ├── fs.realpath@1.0.0

│ │ ├─┬ inflight@1.0.6

│ │ │ └── wrappy@1.0.2

│ │ └── once@1.4.0

│ ├── interpret@1.1.0

│ └─┬ rechoir@0.6.2

│ └─┬ resolve@1.7.0

│ └── path-parse@1.0.5

├─┬ source-map-support@0.5.4

│ └── source-map@0.6.1

├── sprintf-js@1.1.1

├── v8-compile-cache@1.1.2

├─┬ vizion@0.2.13

│ └── async@1.5.2

└─┬ yamljs@0.3.0

└─┬ argparse@1.0.10

└── sprintf-js@1.0.3

PM2 Pricing

PM2 appears to be for high-end apps but I am only using the free version or PM2 (thanks KeyMetrics)

Create a bucket for your node app

Click Generate New Bucket

Give the bucket a name etc.

You can now link your bucket with your local pm2 installation (keep the keys private (this one no longer exists))

Linking your local pm2 installation with your keymetrics bucket

pm2 link l3brztzboz25him i6kofelsyfo7xrd
[KM] Connecting
[Monitoring Enabled] Dashboard access: https://app.keymetrics.io/#/r/i6kofelsyfo7xrd

pm2 link l3brztzboz25him i6kofelsyfo7xrd

[KM] Connecting

[Monitoring Enabled] Dashboard access: https://app.keymetrics.io/#/r/i6kofelsyfo7xrd

To add an existing node app to PM2 type the following.

cd /your-node-application-path/
pm2 start yourapp.js -i 0 --name "myappname"

1 2	cd /your-node-application-path/ pm2 start yourapp.js -i 0 --name "myappname"

You can view node apps that pm2 is managing by typing

pm2 status

1	pm2 status

I had a two CPU VM and I found that the app I added was added to each of the two CPU (I only needed one) so I needed to delete the second app on my second core

pm2 delete 1

1	pm2 delete 1

Restart the API

pm2 restart myappname

1	pm2 restart myappname

You can add a single node apps one 1, 3 or max available CPU’s

# Start the maximum processes depending on available CPUs
pm2 start app.js -i 0

# Start the maximum processes -1 depending on available CPUs
pm2 start app.js -i -1

# Start 3 processes
pm2 start app.js -i 3

# Start the maximum processes depending on available CPUs

pm2 start app.js -i 0

# Start the maximum processes -1 depending on available CPUs

pm2 start app.js -i -1

# Start 3 processes

pm2 start app.js -i 3

Again, to add an existing node app to PM2 type the following.

cd /your-node-application-path/
pm2 start yourapp.js -i 0 --name "myappname"

1 2	cd /your-node-application-path/ pm2 start yourapp.js -i 0 --name "myappname"

Now you can view node app data online. If you don’t have a node app ready you can use the test app.

You can monitor your node app locally too from the CLI.

You can also view a demo bucket at keymetrix.io

PM2’s one age documentation can be found here.

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.0 Initial post

Using Cloudflare DNS servers to speed up the internet and add privacy on OSX

April 2, 2018 by Simon

Below is how I setup my OSX to use Cloudflare’s new DNS servers to speed up internet browsing and add privacy on OSX

Cloudflare has launched a DNS service: https://blog.cloudflare.com/announcing-1111/

DNS Performance

You can view worldwide DNS performance by viewing https://www.dnsperf.com/#!dns-providers

I check the DNS at my router, I am using ISP provided DNS servers.

Cloudflare DNS

On April Fools 2018 Cloudflare Released a DNS server service.

Snip from here: “DNS: Internet’s Directory Nearly everything on the Internet starts with a DNS request. DNS is the Internet’s directory. Click on a link, open an app, send an email and the first thing your device does is ask the directory: Where can I find this? Unfortunately, by default, DNS is usually slow and insecure. Your ISP, and anyone else listening in on the Internet, can see every site you visit and every app you use — even if their content is encrypted. Creepily, some DNS providers sell data about your Internet activity or use it target you with ads.”

https://1.1.1.1/

Set Cloudflare Nameservers using OSX

Open the Apple System Preferences, click Network, click on your Network (Wifi or ethernet), Click Advanced then DNS and add 1.1.1.1 and 1.0.0.1

Alternatively, you can manually set your DNS servers in OSX by editing the /etc/resolv.conf, by default SX will inherit DNS settings from our router.

cat /etc/resolv.conf
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
domain home
nameserver 1.1.1.1
nameserver 1.0.0.1

cat /etc/resolv.conf

# macOS Notice

# This file is not consulted for DNS hostname resolution, address

# resolution, or the DNS query routing mechanism used by most

# processes on this system.

# To view the DNS configuration used by this system, use:

# scutil --dns

# SEE ALSO

# dns-sd(1), scutil(8)

# This file is automatically generated.

domain home

nameserver 1.1.1.1

nameserver 1.0.0.1

Troubleshooting: Clear DNS Cache

sudo killall -HUP mDNSResponder

1	sudo killall -HUP mDNSResponder

Debug DNS Data

scutil --dns
DNS configuration

resolver #1
  search domain[0] : home
  nameserver[0] : 1.1.1.1
  nameserver[1] : 1.0.0.1
  flags    : Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : home
  nameserver[0] : 1.1.1.1
  nameserver[1] : 1.0.0.1
  if_index : 7 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

scutil --dns

DNS configuration

resolver #1

search domain[0] : home

nameserver[0] : 1.1.1.1

nameserver[1] : 1.0.0.1

flags : Request A records

reach : 0x00000002 (Reachable)

resolver #2

domain : local

options : mdns

timeout : 5

flags : Request A records

reach : 0x00000000 (Not Reachable)

order : 300000

resolver #3

domain : 254.169.in-addr.arpa

options : mdns

timeout : 5

flags : Request A records

reach : 0x00000000 (Not Reachable)

order : 300200

resolver #4

domain : 8.e.f.ip6.arpa

options : mdns

timeout : 5

flags : Request A records

reach : 0x00000000 (Not Reachable)

order : 300400

resolver #5

domain : 9.e.f.ip6.arpa

options : mdns

timeout : 5

flags : Request A records

reach : 0x00000000 (Not Reachable)

order : 300600

resolver #6

domain : a.e.f.ip6.arpa

options : mdns

timeout : 5

flags : Request A records

reach : 0x00000000 (Not Reachable)

order : 300800

resolver #7

domain : b.e.f.ip6.arpa

options : mdns

timeout : 5

flags : Request A records

reach : 0x00000000 (Not Reachable)

order : 301000

DNS configuration (for scoped queries)

resolver #1

search domain[0] : home

nameserver[0] : 1.1.1.1

nameserver[1] : 1.0.0.1

if_index : 7 (en0)

flags : Scoped, Request A records

reach : 0x00000002 (Reachable)

Confirm Cloudflare DNS from the OSX Comand line

nslookup www.fearby.com
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	www.fearby.com
Address: 104.27.154.69
Name:	www.fearby.com
Address: 104.27.155.69

nslookup www.fearby.com

Server: 1.1.1.1

Address: 1.1.1.1#53

Non-authoritative answer:

Name: www.fearby.com

Address: 104.27.154.69

Name: www.fearby.com

Address: 104.27.155.69

Privacy

I am not sure if Cloudflare is any more private than using ISP DNS but I’ll happily use it.

Several people have asked me about Cloudflare’s new 1.1.1.1 privacy DNS service. To be clear: it DOES NOT stop your ISPs from collecting your browsing history. ISPs can still see the sites you’re connecting to — even if the site is over HTTPS. You will still send a hostname.
— Zack Whittaker (@zackwhittaker) April 2, 2018

Speed

I can’t tell if DNS is faster, I did ping my ISP DNS before switching and it was about the same (sub 25ms), time will tell.

Conclusion

I have used https://www.opendns.com/ before and loved the dashboards, I hope Cloudflare add dashboard options too.

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.0 Initial post

How to purchase your own domain name and set up a web server from $5 a month

March 28, 2018 by Simon

This guide will show technically minded people how you can purchase your own domain name, set up a web server on Vultr with an online store using WordPress/WooCommerce from $5 a month. Warning this post is technical (if you have never used SSH, Ubuntu, Linux Command Line, hate risk or are not patient then this is NOT the guide you are after).

Draft Post (released early to share)

I personally recommend (not a paid endorsement) the free WooCommerce plugin for the free WordPress.org CMS on the free Ubuntu Operating system with the free NGINX web server and the free MYSQL database engine and free SSL certificates from Lets Encrypt.

Sorry for using the word free a lot but I like free things. One of the benefits of a using a self-managed server is you get the option to install free software and configure the server how you want and secure it how you want. Truth be told managed ho (e.g CPanel, etc) are in the business of making money via monthly feed, expensive SSL certificates, taxing your transactions or pushing you to higher priced tiers.

Legend:

Self Managed Server = A server that you create, you configure patch and support (all the reward and risk is owned by you and costs are low).
Hosted Server = A server you have partial control of and the hosts manage the server and support (You hand away all risk and most of the control and pay for support/features).

I moved to a self-managed server after I was paying $25/m for a poorly performing website and $150/y for a poor quality SSL certificate and a slice of a server that seemed to always say “Usage Limit Exceeded”. Why pay for an insecure website that my visitors could not view because the usage limit was exceeded.

fyi: Fearby.com costs me $10 a month for a server and $5/m for CDN abilities.

CPanel hosts are an option when you don’t want to self-manage a service and take on the hassle but be prepared for server limitations (The image below was taken on an older CPanel based hosts before I moved to a self-managed Vultr server)

I recently discovered a well known and established website hosting service (that I used to use) and a friend is still using is insecure. My friend’s site has a static website on it but the server underneath was very old and insecure. Having a secure web server should be at the top of your list with any self-managed or hosted website (this will help search engine optimization and prevent risks to your website visitors).

Sites like Virus Total, SSL Labs and Alexa Site Info , Qualys are good ways to review a site’s credibility.

fyi: The awesome https://seositecheckup.com/ is awesome for evaluating our sites SEO score.

Before we set up a server with WordPress on your own server let’s quickly look at the alternative commercial ready to go website builders.

Alternative (paid) DIY Website Builders

The following leading commercial sites will allow you to build a site online.

In my opinion, five things matter with setting up site online website.

Setup Costs, Monthly Cost and Commissions (what are the hidden charges)
Security (having a food SSL Certificate is key to having a good organic traffic from search engines)
Site Speed (Having a slow site will impact search engine optimization and drive visitors away)
Accessibility (if your site is not WCAG accessible it will not rank high on search engines).
Control (will you be able to do everything you want too, nothing worse than going so far and being limited)

Ok, let’s see how much it will cost to set up a simple business site on the sites above.

Wix

Setup: Goto https://www.wix.com/, Login to Wix, click Create Site, click Business, Click Choose a Template, Edit the page, Click Save, Click “Connect your own customized domain“, Click “Connect a domain you already own“.

I was redirected to a Wix plan pricing page where I need to choose a plan to continue. From what I researched you cant control HTML on Wix so can’t add a MailChimp newsletter signup form so you would have to go with the $24.5/m option to enable Email Campaigns.

I could not see information about included SSL certificates, SEO or other chargers. SSL is free after you pay right?

The Wix editor appears OK (it may take a bit of learning though).

I clicked publish and the site was live

A quick check of the SSL, Accessibility and SEO and no obvious deal breakers here apart from the price and platform lock-in.

I performed a security check on the site with https://freescan.qualys.com (passed)

Conclusion: I hear Wix templates are hard to change so choose your template wisely, A large collection of apps are available that you can add to the site.

Although Wix was nice and it does include a full-featured look at the engine it is not for me ($24/m USD is too expensive).

Squarespace

Squarespace basic websites cost $16/$25 a month or $34/52 for online stores: https://www.squarespace.com/pricing/

Setup a Squarespace website: Goto https://www.squarespace.com/, Click Start a Free Trial, Choose a Template, Create an Account (a quick read of the terms of service and privacy policy, #scary), SpareSpace sites are pre-published?

Loading the webpage on a non-logged-in (with SquareSpace login) browser displays a trial warning. Trial pages are essentially restricted (unlike Wix).

The mobile view does not match the template? I guess the chosen template is more of a vibe and not a template.

Setting up a Squarespace website may take some time. Squarespace does have some nifty advance options in a slide-out menu though.

Because the public view of the page is restricted I cannot scan it with WCAG accessibility tools. Scanning the site performance speeds with gtmetrix also fails.

Squarespace is well known to be difficult to set up a website when compared to other drag and drop editors (but Squarespace sites do look nice).

I am not paying $54/m for a website so let’s move on.

Shopify

Shopify Setup: Goto https://shopify.com and click Create, Sign up and enter your store name. Complete the wizard.

Choose a Shopify Plan

Scalping transactions, no thanks. let’s move on.

Weebly

Weebly Setup: Goto https://www.weebly.com/au and click Get Started under Create Store. Enter your account details and click Create Your Site, enter the name of the store, Click I’m just trying Weebly, click the type of product you will be selling.

Weebly Site Setup

Theme Selection

Choose a Domain

Publish the site

Clicking publish appears to be a dead end.

“Please contact Weebly Support to verify your account”, No Thanks, let’s move on.

One candidate remains and that is WordPress hosted (wordpress.com not wordpress.org).

WordPress.com

WordPress.com offer hosted plans for WordPress in the cloud.

Setup a WordPress site, the only one that removes WordPress branding and allows third-party plugins to be installed it the Business plans for $33 a month.

Setup Basics

Choose a WordPress theme.

Assign a Domain

In order to buy a domain, you need to log in (top right) with an account

My working WordPress account (is no longer working), it was in my password manager.

I seem to be stuck in a signup loop

Time to move on. Time to set up my own server on Vultr and setup WordPress and WooCommerce,

But, before we do, let’s ensure our name is secure online.

Search for your Name/Brand

Do search for your website (or thing) in search engines to see if your name is already taken, don’t buy a domain that is owned or has IP or trademark presence. It is a good idea to use sites like https://namechk.com/ to see if your site or social media is already taken.

namechk.com will allow you to search for name availability online. The name “mything” is not fully available online.

You will want to see all green squares (name available) below before buying a domain name. This looks better.

I would recommend you create your social media accounts before or right after buying your domain. Sites like Twitter will insist on short usernames names so get your social media sites first.

Trademark and Brand Search

Also, perform a trademark and IP search.

Australian Trademark Search: https://search.ipaustralia.gov.au/trademarks/search/quick

United States Trademark Database: https://www.uspto.gov/trademarks-application-process/search-trademark-database

Global brand Search: http://www.wipo.int/branddb/en/

etc

Self Managed Warning

I tend to go the “self-managed server route” and install the free WordPress CMS because:

I can.
I am tight.
I like having full control (usually the best features for online web hosts are hidden behind subscriber tiers, you can install and do whatever you want on your own server like build API’s, distributed MySQL servers, install MongoDB or Redis , use up to date PHP etc).
I have been stung by CPanel hosts charging $150/y for a crappy SSL certificate (You can set up your own SSL certificate for $0 and set up super secure SSL rules).
I can manage WordPress via the command line
I can upgrade the server and restore it whenever I want.
I can manage my own server performance (e.g setup PHP child workers) or install a Content Delivery Network.
I can direct domain email to google G Suite, see pricing here.
etc.

There are many reasons why you would not want to “self-manage” your own server

Technical Requirements (and time to support).
Higher Risk.
Applying Updates and Patches.
etc.

Being technically minded and choosing a “self-managed web servers” can take away time from the fun stuff like SEO, Site Design, customer needs, branding etc.

Self Managed Costs

For $5 a month you can buy a server with enough memory to install WordPress (cheaper if you don’t need WordPress)

Vultr is great. Vultr does have ready to go servers that you can deploy that have WordPress all set up.

The Vultr template above does use the Centos OS (read my guide setting up Centos on a different service provider here) but I prefer to manually setup a server with Ubuntu 16.04 OS on Vultr.

Wih a $5 server you can do what you want with it. I have blogged before about setting up your own Server. e.g Installing Centos and Ubuntu server on Digital Ocean. Digital Ocean does not have data centres in Australia and this kills scalability. AWS is good but 4x the price of Vultr. I have blogged about setting up and AWS server here (and upgrading an AWS instance). I tried to check out Alibaba Cloud but the verification process was broken so I decided to check our Vultr.

Manual Setup of Vultr on an Ubuntu 16.04 server

Deploy a Vultr Server – Guide here ($2.5/m to New Jersey or Florida or $5/m to Sydney, I would recommend you opt in for the auto backup for $0.50c/m and $1/m respectively).
Setup NGINX.
Setup PHP and PHP-FPM (see guide above), consider adding PHP child workers.
Setup and secure MySQL (see guide above), create a database for WordPress to use.
Instal Adminder MySQL GUI (guide here).
Setup a free Lets Encrypt SSL certificate (guide here).
Install WordPress (and Jetpack plugin).
Install WordPress CLI.
Instal the WooCommerce Storefront WordPress Theme.
Install WooCommerce Plugin.
Secure Ubuntu.
Also consider linking your domain to Cloudflare to boost performance, scanning your site with Qualys Freescan and OWASP ZAP).
Consider setting up a WordPress image compressor and CDN plugin. like EWWW.io

Manual WooCommerce Plugin Setup

Once you setup Woocommerce you can setup the store defaults. Goto the WordPress dashboard and click WooCommerce Settings

Settings – General

Set Address, City and State and Postcode
Set allowed countries to sell in (e.g Australia)
Set allowed countries to ship items to (e.g Australia)
Set Enable Taxes
Set Currency
etc

Settings – Products

Set Weight
Set Dimensions
Enable Product Reviews
Enable Star Ratings on Reviews
etc

Settings – Shipping

Enable Shipping Calculator
Add Shipping Classes
Shipping Zones
etc.

Settings – Checkout

Force Secure Checkup
Create a Terms and Conditions page (and set).
etc

Settings – Account

Set Account Options
etc

Settings – Emails

Set Email Preferences
Set Email Header Image
Set Email Colour
Set Footer Text
etc

Settings – API

API can be disabled if you don’t need it.

Optional Actions

Setup Yoast Plugin
Setup other plugins

Instaling a Woo Commerce Child Theme

Goto https://woocommerce.com/product-category/themes/storefront-child-theme-themes/ and choose a theme.

Purchase and Install the desired child theme (I uploaded it to my /wp-content/themes/ folder with forklift). I chose a free deli theme.

Goto your WordPress then themes folder and activate your new child theme.

Post Site Setup

Just because your site is live does not mean you can rest.

SEO Optimization

Do use sites like https://seositecheckup.com/ and follow recommended actions to improve your SEO like updating meta tags.

More Reading

Attaching an email to your domain

You can pay $5 a month and link a G Suite email to your domain.

Dedicated professional Google G Suite email account for $5 a month with 30GB storage (If you don’t want ot to buy a G Suite email and link it to your domain then you don’t need this).

Once you have a G Suite account you can link other domains (and domain emails) to it. You can login to your G Suite emails via G Mail and send emails from apps or the command line.

Why Vultr

I use the server host Vultr as they have data centres all around the world and the support of great, Digital Ocean is good too but they don’t have data centres in my country (Australia). Vultr allows you to deploy all over the world upgrade servers, move servers, add storage and restore servers.

Alternatively, you can buy a $2.5/m server and generate a static website

I use the Platforma Web HTML generator to build mobile and WCAG compliant websites.

Buying a domain, I buy my domains from https://www.namecheap.com/ it is a good idea to look for coupons first at https://www.namecheap.com/promos/coupons.aspx before buying a domain.

Once you buy a domain you can point it to a Vultr server and upload your website.

I hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

Revision History

v.1.2 WordPress WooCommerce

v1.1 SEO

v1.0 Initial Draft

Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap

March 13, 2018 by Simon

This guide will show how you can set up a website to use Cloudflare on a VM hosted on Vultr and Namecheap

I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. This post will show how to let Cloudflare handle the DNS for the domain.

Snip from here “Cloudflare’s enterprise-class web application firewall (WAF) protects your Internet property from common vulnerabilities like SQL injection attacks, cross-site scripting, and cross-site forgery requests with no changes to your existing infrastructure.”

Cloudflare Benefits (Free Plan)

DDoS Attack Protection (Huge network to absorb attacks DDoS attacks over 600Gbps are no problem for our 15 Tbps networks)
Global CDN
Shared SSL certificate (I disabled this and opted to use my own)
Access to audit logs
3 page rules (maximum)

View paid plan options here.

Cloudflare CDN map

Cloudflare CDN says it can load assets up to 2x faster, 60% less bandwidth from your servers by delivering assets from 127 data centres.

Setup

You will need to sign up at cloudflare.com

After you create an account you will be prompted to add a siteCloudflare will pull your public DNS records to import.

You will be prompted to select a plan (I selected free)

Verify DNS settings to import.

You will now be asked to change your DNS nameservers with your domain reseller

TIP: If you have an SSL cert (e.g Lets Encrypt) already setup head to the crypto section and select ” Full (Strict)” to prevent ERR_TOO_MANY_REDIRECTS errors.

Cloudflare UI

I asked Twitter if they could kindly load my site so I could see if Cloudflare dashboard/stats were loading.

Could I kindly ask if you are reading this that you visit https://t.co/9x5TFARLCt, I am writing a @Cloudflare blog post and need to screenshot stats. Thanks in advance
— Simon Fearby (Developer) (@FearbySoftware) March 13, 2018

The Cloudflare CTO responded. 🙂

Sure thing 🙂
— John Graham-Cumming (@jgrahamc) March 13, 2018

Confirm Cloudflare link to a domain from the OSX Comand line

host -t NS fearby.com
fearby.com name server dane.ns.cloudflare.com.
fearby.com name server nora.ns.cloudflare.com.

host -t NS fearby.com

fearby.com name server dane.ns.cloudflare.com.

fearby.com name server nora.ns.cloudflare.com.

Caching Rule

I set up the following caching rule to cache everything for 8 hours instead of WordPress pages

“fearby.com.com/wp-*” Cache level: Bypass
“fearby.com.com/wp-admin/post.php*” Cache level: Bypass
“fearby.com/*” Cache Everything, Edge Cache TTL: 8 Hours

Cache Results

Cache appears to be sitting at 50% after 12 hours. having cache os dynamic pages out there is ok unless I need to fix a typo, then I need to login to Cloudflare and clear the cache manually (or wait 8 hours)

Performance after a few hours

DNS times in gtmetrix have now fallen to a sub 200ms (Y Slow is now a respectable A, it was a C before). I just need to wait for caching and minification to kick in.

webpagetest.org results are awesome

See here: https://www.webpagetest.org/result/180314_PB_7660dfbe65d56b94a60d7a604ca250b3/

Load Time: 1.80s
First Byte 0.176s
Start Render 1.200s

Google Page Speed Insights Report

Mobile: 78/100

Desktop: 87/100

Check with https://developers.google.com/speed/pagespeed/insights/

Update 24th March 2018 Attacked?

I noticed a spike in and traffic (incoming and threats) on the 24th of March 2018.

I logged into Cloudflare on my mobile device and turned on Under Attack Mode.

Cloudflare was now adding a delay screen in the middle of my initial page load. Read more here. A few hours after the Attach started it was over.

After the Attack

I looked at the bandwidth and found no increase in traffic from my initial host VM. Nice.

Thanks, Cloudflare.

Cloudflare Pros

Enabling Attack mode was simple.
Soaked up an attack.
Free Tier
Many Reports
Option to force HTTPS over HTTP
Option to ban/challenge suspicious IP’s and set challenge timeframes.
Ability to setup IP firewall rules and Application Firewalls.
User-agent blocking
Lockdown URL’s to IP’s (pro feature)
Option to minify Javascript, CSS and HTML
Option to accelerate mobile links
Brotli compression on assets served.
Optio to enable BETA Rocket loader for Javascript performance tweaks.
Run Javascript service workers from the 120+ CDN’s
Page/URL rules o perform custom actions (redirects, skip cache, Encryption etc)
HTTP/2 on, IPV6 ON
Option to setup load balancing/failover
CTO of Cloudflare responded in Twitter 🙂
Option to enable rate limiting (charged at 10,000 hits for $0.05c)
Option to block countries (pro feature)
Option to install apps in Cloudflare like(Goole Analytics,

Cloudflare Cons

No more logging into NameCheap to perform DNS management (I now goto Cloudflare, Namecheap are awesome).
Cloudflare Support was slow/confusing (I ended up figuring out the redirect problem myself).
Some sort of verify Cloudflare Setup/DNS/CDN access would be nice. After I set this up my gtmetrix load times were the same and I was not sure if DNS needs to replicate? Changing minify settings in Cloudflare did not seem to happen.
WordPress draft posts are being cached even though page riles block wp-admin page caching.
Would be nice to have ad automatic Under Attack mode
Now all sub-domains were transferred in the setup ( id did not know for weeks)

Cloudflare status

Check out https://www.cloudflarestatus.com/ for status updates.

Don’t forget to install the CloudFlare Plugin for WordPress if you use WordPress.

More Reading

Check out my OWASP Zap and Kali Linux self-application Penetration testing posts.

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.8 host Command from the OSX CLI

v1.7 Subdomain error

v1.6 Cloudflare Attack

v1.5 WordPress Plugin

v1.4 More Reading

v1.3 added WAF snip

v1.2 Added Google Page Speed Insights and webpage rest results

v1.1 Added Y-Slow

v1.0 Initial post

Personal Development Blog

Coding for fun since 1996, Learn by doing and sharing.

Create your own server on UpCloud here

Follow me on Twitter: @FearbySoftware

View My Old Hobby /blog/

View All Posts.

and

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 3 of 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4

Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare

Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare

Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

Using Cloudflare DNS servers to speed up the internet and add privacy on OSX

How to purchase your own domain name and set up a web server from $5 a month

Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap

Popular

Security

Code

Tech

Wordpress

General

and

Footer

Popular

Security

Code

Tech

Wordpress

General