Code, InfoSe and Server Stuff

Views are personal and not my employers.

Personal Development Blog

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

Follow me on Twitter: @FearbySoftware

View My Old Hobby /blog/

View All Posts.

cpu

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

by

Before UpCloud, when I had 512MB ram on Vultr I had multiple NGINX crashed a day so I used a bash script and scheduled a cron job to clear memory cache when memory fell below 100MB (view the script here).  To further increase the speed of the WordPress I have configured the OS to use memory over the disk.  About once a day free memory falls below 100MB (this is not a problem as my script clears cached items automatically).

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

by

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

Advertisement:



If you have not read my previous posts I have now moved my blog from Vultr to the awesome UpCloud host (signup using this link to get $25 free credit).

Recently I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Spoiler: UpCloud performance is great.

Upcloud Site Speed in GTMetrix

Why Upgrade

I have 1 CPU, 1 GB memory and 50GB storage and it is running well?  I have PHP child workers set up and have set up the preferred use of memory over swap file usage.

View of htop querying processes on Ubuntu

Before UpCloud, when I had 512MB ram on Vultr I had multiple NGINX crashed a day so I used a bash script and scheduled a cron job to clear memory cache when memory fell below 100MB (view the script here).  To further increase the speed of the WordPress I have configured the OS to use memory over the disk.  About once a day free memory falls below 100MB (this is not a problem as my script clears cached items automatically).

Graph of memory falling below 100MB every day

I’d like to add more memory as I am working on some things (watch this space) and I will use the extra memory. I’d prefer the server is set up now for the expected workload.

How to Upgrade

This is how I upgraded from 1xCPU/2GB Memory/50GB Storage/2TB Transferred Data to a 2 CPU/4GB Mmeory/80GB Storage/4TB Transferred Data server.

UpCloud Pricing: https://www.upcloud.com/pricing/

Pricing table form https://www.upcloud.com/pricing/

Upgrade an UpCloud VM

I shut down my existing VM. Read this guide to setup a VM.

Login to the UpCloud dashboard, select your server (confirm the server has shut down) and click General Settings, choose the upgrade and click Update.

Upgrade the Server, shut it down the server and choose upgrade

I confirmed the upgrade options (2x CPU, 4096 MB Memory).

Confirm Upgrade Options

Click Update

Upgrade Applied

After 10 seconds you can start your server from the UpCloud server.

I confirmed the CPU upgrade was visible in the VM

Software Tweaks Post Upgrade.

I added these settings to the top of /etc/nginx/nginx.conf to ensure the extra CPU was used.

I increased PHP FPM ( /etc/php/7.2/fpm/php.ini ) to increase the memory usage and child workers. I doubled child workers and max memeory limit.

I restarted NGINX and PHP

I tweaked WordPress max memory limits

MySQL Tweaks: I logged into MySQL

I ran “SHOW GLOBAL STATUS” to view stats

Read more on SHOW GLOBAL STATUS here. Read more on the values here.

I can see NO major errors here (possibly due to UpClouds awesome disk IO) so I won’t be making memory tweaks in MySQL. Sign Up using this link and get $25 credit free on UpCloud and see for yourself how fast they are.

Configure Ubuntu System Memory Usage

Edit /etc/sysctl.conf

Add the following to allow things to sit in ram longer

Snip from: https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Read these pages here and here regarding setting MySQL memory.

Reboot

Resize the disk

The 2x CPU, 4GB memory plan comes with 80GB storage allowance.  My disk at present is 50GB and I will update the size soon following this guide soon.

Upgrade disk from 50gb to 80GB soon

Quick Benchmark

I used loader.io to load 500 users to access my site in 1 minute.

HTOP showing 2x busy CPU's running at 60%

The benchmark worked with no errors.

Loader.io Success with 500 concurrent users

This benchmark was performed with no Cloudflare caching. I should get Cloudflare caching working again to lower the average response time. I loaded my website manually in Google Chrome while loader.io threw 500 users at my site and it loaded very fast.

Conclusion

After a few days, I checked my memory logs and there were no low memory triggers (just normal internal memory management triggers). Ubuntu was happier.

No Low memory low triggers

This graph was taken before I set “vm.vfs_cache_pressure” so I will update this graph in a few days.

I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

1.0 Initial Post

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

by

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 4 of 4

Advertisement:



Read Part 1, Part 2, Part 3 or Part 4

I ran the MySQL benchmark preparation command again (no problem this time).

Test table and records created

Test Records Created

Now I can benchmark MySQL on my main server.

RAW Output

Results

queries performed (in 60 seconds):

  • read: 336210
  • other: 48030
  • total: 384240

I decided to add an index to see if I can speed this query up (read the MySQL index page here). I added an index (in Adminer) on the columns “Id” and “pad” for the sbtest table in the test database

I restarted the MySQL process

I ran the same benchmark again.

Raw Output

Results

The quick index added 20% extra throughput on queries 🙂

Mysql before and after an index

Don’t forget to delete your test database

Viewing MySQL Index Usage (on the “test” database)

Query to show Index stats for a table ‘test’

I can see the MySQL PRIMARY index is getting used 🙂

Index Summary

Read more in viewable query stats (columns) here.

Other System Information Tools

Show processor information

Output

Memory Information

You can assign 512MB, 1GB, 2GB or more memory to a server on Vultr, Read my guide here on upgrading resources for Vultr VM’s here.

Only upgrade your server’s memory when server processes demand it, there is no need to pay for extra idle memory. Read my older guides on upgrading Digital Ocean and AWS servers.

I use the htop utility to monitor memory and processes. The memory usage will depend on how you have configured your server to use connection pools in code, MySQL or services.  Also what memory demands do you get in pean bandwidth times?

HTOP

You can check your server memory details on Ubuntu with this command

Output

Use Memory or Disk (Swap)

You can configure the use of Memory over Disk by configuring your/etc/sysctl.conf file (setting value “vm.swappiness”)

You can check your swap file settings by running the following command

Or By running

Set a new swap file value by editing /etc/sysctl.conf

Set the following to use more ram over the swap disk.

Read about swappiness values here: https://en.wikipedia.org/wiki/Swappiness

Service Performance

Performance (and allocated resources) depends on the demands of your operating system and installed software

What operating system do you have?

View NGINX Status, how much memory does it use?

PHP (and Child Worker) status how much memory does it use and how many child workers do you have? Read my add PHP child workers post here (and update to PHP 7.2 here)

MySQL Status

Shared VM Hosts

One of the biggest impacts (after server latency) for your server is not the disk performance but the number of hosts/websites on the server who are also using the disk and server resources.

Reverse IP Lookup

I have 80 other web servers on my server (based on a reverse lookup).

I may move to a dedicated box when I can afford it.

Security

Above all else ensure that security is number 1 priority and make performance second priority.

Scan your site with Zap, Qualys and Kali Linux. Performance means nothing if you are hacked.

website-report

Simulated concurrent users

You can use Siege to test the maximum concurrent users accessing your site before the server starts to drop connections.

FYI: If you use Cloudflare (you should) this may not work as it will block connections.

Install Siege

Test  your server with 10 concurrent serves for 1 minute

Results

Keep upping the connections (from 10 above) to a limit where connections start dropping.

I tried 25 then 50 concurrent users hitting a server on Digital Ocean and it did not fail.

Conclusion

  • Choose a server near your customers
  • Change hosts if one is faster and cheaper
  • Measure or benchmark your server (and compare over time).
  • Use Cloudflare

Create your own server today

  • Create your own server on Vultr here.
  • Create your own server on Digital Ocean here.
  • Create your own server on UpCloud here.

And remember you can install the Runcloud server management dashboard here.

I hope this guide helps someone.

 

< Previous

Read Part 1, Part 2, Part 3 or Part 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 3 of 4

by

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 3 of 4

Advertisement:



Read Part 1, Part 2, Part 3 or Part 4

I used these commands to generate bonnie++ reports from the data in part 2

Image of results here

Bonnie Results

Network Performace

IMHO Network Latency is the biggest impact on server performance, Read my old post on scalability on a budget here. I am in Australia an having a server in Singapore was too far away and latency was terrible.

Here is a non-scientific example of pinging a Vultr, Digital Ocean and UpCloud server in three different locations (and Google).

Ping Test

Test Ping Results

Latency matters, run a https://www.webpagetest.org/ scan over your site to see why.

Adding https added almost 0.7 seconds to https communications in the past on Digital Ocean (a few thousand kilometres away). The longer the latency the longer HTTPS handshakes take.

SSL

Deploying a server to Singapore (in my experience) is bad if your visitors are in Australia. But deploying to other regions may be lower in cost though. It’s a trade off.

Server Location

Deploy servers as close as you can to your customers is the best tip for performance.

Deploy serves close to your customers

Also, consider setting up Image Optimization and Image CDN plugins (guide here) in WordPress and using Cloudflare (guide here)

Benchmarking with SysBench

Install CPU Benchmark

CPU Benchmark (Vultr/Sydney)

Result

39.15 seconds

CPU Benchmark (Digital Ocean/London)

Result

33.43 sec

CPU Benchmark (UpCloud/Singapore)

Result

23.77 sec

Surprisingly, 1st place in prime generation goes to UpCloud, then Digital Ocean then Vultr.  UpCloud has some good processors.

Processors:

  • UpCLoud (Singapore): Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz
  • Digital Ocean (London): Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
  • Vultr (Sydney): Virtual CPU a7769a6388d5 (Masked/Hidden CPU @ 2.40GHz)

(Lower is better)

prime benchmark results

(oops, typo in the chart should say Vultr)

Benchmark the file IO

Confirm free space

Install Sysbench

I had 10GB free on all servers (Vultr, Digitial Ocean and UpCloud) so I created a 10GB test file.

Now I can run the benchmark and use the pre-created text file.

SysBench description from the Ubuntu manpage.

“SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. The idea of this benchmark suite is to quickly get an impression about system performance without setting up complex database benchmarks or even without installing a database at all.”

SysBench Results (Vultr/Sydney)

SysBench Results (Digital Ocean/London)

SysBench Results (UpCloud/Singapore)

Comparison

Sysbench Results table

sysbench fileio results (text)

Read

  • Vultr (Sydney): 385,920
  • Digital Ocean (London): 944,280
  • UpCloud (Singapore): 944,320

Write

  • Vultr (Sydney): 823,266
  • Digital Ocean (London): 629,520
  • UpCloud (Singapore): 662,880

Other

  • Vultr (Sydney): 1,466,466
  • Digital Ocean (London): 3,588,232
  • UpCloud (Singapore): 2,121,090

Total Read Gb

  • Vultr (Sydney): 5.8887 Gb
  • Digital Ocean (London): 14.409 Gb
  • UpCloud (Singapore): 15.172 Gb

Total Written Gb

  • Vultr (Sydney): 3.9258 Gb
  • Digital Ocean (London): 9.6057 Gb
  • UpCloud (Singapore): 10.115 Gb

Total Transferred Gb

  • Vultr (Sydney): 9.8145 Gb
  • Digital Ocean (London): 24.014 Gb
  • UpCloud (Singapore): 25.287 Gb

Now I can remove test file io benchmark file

Confirm the test file has been deleted

Bonus: Benchmark MySQL (on my main server (Vultr) not on Digital Ocean and UpCLoud)

I tried to run a command

To fix the error I created a test table with Adminer (guide here).

Create Test Table

< Previous – Next >

Read Part 1, Part 2, Part 3 or Part 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

by

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

Advertisement:



Read Part 1, Part 2, Part 3 or Part 4

Measure Disk Performance with Bonnie++

Installing Bonnie++ on Ubuntu

Read this. post on using Bonnie++

Benchmark disk IO with DD and Bonnie++

Starting Bonnie++

Bonnie++ Readme.

Disk io with bonnie++ on Vultr/Sydney

Disk io with bonnie++ on Digital Ocean/London

Disk io with bonnie++ on UpCloud/Singapore

Now read this site on how to make sense of this data

 

< Previous – Next >

Read Part 1, Part 2, Part 3 or Part 4

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4

by

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4. Update: I moved my domain to UpCloud.

Advertisement:



Update (June 2018): I moved my domain to UpCloud (they are that awesome). Use this link to signup and get $25 free credit. Read the steps I took to move my domain to UpCloud here.

Upcloud Site Speed in GTMetrix

Comparing Digital Ocean/Vultr and UpCloud Disk IO

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean (all in the search of extra performance) but how do you know when a server performance is ok apart from running GT Metrix and other external site benchmarking tools.

This post is split up as it was too long.

Read Part 1, Part 2, Part 3 or Part 4

Spoiler: It all depends on where your server is located and what you do with it (Tweaks will improve the performance).

P.S This is NOT a paid endorsement or conclusive test (just a quick benchmark/review).

What does your server do?

You need to know what your server does 24/7 and what resources the services need.

I use htop to view real-time and historical usage data for each process.

htop

Tweaking Advice

A friend gave me good advice re-tweaking a cheap host to get good performance

I am not a fan of just throwing more money at a host and expecting better performance. Host have unique features and cons., there is no shortage of hosts or host cons.

How can you run synthetic benchmarks to determine comparable performance metrics?

WARNING: Comparing synthetic benchmarks can be far removed from real-world speeds. Benchmark results below were from 3 different servers I have on 3 different hosts in three different locations (the only thing the same was the use of Ubuntu 16.04 $5/m servers). These results are not scientific and should not be used to compare host providers. Benchmark runs were one-off (not averages over multiple timezones/days).

Disk Performance

Speaking of disk performance I noticed this the other day on the RunCloud blog. Faster than SSD (UpCloud)?

UpCloud Faster-than-SSD Cloud Hosting Server (Promo Code Inside)

Runcloud is a server management console that can interface with your domains (read my old review here).  I don’t use Runcloud but it is great for those who need a GUI to help manage VM via a dashboard. However, I prefer to know what is going on under the hood. I have investigated webmin in the past though.

Let’s do a quick IO benchmark test between UpCloud, Digital Ocean and Vultr on similarly low end $5/m servers,

Good advice on command line benchmarking tools from a friend.


Installing iozone to test disk performance

I searched for a post on using iozone (Thanks thegeekstuff).  I will be reviewing the “Writer report” and “Reader report”. Read more about iozone here.

View the iozone page for how to break down results.

iozone results breakdown

(image snip from http://www.iozone.org/)

Install iozone on Ubuntu

Run an iozone disk test and output the results to a spreadsheet.

Now let’s run a Read/Write test on Vultr/Digital Ocean and UpCloud. Multiple runs were not performed, this is not a scientific test (just a simple benchmark test (as is, ignoring sever load and local infrastructure/timezone load)).

iozone Benchmark results for Vultr “Read” (Sydney)

 “4” “8” “16” “32” “64” “128” “256” “512” “1024” “2048” “4096” “8192” “16384”
6421337303363612427406245647866421025
1282248149353656641359587082197413595811720614
2561884399269916138790453667079597167856870205687020
51231404883736016368473342625234610256263881650671425684095
1024161780819392073411938399976240487784614246308368058850836609617
2048192651025696784423683499761839370754596052896324354252449715854707314
409617016832151300420992050017004751325486984553892463647681492852162070354347346
8192206342423293463203763293728032214853232699362643136507063789200411060337150454350542
163841738553277836233976133679205369344231715013524291339358630040243552531345657426938452488861
327680000295289435371533574875376815547196133890280339499527352222542914
655360000405748936107893619967380007832753273591212360718817704262826659
1310720000355227018907425275167372733935276071753893323473623411111378601
2621440000379858613020211491429371282532288163757963371551025924852481061
5242880000275875624879233705741180732821183093675988319636733943302396842

iozone Benchmark results for Digital Ocean “Read” (London)

4 “8” “16” “32” “64” “128” “256”  “512”  “1024”  “2048” “4096”  “8192”  “16384”
644564786710039790061791040217812902017
12847174347082197854812497958961056714010779307
256484091170731328271916986843310148242106515981E+07
51247426166909408814039993042929638369100440891E+0710044089
102442490535917516620834375375999300377104549847E+0671131619946527
204838854316967792660354968456291040188398080369E+06790383693084977817519
40962506983595323162636116953144777437962250286E+068081580768397280815808240513
81923665114485046354793176141364627712061086086E+0665699835732541716603366334025479317
163843673501482858454161826187150661476162988726E+06643031059840336402750604615947918833405527
327680000469254261409296E+06629564252312246545707578110845134753702577
655360000631543058301316E+06644469562191256473838533859542481183679324
1310720000613000264614966E+06595806859834236387547613807839948883602079
2621440000645674663237276E+06650414663901766486151643396339551653654188
5242880000166733763814566E+06644570864487146421071598120041551853770740

iozone Benchmark results for UpCloud “Read” (Singapore)

 “4” “8” “16” “32” “64” “128” “256” “512” “1024” “2048” “4096” “8192” “16384”
6464210256421025108215241290201715972885
128488928164061389129573107793071420079414200794
25653206713879045107583228815202102450711281227712228612
512430525051154228844453823403670919528394979754017010235583
10244339202476263058212716163794681951146745106479979818391810230845
20484204968531948458008515816563624356663780055953632685108979403678229438
409645260135556581481794854045047301864575963458102806007355691953886209456281934
8192429829550190935927357603670267813416082655585563665275466553692679206564661264437634
163844282172584955863139196635840674195866570546423097553662265585756442970452703237847773901898
327680000582546054234086504198666538563653296426343526307637186053705971
655360000690807566231166493259660973863118056483610548967440359823561526
1310720000565018057189492465429539125334959115784844536740837334903582175
2621440000681462766912506189661590678660816455799913524791941212503637601
5242880000640476463092635673979575160962882456305103597868039119843767116

iozone Benchmark results for Vultr “Write” (Sydney)

 “4” “8” “16” “32” “64” “128” “256” “512” “1024” “2048” “4096” “8192” “16384”
64289322532815507625429630566551
128398921465304434078417212669577821147
256530031613985820398474937891956815414370025
512387576754083709019819085702295609421924123496091
102429723344852271608992348885407381734012031371072453601636
204840869763465569538313581345496571295458821154797520964207258493
409623615043380412157741245025820832809958137133991426992108310046821481431
819261111366667780628671521977982582429487594787009110463787911921023592453248
16384435454706149718313845499893495888068812778842885820591941120839610862672406590
327680000465196786067938881627294890917968147872369871329842843
655360000515057790172937568915601897235867197907562852002743856
13107200005010914804928131478708868802398053336846301117578633185
2621440000387126323185323656473258405744369599422554468992453563
5242880000325588380450392965451608303255355148386250432054416512

iozone Benchmark results for Digital Ocean “Write” (London)

 “4” “8” “16” “32” “64” “128” “256” “512” “1024” “2048” “4096” “8192” “16384”
64831569566551127944713639611392258
1286524881319723142102399089116631391561553
2561185399115232315343421598292182669517075891514860
51211665991296159139918916209801620980136192015897791672748
1024107919013212001584972191756215926121701108171812014629601643814
20481210394147017216217191550584179637816437531713598175958116491171488257
40969165131287575157471814065941742237173414816524181583280159934616610451533532
8192110974513187481178567154420115023401371492146674714995211479759156487812912921347609
163841106205128208413740371503649142939814614071496119157813215472891333431120337111988151501316
327680000127091414065891513114146822615583031552038151633614432801440360
655360000131932213279841311504141195512669881359645138644613470921368295
1310720000110065812293261227197131863112655521233306122774712378961233502
2621440000116716010640781155828118518510861521193673108087210626111141960
524288000097783511248161052757121918311289721140177109195411416351132063

iozone Benchmark results for UpCloud “Write” (Singapore)

 “8” “16” “32” “64” “128” “256” “512” “1024” “2048” “4096” “8192” “16384”
6411432231255511156243614525281279447
128145176414061361543594150465918525201749872
2561642294182980819708711855098180216719529472000242
51215374241854787180187322947961983258212452618957211417662
1024143413815534421609925193135920983752044438187241917683451892218
20481562145190177118172811848169196709712962402267786208149719157682007554
409616253721966378192474113420921950306207817519148731459656199515221028491326855
8192144406218083301956503192439721273002042328213563019864782062557206131913370161812049
163841667066182024818984952051339201253021110802119806149121720608751974254193478918158231921911
327680000205750614545372075621207089918697952052896189234718553821873440
655360000206712720776732088994217980920874712099108190472316425051832204
1310720000123466318249591304340177551412874811560379163199210856091675467
2621440000685774808487823824662524681762548308814946645663732176
5242880000547296517384503422521173538714518429528950529593512944

Here is my quick unscientific take on a one pass benchmark results above.

Vultr (Read) Vultr (Write) Digital Ocean (Write) UpCloud (Read) UpCloud (Write)

These results need some decoding.

Next >>

Read Part 1, Part 2, Part 3 or Part 4

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Update (June 2018): I moved my domain to UpCloud (they are that awesome). Use this link to signup and get $25 free credit. Read the steps I took to move my domain to UpCloud here.

Upcloud Site Speed in GTMetrix

Revision History

v1.2 added the fact that I Moved to UpCloud.

v1.1 Re ran iozone -a -b iozone.xls on all servers.

v1.0 Initial post

Upgrading the RAM, CPU and Memory on a Vultr Ubuntu VM in the cloud

by

Upgrading the RAM, CPU and Memory on a Vultr Ubuntu VM in the cloud is quite simple.

Advertisement:



I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line.  I prefer Vultr as they are located in the country (Australia) and are easy to use.

First, you need to shut down the server from within the VM (SSH), I used the command.

Once the VM is shut down (wait a few minutes) you can turn off the VM in the Vultr GUI.

Shutdown

You can then go to Settings, Change Plan and review upgrade options.

Upgrade Options

Snapshot

Don’t forget to take a final snapshot.

Snapshot reminder

Goto the Snapshots page (read this guide to restore a snapshot) and click Take Snapshot.

Take Snapshop

You can see snapshot progress on the main screen.

Snapshot Progress

It may take a while for your snapshot to change from Pending to Processing.

Processing

Upgrade

When the snapshot is done it will auto boot and allow you to upgrade.

Manage

Choose the Upgrade specifications (Settings, Change Plan)

Upgrade Specs

Click Upgrade

Upgrade

Confirm

Confirm

The upgrade process will take a few minutes (I could see the CU and Ram was updated but the Storage was pending)

Upgrade Pending

Testing

After the upgrade happened the VM will autoboot, login and check tour specifications (Useful Linux Commands).

I use the htop command to view specification information.

I did a quick benchmark pre-optimizing and I can see a speed bump of 0.2s. Time to optimize.

Benchmark

I threw 50 concurrent clients at my website (with loader.io) and the server handled it fine with no increase above memory capacity like before.

Concurent Users

Optimize

Now I need to Optimize.  Truth be told  I did optimize and harden PHP and crashed PHP-FPM so I had o restore a VM snapshot.

Troubleshooting

If all else fails (post-upgrade configuration) you can restore the Vultr VM from a snapshot.

I hope this guide helps someone.

 

P.S If you don’t have a VM on Vultr click this link to set one up in minutes (setup guide here).

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.0 Initial post

Ubuntu 16.04: Spectre, Meltdown Security Vulnerabilities (and how to patch).

by

Below is a post about the Spectre and Meltdown Security vulnerabilities and mostly how it relates to Ubuntu.

Advertisement:



Spectre and Meltdown Background

Google Project Zero found a server-side hardware bug (undocumented feature) that allows reading of privileged memory by leveraging a CPU (and possibly any GPU and SOC) feature to execute code ahead of time in “if” code branches before the result of the “if” case is known. This execute code ahead of demand feature was added to speed up processors to assists the FETCH, DECODE, EXECUTE and WRITE-BACK stages in the execution pipeline preparation.

Processors hate reading from main memory (it is too slow) so if data can be PREFETCHED or CACHED before being executed in the CPU allowing the CPU can do more work. This bug/flaw is not really a bug/flaw IMHO but an insecure efficiency feature.

Read more on the Spectre and Meltdown bug here at Wired.

CPU History

Aside: Check out the Red Hill Hardware guide and the evolution (documentation) of early CPU’s.

You can read more about the Pentium 4’s cache, rapid execution engine and instruction set additions to learn more about the evolution of CPU efficiency here.

Making processors faster (adding more MHZ) may be futile if the cache is too small or slow, and simply adding more cache can increase costs. Branch prediction was a way to increase performance (by using idle clock cycles or saving clock cycles) without adding extra cache or silicone (extra cost). I suspect in the future branch prediction and read ahead features may be locked down or processor manufacturers may swing back to adding more MHZ/Cores/Cache.

Anandtech https://www.anandtech.com have a great article on branch prediction (I can’t find the article now but will add it when I find it later) but this guide gives the gist.

CPU 101

A CPU is much like a checkout area at a grocery store, and a multi-core CPU is like a grocery store with multiple checkouts.

  • Things (processing and reading to/from memory) happen sequentially (per core).
  • Only one item can be scanned (processed) at a time (per core).
  • Customers trolleys and items are like program threads and items to scan (to be calculated in the CPU).
  • Customers trolleys (programs with things to calculate) line up and wait for the CPU (attendant) to scan (execute) items. PRE-FETCH and other CPU tasks help organize data related to instructions.
  • One checkout line (core) cannot read or affect items at another checkout (thread safety).

When a price check is called on an item (causing huge delays while the price is being checked by a runner (reading from main memory)) the checkout attendant (CPU core) processes the next items at the checkout (items in the processor execution pipeline). Branch predicting will read ahead in idle times to prevent idle delays or cache-misses to prevent slowdown. Processors usually make sure things are in the processors L3, L2 or L1 memory before they are executed but some commands with pre-requisite data cannot be pre-cached.

CPU instruction information

Here is a list of x86 instructions

Troy Hunt in Weekly Update 68 https://www.troyhunt.com/weekly-update-68/ mentioned a twitter thread by Graham Sutherland (@gsuberland) https://twitter.com/gsuberland/status/948907452786933762 that summaries speculative execution more succinctly. Meltdown and Spectre bugs are due to the speculative execution in the processor.

Official Information on Spectre and Meltdown

Spectre (Security Vulnerability Wikipedia Article)

Meltdown (Security Vulnerability Wikipedia Article)

Proof of concepts exploits in the wide

Proof of concept and exploits are no doubt in the wild (as reported by Michael Schwarz – @misc0110)

Ubuntu Impact

I have a number of Ubuntu servers and I have updated them to fix Spectre and Meltdown issues.

UpCloud is my favourite cloud provider.

    Ubuntu said here that is has been notified by Intel of this issue since November 09 2017.

    Ubuntu Timeline (16.04 related snip from here)

      • 2017 Nov 09: the Ubuntu Security team is notified by Intel under NDA.
      • 2017 Nov 20: the CRD is established as 2018-01-09.
      • 2017 Dec: the Ubuntu Security team receives notifications from additional silicon vendors about the impact to their products.
      • 2018 Jan 03: issue becomes public a few days before the CRD.
      • 2018 Jan 04: Canonical publicly communicates the planned update schedule.
      • 2018 Jan 04: Mozilla releases timing attack mitigations.
      • 2018 Jan 05: Ubuntu Firefox updates are made available in USN 3516-1.
      • 2018 Jan 07: Candidate kernels are beginning to be made available for testing at ppa:canonical-kernel-team/pti. This initial round will address CVE-2017-5754 (aka Meltdown or Variant 3) for x86_64. We will address CVE-2017-5715 and CVE-2017-5753 (aka Spectre or Variant 1 & 2) in a subsequent round. We will also address additional architectures in subsequent rounds. Kernels currently available are as follows. We will continue to update this table as more become available:
      • Package: linux, Version: 4.4.0-108.131, Series: Xenial 16.04
      • -2018 Jan 09: NVIDIA driver updates published, see USN-3521-1.
      • Cloud image updates.
      • Core image updates.

      At this time it looks like this has been fixed on Ubuntu 16.04 LTS (Xenial Xerus) with released (57.0.4+build1-0ubuntu0.16.04.1). Consider updating your Ubuntu servers.

      You can follow the Ubuntu CVE listing here to be ahead of future security issues.
      https://people.canonical.com/~ubuntu-security/cve/main.html

      Spectre and Meltdown related Ubuntu CVE’s

      Spectre – CVE-2017-5715

      Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.

      Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5715.html

      Spectre – CVE-2017-5753

      Description: Systems with microprocessors utilizing speculative execution and branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.

      Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Spectre. A local attacker could use this to expose sensitive information, including kernel memory.

      Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5753.html

      Meltdown – CVE-2017-5754

      Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis of the data cache.

      Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Meltdown. A local attacker could use this to expose sensitive information, including kernel memory.

      Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5754.html

      Links

      Ubuntu Security News https://usn.ubuntu.com/usn/

      Subscribe to the Ubuntu Security Announcement Distribution List https://lists.ubuntu.com/mailman/listinfo/ubuntu-security-announce

      Ubuntu CVE Tracker (Main) http://people.canonical.com/~ubuntu-security/cve/main.html

      Links from CVE articles

      https://spectreattack.com/
      https://meltdownattack.com/
      https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715
      https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html
      https://googleprojectzero.blogspot.co.uk/2018/01/reading-privileged-memory-with-side.html
      https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr
      http://www.amd.com/en/corporate/speculative-execution
      https://developer.arm.com/support/security-update
      https://www.qemu.org/2018/01/04/spectre/
      https://usn.ubuntu.com/usn/usn-3516-1
      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753
      http://nvidia.custhelp.com/app/answers/detail/a_id/4611
      https://usn.ubuntu.com/usn/usn-3521-1
      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754
      https://github.com/IAIK/KAISER
      https://gruss.cc/files/kaiser.pdf
      https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html
      https://googleprojectzero.blogspot.co.uk/2018/01/reading-privileged-memory-with-side.html

      FYI: Ubuntu 17.04 will not be getting the Spectre and Meltdown fixes, this is a good reason why not to use a non-LTS (long time support) release of Ubuntu (abandoned after 9 months):
      https://lists.ubuntu.com/archives/ubuntu-announce/2018-January/000227.html

      How to update Ubuntu

      As always backup your server and configuration first (consider taking a snapshot). I run the following command to update my system and reboot.

      Warning: Some packages may overwrite in-production configuration files (or break production servers) so take your time updating, use test servers (green and blue or dev, test and prod) and only upgrade production when you are ready.

      fyi: AWS related Speculative Execution post: https://aws.amazon.com/security/security-bulletins/AWS-2018-013/

      Impact on Future Program Build Times

      Twitter user Peter Czanik (@PCzanik https://twitter.com/PCzanik) reports that compile times that fix speculative execution have increased his build times from 4 minutes to 21 minutes.

      Windows Impacts

      Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems

      https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the-performance-impact-of-spectre-and-meltdown-mitigations-on-windows-systems/

      OSX Impacts
      Report: Intel CPUs suffer from major security flaw, fix could bring notable performance hit to macOS

      Web Browser and JavaScript Impacts

      General

      Here’s what every Chrome user should do in the wake of #Spectre

      http://mashable.com/2018/01/04/google-chrome-spectre-precaution-meltdown/

      Microsoft reveals how Spectre updates can slow your PC down

      https://www.theverge.com/2018/1/9/16868290/microsoft-meltdown-spectre-firmware-updates-pc-slowdown

      Microsoft halts AMD Meltdown and Spectre patches after reports of unbootable PCs

      Review: https://twitter.com/search?q=spectre%20meltdown

      Viewing the Change log of updatable packages

      View the change log of updatable packages for a certain Cve.

      sudo apt-get update

      sudo apt-get changelog ntp | grep CVE-2017-5715

      The output will show matches of updatable packages that match.

      Ubuntu Cloud Tips

      Read my guide on Useful Linux Terminal Commands https://fearby.com/article/useful-linux-terminal-commands/

      Read my guide on how to setting up a Vultr VM (Ubuntu) and configuring it https://fearby.com/article/setting-vultr-vm-configuring/

      Good luck.

      Scott Manleys breakdown of Spectre and Meltdown

      More Reading

      Anandtech – Understanding Meltdown & Spectre: What To Know About New Exploits That Affect Virtually All CPUs.

      More Fearby.com Reading

      Donate and make this blog better

      Ask a question or recommend an article

      Your Name (required)

      Your Email (required)

      Your Question

      Revision History

      v1.4 Scott Manleys link

      v1.3 Added Anandtech article.

      v1.2 Wired link.

      v1.1 view the changelog of updatable packages.

      v1.0 Initial Copy.

      Hope this helps someone.