Here is how I set NGINX to the development branch to get more frequent updates and features over the stable branch
Updating NGINX to the development branch (on Ubuntu) to get more frequent updates and features over the stable branch
I have a number of guides on moving away from CPanel, Setting up VM’s on UpCloud, AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. View all recent posts here https://fearby.com/all/
Now on with the post
Backup your Nginx and Server before making any changes. The Nginx development branch is quite stable but anything can happen. If your site is mission critical then stay on the stable branch.
By default, you will most likely get the stable branch of Nginx when instaling and updating Nginx. I have been running the stable version for the last few years but was made aware of a DDoS vulnerability in Nginx.
Here is a good write-up on development merges into the stable branch.
Widely-used #Nginx server releases versions 1.15.6 and 1.14.1 to patch two HTTP/2 implementation vulnerabilities that might cause excessive memory consumption (CVE-2018-16843) & CPU usage (CVE-2018-16844), allowing a remote attacker to perform #DoS attackhttps://t.co/1Z3JoghoBr pic.twitter.com/qQ3pOFD1Lk
— The Hacker News (@TheHackersNews) November 9, 2018
I was aware recently of a DDoS bug affecting Nginx and the recommendation was to update ot Nginx 1.15.6 development branch (or 1.14.1 stable branch).
A few days ago no 1.14.1 update was available but a 1.15.6 was, should I switch to the development branch to get updates earlier?
Reminder to update your #nginx installations to the 1.14.1 stable or the 1.15.6 mainline versions for critical security patches released this week. #NGINXPlus customers, see instructions for updating based on the patch released 10/30 https://t.co/KitsOWIJkb
— NGINX, Inc. (@nginx) November 8, 2018
Recent Nginx Changes
Here are the recent changes to Nginx: http://nginx.org/en/CHANGES
Changes with nginx 1.15.6 06 Nov 2018
*) Security: when using HTTP/2 a client might cause excessive memory
consumption (CVE-2018-16843) and CPU usage (CVE-2018-16844).
*) Security: processing of a specially crafted mp4 file with the
ngx_http_mp4_module might result in worker process memory disclosure
*) Feature: the "proxy_socket_keepalive", "fastcgi_socket_keepalive",
"scgi_socket_keepalive", and "uwsgi_socket_keepalive" directives.
*) Bugfix: if nginx was built with OpenSSL 1.1.0 and used with OpenSSL
1.1.1, the TLS 1.3 protocol was always enabled.
*) Bugfix: working with gRPC backends might result in excessive memory
Changes with nginx 1.15.5 02 Oct 2018
*) Bugfix: a segmentation fault might occur in a worker process when
using OpenSSL 1.1.0h or newer; the bug had appeared in 1.15.4.
*) Bugfix: of minor potential bugs.
Changes with nginx 1.15.4 25 Sep 2018
*) Feature: now the "ssl_early_data" directive can be used with OpenSSL.
*) Bugfix: in the ngx_http_uwsgi_module.
Thanks to Chris Caputo.
*) Bugfix: connections with some gRPC backends might not be cached when
using the "keepalive" directive.
*) Bugfix: a socket leak might occur when using the "error_page"
directive to redirect early request processing errors, notably errors
with code 400.
*) Bugfix: the "return" directive did not change the response code when
returning errors if the request was redirected by the "error_page"
*) Bugfix: standard error pages and responses of the
ngx_http_autoindex_module module used the "bgcolor" attribute, and
might be displayed incorrectly when using custom color settings in
Thanks to Nova DasSarma.
*) Change: the logging level of the "no suitable key share" and "no
suitable signature algorithm" SSL errors has been lowered from "crit"
Changes with nginx 1.15.3 28 Aug 2018
*) Feature: now TLSv1.3 can be used with BoringSSL.
*) Feature: the "ssl_early_data" directive, currently available with
*) Feature: the "keepalive_timeout" and "keepalive_requests" directives
in the "upstream" block.
*) Bugfix: the ngx_http_dav_module did not truncate destination file
when copying a file over an existing one with the COPY method.
*) Bugfix: the ngx_http_dav_module used zero access rights on the
destination file and did not preserve file modification time when
moving a file between different file systems with the MOVE method.
*) Bugfix: the ngx_http_dav_module used default access rights when
copying a file with the COPY method.
*) Workaround: some clients might not work when using HTTP/2; the bug
had appeared in 1.13.5.
*) Bugfix: nginx could not be built with LibreSSL 2.8.0.
Changes with nginx 1.15.2 24 Jul 2018
*) Feature: the $ssl_preread_protocol variable in the
*) Feature: now when using the "reset_timedout_connection" directive
nginx will reset connections being closed with the 444 code.
*) Change: a logging level of the "http request", "https proxy request",
"unsupported protocol", and "version too low" SSL errors has been
lowered from "crit" to "info".
*) Bugfix: DNS requests were not resent if initial sending of a request
*) Bugfix: the "reuseport" parameter of the "listen" directive was
ignored if the number of worker processes was specified after the
*) Bugfix: when using OpenSSL 1.1.0 or newer it was not possible to
switch off "ssl_prefer_server_ciphers" in a virtual server if it was
switched on in the default server.
*) Bugfix: SSL session reuse with upstream servers did not work with the
TLS 1.3 protocol.
Changes with nginx 1.15.1 03 Jul 2018
*) Feature: the "random" directive inside the "upstream" block.
*) Feature: improved performance when using the "hash" and "ip_hash"
directives with the "zone" directive.
*) Feature: the "reuseport" parameter of the "listen" directive now uses
SO_REUSEPORT_LB on FreeBSD 12.
*) Bugfix: HTTP/2 server push did not work if SSL was terminated by a
proxy server in front of nginx.
*) Bugfix: the "tcp_nopush" directive was always used on backend
*) Bugfix: sending a disk-buffered request body to a gRPC backend might
Changes with nginx 1.15.0 05 Jun 2018
*) Change: the "ssl" directive is deprecated; the "ssl" parameter of the
"listen" directive should be used instead.
*) Change: now nginx detects missing SSL certificates during
configuration testing when using the "ssl" parameter of the "listen"
*) Feature: now the stream module can handle multiple incoming UDP
datagrams from a client within a single session.
*) Bugfix: it was possible to specify an incorrect response code in the
*) Bugfix: nginx could not be built by gcc 8.1.
*) Bugfix: logging to syslog stopped on local IP address changes.
*) Bugfix: nginx could not be built by clang with CUDA SDK installed;
the bug had appeared in 1.13.8.
*) Bugfix: "getsockopt(TCP_FASTOPEN) ... failed" messages might appear
in logs during binary upgrade when using unix domain listen sockets
*) Bugfix: nginx could not be built on Fedora 28 Linux.
*) Bugfix: request processing rate might exceed configured rate when
using the "limit_req" directive.
*) Bugfix: in handling of client addresses when using unix domain listen
sockets to work with datagrams on Linux.
*) Bugfix: in memory allocation error handling.
Development branch changes are made every few weeks and stable branch changes are made less often.
Normally you update Nginx bu running an update and upgrade
apt-get update && apt-get upgrade
Restart Nginx for good measure
Checking NGINX Version
nginx version: nginx/1.14.1
Changing your repository to the development branch
I changed ot the development branch by running
sudo add-apt-repository ppa:nginx/development
Update and upgrade Nginx
apt-get update && apt-get upgrade
Restart Nginx for good measure
Checking NGINX Version
nginx version: nginx/1.16.6
Removing the stable Nginx repository
Run this command to remove the stable branch of Nginx
sudo add-apt-repository -r ppa:nginx/stable
Check to see if the development branch is listed
grep -r --include '*.list' '^deb ' /etc/apt/sources.list* |grep nginx
/etc/apt/sources.list.d/nginx-ubuntu-development-bionic.list:deb http://ppa.launchpad.net/nginx/development/ubuntu bionic main
Good luck and I hope this guide helps someone
Ask a question or recommend an article
v1.0 Initial post
Below is a post about the Spectre and Meltdown Security vulnerabilities and mostly how it relates to Ubuntu.
Spectre and Meltdown Background
Google Project Zero found a server-side hardware bug (undocumented feature) that allows reading of privileged memory by leveraging a CPU (and possibly any GPU and SOC) feature to execute code ahead of time in “if” code branches before the result of the “if” case is known. This execute code ahead of demand feature was added to speed up processors to assists the FETCH, DECODE, EXECUTE and WRITE-BACK stages in the execution pipeline preparation.
Processors hate reading from main memory (it is too slow) so if data can be PREFETCHED or CACHED before being executed in the CPU allowing the CPU can do more work. This bug/flaw is not really a bug/flaw IMHO but an insecure efficiency feature.
Read more on the Spectre and Meltdown bug here at Wired.
Aside: Check out the Red Hill Hardware guide and the evolution (documentation) of early CPU’s.
You can read more about the Pentium 4’s cache, rapid execution engine and instruction set additions to learn more about the evolution of CPU efficiency here.
Making processors faster (adding more MHZ) may be futile if the cache is too small or slow, and simply adding more cache can increase costs. Branch prediction was a way to increase performance (by using idle clock cycles or saving clock cycles) without adding extra cache or silicone (extra cost). I suspect in the future branch prediction and read ahead features may be locked down or processor manufacturers may swing back to adding more MHZ/Cores/Cache.
Anandtech https://www.anandtech.com have a great article on branch prediction (I can’t find the article now but will add it when I find it later) but this guide gives the gist.
A CPU is much like a checkout area at a grocery store, and a multi-core CPU is like a grocery store with multiple checkouts.
- Things (processing and reading to/from memory) happen sequentially (per core).
- Only one item can be scanned (processed) at a time (per core).
- Customers trolleys and items are like program threads and items to scan (to be calculated in the CPU).
- Customers trolleys (programs with things to calculate) line up and wait for the CPU (attendant) to scan (execute) items. PRE-FETCH and other CPU tasks help organize data related to instructions.
- One checkout line (core) cannot read or affect items at another checkout (thread safety).
When a price check is called on an item (causing huge delays while the price is being checked by a runner (reading from main memory)) the checkout attendant (CPU core) processes the next items at the checkout (items in the processor execution pipeline). Branch predicting will read ahead in idle times to prevent idle delays or cache-misses to prevent slowdown. Processors usually make sure things are in the processors L3, L2 or L1 memory before they are executed but some commands with pre-requisite data cannot be pre-cached.
CPU instruction information
Here is a list of x86 instructions
Troy Hunt in Weekly Update 68 https://www.troyhunt.com/weekly-update-68/ mentioned a twitter thread by Graham Sutherland (@gsuberland) https://twitter.com/gsuberland/status/948907452786933762 that summaries speculative execution more succinctly. Meltdown and Spectre bugs are due to the speculative execution in the processor.
Official Information on Spectre and Meltdown
Proof of concepts exploits in the wide
I have a number of Ubuntu servers and I have updated them to fix Spectre and Meltdown issues.
UpCloud is my favourite cloud provider.
- How to buy a new domain and SSL cert from NameCheap, (Ubuntu) Server from Digital Ocean and configure it
Ubuntu said here that is has been notified by Intel of this issue since November 09 2017.
Ubuntu Timeline (16.04 related snip from here)
- 2017 Nov 09: the Ubuntu Security team is notified by Intel under NDA.
- 2017 Nov 20: the CRD is established as 2018-01-09.
- 2017 Dec: the Ubuntu Security team receives notifications from additional silicon vendors about the impact to their products.
- 2018 Jan 03: issue becomes public a few days before the CRD.
- 2018 Jan 04: Canonical publicly communicates the planned update schedule.
- 2018 Jan 04: Mozilla releases timing attack mitigations.
- 2018 Jan 05: Ubuntu Firefox updates are made available in USN 3516-1.
- 2018 Jan 07: Candidate kernels are beginning to be made available for testing at ppa:canonical-kernel-team/pti. This initial round will address CVE-2017-5754 (aka Meltdown or Variant 3) for x86_64. We will address CVE-2017-5715 and CVE-2017-5753 (aka Spectre or Variant 1 & 2) in a subsequent round. We will also address additional architectures in subsequent rounds. Kernels currently available are as follows. We will continue to update this table as more become available:
- Package: linux, Version: 4.4.0-108.131, Series: Xenial 16.04
- -2018 Jan 09: NVIDIA driver updates published, see USN-3521-1.
- Cloud image updates.
- Core image updates.
At this time it looks like this has been fixed on Ubuntu 16.04 LTS (Xenial Xerus) with released (57.0.4+build1-0ubuntu0.16.04.1). Consider updating your Ubuntu servers.
You can follow the Ubuntu CVE listing here to be ahead of future security issues.
Spectre and Meltdown related Ubuntu CVE’s
Spectre – CVE-2017-5715
Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.
Spectre – CVE-2017-5753
Description: Systems with microprocessors utilizing speculative execution and branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.
Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Spectre. A local attacker could use this to expose sensitive information, including kernel memory.
Meltdown – CVE-2017-5754
Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis of the data cache.
Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Meltdown. A local attacker could use this to expose sensitive information, including kernel memory.
Ubuntu Security News https://usn.ubuntu.com/usn/
Subscribe to the Ubuntu Security Announcement Distribution List https://lists.ubuntu.com/mailman/listinfo/ubuntu-security-announce
Ubuntu CVE Tracker (Main) http://people.canonical.com/~ubuntu-security/cve/main.html
Links from CVE articles
FYI: Ubuntu 17.04 will not be getting the Spectre and Meltdown fixes, this is a good reason why not to use a non-LTS (long time support) release of Ubuntu (abandoned after 9 months):
How to update Ubuntu
As always backup your server and configuration first (consider taking a snapshot). I run the following command to update my system and reboot.
Warning: Some packages may overwrite in-production configuration files (or break production servers) so take your time updating, use test servers (green and blue or dev, test and prod) and only upgrade production when you are ready.
sudo apt update && sudo apt upgrade && shutdown -r now
fyi: AWS related Speculative Execution post: https://aws.amazon.com/security/security-bulletins/AWS-2018-013/
Impact on Future Program Build Times
Twitter user Peter Czanik (@PCzanik https://twitter.com/PCzanik) reports that compile times that fix speculative execution have increased his build times from 4 minutes to 21 minutes.
Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems
Report: Intel CPUs suffer from major security flaw, fix could bring notable performance hit to macOS
Here’s what every Chrome user should do in the wake of #Spectre
Microsoft reveals how Spectre updates can slow your PC down
Microsoft halts AMD Meltdown and Spectre patches after reports of unbootable PCs
Viewing the Change log of updatable packages
View the change log of updatable packages for a certain Cve.
sudo apt-get update
sudo apt-get changelog ntp | grep CVE-2017-5715
The output will show matches of updatable packages that match.
Ubuntu Cloud Tips
Read my guide on Useful Linux Terminal Commands https://fearby.com/article/useful-linux-terminal-commands/
Read my guide on how to setting up a Vultr VM (Ubuntu) and configuring it https://fearby.com/article/setting-vultr-vm-configuring/
Scott Manleys breakdown of Spectre and Meltdown
More Fearby.com Reading
- Run and Ubuntu Security scan with Lynis
- WordFence security plugin for WordPress
- Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
- Setting up additional server storage on cloud servers (block storage on Vultr)
Donate and make this blog better
Ask a question or recommend an article
v1.4 Scott Manleys link
v1.3 Added Anandtech article.
v1.2 Wired link.
v1.1 view the changelog of updatable packages.
v1.0 Initial Copy.
Hope this helps someone.