• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Create a VM ($25 Credit)
  • Buy a Domain
  • 1 Month free Back Blaze Backup
  • Other Deals
    • Domain Email
    • Nixstats Server Monitoring
    • ewww.io Auto WordPress Image Resizing and Acceleration
  • About
  • Links

IoT, Code, Security, Server Stuff etc

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

  • Cloud
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to buy a new domain and SSL cert from NameCheap, a Server from Digital Ocean and configure it.
    • Setting up a Vultr VM and configuring it
    • All Cloud Articles
  • Dev
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to setup pooled MySQL connections in Node JS that don’t disconnect
    • NodeJS code to handle App logins via API (using MySQL connection pools (1000 connections) and query parameters)
    • Infographic: So you have an idea for an app
    • All Development Articles
  • MySQL
    • Using the free Adminer GUI for MySQL on your website
    • All MySQL Articles
  • Perf
    • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap
    • All Performance Articles
  • Sec
    • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
    • Using OWASP ZAP GUI to scan your Applications for security issues
    • Setting up the Debian Kali Linux distro to perform penetration testing of your systems
    • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
    • PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API
    • Setting strong SSL cryptographic protocols and ciphers on Ubuntu and NGINX
    • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
    • All Security Articles
  • Server
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All Server Articles
  • Ubuntu
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Useful Linux Terminal Commands
    • All Ubuntu Articles
  • VM
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All VM Articles
  • WordPress
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
    • How to backup WordPress on a host that has CPanel
    • Moving WordPress to a new self managed server away from CPanel
    • Moving a CPanel domain with email to a self managed VPS and Gmail
    • All WordPress Articles
  • All

ubuntu

Backing up OSX or an Ubuntu server with Backblaze B2 Cloud Storage from the Command Line

March 14, 2018 by Simon

This computer will show you can back up computer or server with Backblaze B2 Cloud Storage from the Command Line n OSX and Ubuntu.

This post is still being written. I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Also, I have blogged about how you can add block storage to a Vultr server, backup and restore snapshots , syncing files with rsync along with using GitHub and Bitbucket but what do you do if you need to backup large amounts of data?

Backblaze has a Cloud storage solution that costs as low as $0.005c a GB (a month), The first 10G is free. Backblaze say “From bytes to petabytes Backblaze B2 is the lowest cost high-performance cloud storage in the world. ”

Back Blaze have open sourced internal drive enclosure designs and drive failure stats and it’s time I gave them a try.

Goto https://www.backblaze.com

Backblaze

Create or sign in.

Backblaze Login

After you login got the dashboard.

B2 Cloud

Click Backblaze B2 Cloud Storage

Activate B2 Cloud

Signup

Create a Bucket

Create Bucket

Name the bucket (long names with a GUID are good).

Name the Bucket

You can rename the bucket here and change public/private and or upload/download files manually.

Manage Bucket

The first thing I did was limit the versions of files under the lifecycle settings for the bucket.

Version Settings

Now I created a series of subfolders to store files from different servers (I could have used many buckets but one bucket will do).

Folders

I can upload files via the Backblaze bucket GUI if I needed to.

Upload and Download

Back Blaze has a command line tool for uploading: https://www.backblaze.com/b2/docs/quick_command_line.html

Install Steps

Backblaze state “The B2 command-line tool is available from the Python Package Index (PyPI) using the standard pip installation tool. Your first step is to make sure that you have either Python 2 (2.6 or later) or Python 3 (3.2 or later) installed.”

I have Python 2.7 installed

python --version
Python 2.7.10

Install PIP

sudo easy_install pip
Password:
Searching for pip
Best match: pip 1.5.6
Processing pip-1.5.6-py2.7.egg
pip 1.5.6 is already the active version in easy-install.pth
Installing pip script to /usr/local/bin
Installing pip2.7 script to /usr/local/bin
Installing pip2 script to /usr/local/bin

Using /Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

I ran into issues updating b2 CLI

sudo pip install --upgrade b2
Requirement already up-to-date: b2 in /Library/Python/2.7/site-packages
Requirement already up-to-date: arrow>=0.8.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: logfury>=0.1.2 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: requests>=2.9.1 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: six>=1.10 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: tqdm>=4.5.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: futures>=3.0.5 in /Library/Python/2.7/site-packages (from b2)
Downloading/unpacking python-dateutil from https://pypi.python.org/packages/bc/c5/3449988d33baca4e9619f49a14e28026399b0a8c32817e28b503923a04ab/python_dateutil-2.7.0-py2.py3-none-any.whl#md5=5a86a548fe776cc079bf4a835473e3f8 (from arrow>=0.8.0->b2)
  Downloading python_dateutil-2.7.0-py2.py3-none-any.whl (207kB): 207kB downloaded
Installing collected packages: python-dateutil
  Found existing installation: python-dateutil 1.5
    Uninstalling python-dateutil:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

I tried installing via the alternative method (with no luck)

git clone https://github.com/Backblaze/B2_Command_Line_Tool.git
Cloning into 'B2_Command_Line_Tool'...
remote: Counting objects: 5084, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 5084 (delta 1), reused 1 (delta 0), pack-reused 5076
Receiving objects: 100% (5084/5084), 1.25 MiB | 689.00 KiB/s, done.
Resolving deltas: 100% (3622/3622), done.
cd B2_Command_Line_Tool/

I tried running the setup script (with no luck)

sudo python setup.py install
setuptools 20.2 or later is required. To fix, try running: pip install "setuptools>=20.2"

Upgrading setup tools also failed

sudo pip install "setuptools>=20.2"
Downloading/unpacking setuptools>=20.2
  Downloading setuptools-38.5.2-py2.py3-none-any.whl (490kB): 490kB downloaded
Installing collected packages: setuptools
  Found existing installation: setuptools 18.5
    Uninstalling setuptools:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

Backing up a Mac via command line with B2

More to come when I can get B2 CLI Installed.

Backing up an Ubuntu machine via command line with B2

More to come when I can get B2 CLI Installed.

Update

My ticket with Backblaze was automatically closed with this note “If the issue is persisting, it may be easiest to map the installation to the user folder, rather than the system level.”

No ideas how but something to research.

Ask a question or the recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

V1.1 ticket closed

v1.0 Initial post

Filed Under: Backup Tagged With: an, B2, backblaze, Backing, cloud, command, from, line, or, OSX, server, storage, the, ubuntu, up, with

Upgrading the RAM, CPU and Memory on a Vultr Ubuntu VM in the cloud

March 7, 2018 by Simon

Upgrading the RAM, CPU and Memory on a Vultr Ubuntu VM in the cloud is quite simple.

I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line.  I prefer Vultr as they are located in the country (Australia) and are easy to use.

First, you need to shut down the server from within the VM (SSH), I used the command.

sudo shutdown now

Once the VM is shut down (wait a few minutes) you can turn off the VM in the Vultr GUI.

Shutdown

You can then go to Settings, Change Plan and review upgrade options.

Upgrade Options

Snapshot

Don’t forget to take a final snapshot.

Snapshot reminder

Goto the Snapshots page (read this guide to restore a snapshot) and click Take Snapshot.

Take Snapshop

You can see snapshot progress on the main screen.

Snapshot Progress

It may take a while for your snapshot to change from Pending to Processing.

Processing

Upgrade

When the snapshot is done it will auto boot and allow you to upgrade.

Manage

Choose the Upgrade specifications (Settings, Change Plan)

Upgrade Specs

Click Upgrade

Upgrade

Confirm

Confirm

The upgrade process will take a few minutes (I could see the CU and Ram was updated but the Storage was pending)

Upgrade Pending

Testing

After the upgrade happened the VM will autoboot, login and check tour specifications (Useful Linux Commands).

I use the htop command to view specification information.

I did a quick benchmark pre-optimizing and I can see a speed bump of 0.2s. Time to optimize.

Benchmark

I threw 50 concurrent clients at my website (with loader.io) and the server handled it fine with no increase above memory capacity like before.

Concurent Users

Optimize

Now I need to Optimize.  Truth be told  I did optimize and harden PHP and crashed PHP-FPM so I had o restore a VM snapshot.

Troubleshooting

If all else fails (post-upgrade configuration) you can restore the Vultr VM from a snapshot.

I hope this guide helps someone.

P.S If you don’t have a VM on Vultr click this link to set one up in minutes (setup guide here).

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.0 Initial post

Filed Under: Cloud, Server, Ubuntu, VM Tagged With: a, and, cloud, cpu, in, memory, on, ram, the, ubuntu, Upgrading, vm, vultr

How to upgrade a Digital Ocean Ubuntu VM and increase the vCPU or memory

February 6, 2018 by Simon

This blog post will show you how you can increase the memory and CPU allocation of an Ubuntu Server (Droplet) on Digital Ocean.

If you don’t have an Ubuntu server on Digital Ocean use this link ( https://m.do.co/c/99a5082b6de5 ) and get $10 free credit (2 months free). Read my guide here on setting it up.

Before you begin, ensure you have backed up your server.  You can read here about setting up a new server on Digital Ocean from scratch, connecting to your server via SCP or automatically syncing files away from your server to another server with rsync .

In Jan 2018 Digital Ocean doubled the ram of $5/m servers from 512MB to 1GB so it’s time for me to get the free upgrade.

Connect to your server (via SSH or Web Console) and shut it down

shutdown -h now

After the server has shut down login to digital Ocean GUI and click (open) the server you want to upgrade.

resize-droplet-002

Click Resize and choose the new upgraded server capacity

resize-droplet-003

Click Resize (if the resize button is disabled you need to power off the server (via command line or via the power menu in Digital Ocean for the Droplet))

resize-droplet-004

Click the Power On button under the Access tab when the resize is completed to restart the VM.

resize-droplet-005

Congratulations, you will now have an upgraded server 🙂 Thank You Digital Ocean for the free RAM.

If you don’t have an Ubuntu server on Digital Ocean use this link ( https://m.do.co/c/99a5082b6de5 ) and get $10 free credit (2 months free). Read my guide here on setting it up.

Hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.0 Initial Post

Filed Under: Upgrade VM Tagged With: a, and, Digital, How, increase, memory, Ocean, or, the vCPU, to, ubuntu, upgrade, vm

Restoring a Vultr VM from a snapshot

January 26, 2018 by Simon

This is a short post showing how easy it is to restore a Vultr VM (Ubuntu) to a previous snapshot.

I have set up servers on Digital Ocean, AWS and Vultr before but it is always good to know how to restore a system when disaster strikes. Never rely on backups or snapshots to save you in the time of disaster but restoring a backup or snapshot will get you going to fast (depending on how frequent you backup). It is a good idea to know how to reconfigure your servers and also copy files and code to git or other offsite platforms regularly.

I recently had a test server fall over after upgrading MongoDB 3.4 to 3.6, I think I installed upstart to try and get MongoDB to run at startup.

My server won’t boot, bad sectors, invalid VM errors etc.

Server wont boot

I tried repairing (based on advice around removing upstart etc)

Tried Repairing

But the suggested repair was for a GUI Ubuntu (and the server did not boot anyway).

Manual repair was for GUI ubuntu

3-week old snapshots available

Snapshots available

Login to Vultr, click the server you want to restore, click Snapshots and then select the select the snapshot to restore and click the restore button on the right.

Click the server you want to restore

Restore will be started

Restore will be started

Restore may take up to 1-hour ot more. Obviously, the server will be offline during the restore.

Restore may take up to 1 hour or more.

Restore progress will be available from the main server list.

Restore progress will be available from the main server list

You will be able to see when the snapshot has been restored.

You will be able to see when the snapshot has been restored.

You can verify the server is up by accessing the server (http or root console)

You can verify this by accessing the server.

Don’t forget to take new snapshots regularly.

Don't forget to take new snapshots regularly

Consider automated backups

Consider automated backups

Now I just needed to follow my guide on installing MongoDb 3.6 as this was the only change in the last 3 weeks (festive holidays).

Oh, and I had to patch for Spectre and Meltdown.

Yes you can restore a  smaller VM onto a larger VM just not  a larger snapshot onto a smaller,

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.2 Restoring a smaller snapshot onto a larger VM info

v1.1 Patch for Spectre and Meltdown

v1.0 Initial post

Filed Under: Restore, Snapshot, VM, Vultr Tagged With: restore, snapshot, ubuntu, vm, vultr

Ubuntu 16.04: Spectre, Meltdown Security Vulnerabilities (and how to patch).

January 10, 2018 by Simon

Below is a post about the Spectre and Meltdown Security vulnerabilities and mostly how it relates to Ubuntu.

Spectre and Meltdown Background

Google Project Zero found a server-side hardware bug (undocumented feature) that allows reading of privileged memory by leveraging a CPU (and possibly any GPU and SOC) feature to execute code ahead of time in “if” code branches before the result of the “if” case is known. This execute code ahead of demand feature was added to speed up processors to assists the FETCH, DECODE, EXECUTE and WRITE-BACK stages in the execution pipeline preparation.

Processors hate reading from main memory (it is too slow) so if data can be PREFETCHED or CACHED before being executed in the CPU allowing the CPU can do more work. This bug/flaw is not really a bug/flaw IMHO but an insecure efficiency feature.

Read more on the Spectre and Meltdown bug here at Wired.

CPU History

Aside: Check out the Red Hill Hardware guide and the evolution (documentation) of early CPU’s.

  • Intel 4004
  • Intel 286
  • Intel 386
  • Intel 486
  • Pentium and 686
  • AMD Athlon
  • Further Reading

You can read more about the Pentium 4’s cache, rapid execution engine and instruction set additions to learn more about the evolution of CPU efficiency here.

Making processors faster (adding more MHZ) may be futile if the cache is too small or slow, and simply adding more cache can increase costs. Branch prediction was a way to increase performance (by using idle clock cycles or saving clock cycles) without adding extra cache or silicone (extra cost). I suspect in the future branch prediction and read ahead features may be locked down or processor manufacturers may swing back to adding more MHZ/Cores/Cache.

Anandtech https://www.anandtech.com have a great article on branch prediction (I can’t find the article now but will add it when I find it later) but this guide gives the gist.

CPU 101

A CPU is much like a checkout area at a grocery store, and a multi-core CPU is like a grocery store with multiple checkouts.

  • Things (processing and reading to/from memory) happen sequentially (per core).
  • Only one item can be scanned (processed) at a time (per core).
  • Customers trolleys and items are like program threads and items to scan (to be calculated in the CPU).
  • Customers trolleys (programs with things to calculate) line up and wait for the CPU (attendant) to scan (execute) items. PRE-FETCH and other CPU tasks help organize data related to instructions.
  • One checkout line (core) cannot read or affect items at another checkout (thread safety).

When a price check is called on an item (causing huge delays while the price is being checked by a runner (reading from main memory)) the checkout attendant (CPU core) processes the next items at the checkout (items in the processor execution pipeline). Branch predicting will read ahead in idle times to prevent idle delays or cache-misses to prevent slowdown. Processors usually make sure things are in the processors L3, L2 or L1 memory before they are executed but some commands with pre-requisite data cannot be pre-cached.

CPU instruction information

Here is a list of x86 instructions

Troy Hunt in Weekly Update 68 https://www.troyhunt.com/weekly-update-68/ mentioned a twitter thread by Graham Sutherland (@gsuberland) https://twitter.com/gsuberland/status/948907452786933762 that summaries speculative execution more succinctly. Meltdown and Spectre bugs are due to the speculative execution in the processor.

Official Information on Spectre and Meltdown

Spectre (Security Vulnerability Wikipedia Article)

Meltdown (Security Vulnerability Wikipedia Article)

Proof of concepts exploits in the wide

Proof of concept and exploits are no doubt in the wild (as reported by Michael Schwarz – @misc0110)

Ubuntu Impact

I have a number of Ubuntu servers and I have updated them to fix Spectre and Meltdown issues.

UpCloud is my favourite cloud provider.

  • Setting up a Vultr VM (Ubuntu) and configuring it
  • How to buy a new domain and SSL cert from NameCheap, (Ubuntu) Server from Digital Ocean and configure it
  • Creating and configuring a CentOS server on Digital Ocean
  • Creating an AWS EC2 Ubuntu 14.04 server with NGINX, Node and MySQL and phpMyAdmin

Ubuntu said here that is has been notified by Intel of this issue since November 09 2017.

Ubuntu Timeline (16.04 related snip from here)

  • 2017 Nov 09: the Ubuntu Security team is notified by Intel under NDA.
  • 2017 Nov 20: the CRD is established as 2018-01-09.
  • 2017 Dec: the Ubuntu Security team receives notifications from additional silicon vendors about the impact to their products.
  • 2018 Jan 03: issue becomes public a few days before the CRD.
  • 2018 Jan 04: Canonical publicly communicates the planned update schedule.
  • 2018 Jan 04: Mozilla releases timing attack mitigations.
  • 2018 Jan 05: Ubuntu Firefox updates are made available in USN 3516-1.
  • 2018 Jan 07: Candidate kernels are beginning to be made available for testing at ppa:canonical-kernel-team/pti. This initial round will address CVE-2017-5754 (aka Meltdown or Variant 3) for x86_64. We will address CVE-2017-5715 and CVE-2017-5753 (aka Spectre or Variant 1 & 2) in a subsequent round. We will also address additional architectures in subsequent rounds. Kernels currently available are as follows. We will continue to update this table as more become available:
  • Package: linux, Version: 4.4.0-108.131, Series: Xenial 16.04
  • -2018 Jan 09: NVIDIA driver updates published, see USN-3521-1.
  • Cloud image updates.
  • Core image updates.

At this time it looks like this has been fixed on Ubuntu 16.04 LTS (Xenial Xerus) with released (57.0.4+build1-0ubuntu0.16.04.1). Consider updating your Ubuntu servers.

You can follow the Ubuntu CVE listing here to be ahead of future security issues.
https://people.canonical.com/~ubuntu-security/cve/main.html

Spectre and Meltdown related Ubuntu CVE’s

Spectre – CVE-2017-5715

Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.

Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5715.html

Spectre – CVE-2017-5753

Description: Systems with microprocessors utilizing speculative execution and branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.

Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Spectre. A local attacker could use this to expose sensitive information, including kernel memory.

Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5753.html

Meltdown – CVE-2017-5754

Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis of the data cache.

Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Meltdown. A local attacker could use this to expose sensitive information, including kernel memory.

Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5754.html

Links

Ubuntu Security News https://usn.ubuntu.com/usn/

Subscribe to the Ubuntu Security Announcement Distribution List https://lists.ubuntu.com/mailman/listinfo/ubuntu-security-announce

Ubuntu CVE Tracker (Main) http://people.canonical.com/~ubuntu-security/cve/main.html

Links from CVE articles

https://spectreattack.com/
https://meltdownattack.com/
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715
https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html
https://googleprojectzero.blogspot.co.uk/2018/01/reading-privileged-memory-with-side.html
https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr
http://www.amd.com/en/corporate/speculative-execution
https://developer.arm.com/support/security-update
https://www.qemu.org/2018/01/04/spectre/
https://usn.ubuntu.com/usn/usn-3516-1
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753
http://nvidia.custhelp.com/app/answers/detail/a_id/4611
https://usn.ubuntu.com/usn/usn-3521-1
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754
https://github.com/IAIK/KAISER
https://gruss.cc/files/kaiser.pdf
https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html
https://googleprojectzero.blogspot.co.uk/2018/01/reading-privileged-memory-with-side.html

FYI: Ubuntu 17.04 will not be getting the Spectre and Meltdown fixes, this is a good reason why not to use a non-LTS (long time support) release of Ubuntu (abandoned after 9 months):
https://lists.ubuntu.com/archives/ubuntu-announce/2018-January/000227.html

How to update Ubuntu

As always backup your server and configuration first (consider taking a snapshot). I run the following command to update my system and reboot.

Warning: Some packages may overwrite in-production configuration files (or break production servers) so take your time updating, use test servers (green and blue or dev, test and prod) and only upgrade production when you are ready.

sudo apt update && sudo apt upgrade && shutdown -r now

fyi: AWS related Speculative Execution post: https://aws.amazon.com/security/security-bulletins/AWS-2018-013/

Impact on Future Program Build Times

Twitter user Peter Czanik (@PCzanik https://twitter.com/PCzanik) reports that compile times that fix speculative execution have increased his build times from 4 minutes to 21 minutes.

Windows Impacts

Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems

https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the-performance-impact-of-spectre-and-meltdown-mitigations-on-windows-systems/

OSX Impacts
Report: Intel CPUs suffer from major security flaw, fix could bring notable performance hit to macOS

Web Browser and JavaScript Impacts

General

Here’s what every Chrome user should do in the wake of #Spectre

http://mashable.com/2018/01/04/google-chrome-spectre-precaution-meltdown/

Microsoft reveals how Spectre updates can slow your PC down

https://www.theverge.com/2018/1/9/16868290/microsoft-meltdown-spectre-firmware-updates-pc-slowdown

Microsoft halts AMD Meltdown and Spectre patches after reports of unbootable PCs

Review: https://twitter.com/search?q=spectre%20meltdown

Viewing the Change log of updatable packages

View the changelog of updatable packages for a certain Cve.

sudo apt-get update

sudo apt-get changelog ntp | grep CVE-2017-5715

The output will show matches of updatable packages that match.

Ubuntu Cloud Tips

Read my guide on Useful Linux Terminal Commands https://fearby.com/article/useful-linux-terminal-commands/

Read my guide on how to setting up a Vultr VM (Ubuntu) and configuring it https://fearby.com/article/setting-vultr-vm-configuring/

Good luck.

Scott Manley’s breakdown of Spectre and Meltdown

More Reading

Anandtech – Understanding Meltdown & Spectre: What To Know About New Exploits That Affect Virtually All CPUs.

More Fearby.com Reading

  • Run and Ubuntu Security scan with Lynis
  • WordFence security plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Setting up additional server storage on cloud servers (block storage on Vultr)

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.4 Scott Manleys link

v1.3 Added Anandtech article.

v1.2 Wired link.

v1.1 view the changelog of updatable packages.

v1.0 Initial Copy.

Hope this helps someone.

Filed Under: SpectreMeltdown Tagged With: 16.04, Branch, cpu, CVE, Execulative Execution, How, Meltdown, patch, security, Spectre, to, ubuntu, Vulnerabilities

Updating PHP 7.0 to 7.1 on an Ubuntu 16.04 Vultr VM

November 21, 2017 by Simon

Here is how you can quickly update PHP 7.0 to 7.1 on a Vultr Ubuntu domain.

I have configured a number of Vultr domains with NGINX and PHP 7.1 FPM and today I realised I need to update PHP 7.0 to 7.1 to fix a  few security exploits (read more here and here on securing Ubuntu in the cloud). PHP has a good page where you can keep up to date with PHP news here https://secure.php.net/. You can also view the PHP bug tracker to view bugs here. PHP aggregation user @php_net on twitter is good to follow, the official PHP twitter account is @official_php.

I have not noticed in daily Ubuntu package updates no option to update PHP 7.0 to 7.1, I must have to update manually.

WARNING: Backup your site and test this on a non-production server before doing it on a live server.  I had an issue with PHP 7.1 breaking WordPress 3.9 (MySQL issues with some plugins) and I had to roll back to 7.0 (see rollback tips in troubleshooting below). WordPress says it is PHP 7.1 compatible but issues exist. WordPress 3.9 ditches “mysql” and used “mysqli” and when instead PHP 7.1 WordPress could not find “mysqli”?

List packages with updates

sudo /usr/lib/update-notifier/apt-check -p
linux-libc-dev
python3-apport
python3-problem-report

You can run the following to view upgradable packages (TIP: Backup NGINX and other configuration files before any upgrades).

apt list --upgradable
Listing... Done
apport/xenial-updates,xenial-updates,xenial-security,xenial-security 2.20.1-0ubuntu2.13 all [upgradable from: 2.20.1-0ubuntu2.12]
linux-generic/xenial-updates,xenial-security 4.4.0.101.106 amd64 [upgradable from: 4.4.0.87.93]
linux-headers-generic/xenial-updates,xenial-security 4.4.0.101.106 amd64 [upgradable from: 4.4.0.87.93]
linux-image-generic/xenial-updates,xenial-security 4.4.0.101.106 amd64 [upgradable from: 4.4.0.87.93]
linux-libc-dev/xenial-updates,xenial-security 4.4.0-101.124 amd64 [upgradable from: 4.4.0-98.121]
nginx/xenial,xenial 1.13.6-2chl1~xenial1 all [upgradable from: 1.13.3-1chl1~xenial1]
nginx-common/xenial,xenial 1.13.6-2chl1~xenial1 all [upgradable from: 1.13.3-1chl1~xenial1]
nginx-core/xenial 1.13.4-1chl1~xenial1 amd64 [upgradable from: 1.13.3-1chl1~xenial1]
procmail/xenial-updates,xenial-security 3.22-25ubuntu0.16.04.1 amd64 [upgradable from: 3.22-25]
python-cryptography/xenial 1.9-1+ubuntu16.04.1+certbot+2 amd64 [upgradable from: 1.7.1-2+certbot~xenial+1]
python-openssl/xenial,xenial 17.3.0-1~0+ubuntu16.04.1+certbot+1 all [upgradable from: 17.0.0-0+certbot~xenial+1]
python-requests/xenial,xenial 2.18.1-1+ubuntu16.04.1+certbot+1 all [upgradable from: 2.12.4-1+certbot~xenial+1]
python-urllib3/xenial,xenial 1.21.1-1+ubuntu16.04.1+certbot+1 all [upgradable from: 1.19.1-1+certbot~xenial+1]
python3-apport/xenial-updates,xenial-updates,xenial-security,xenial-security 2.20.1-0ubuntu2.13 all [upgradable from: 2.20.1-0ubuntu2.12]
python3-problem-report/xenial-updates,xenial-updates,xenial-security,xenial-security 2.20.1-0ubuntu2.13 all [upgradable from: 2.20.1-0ubuntu2.12]
python3-requests/xenial,xenial 2.18.1-1+ubuntu16.04.1+certbot+1 all [upgradable from: 2.12.4-1+certbot~xenial+1]
python3-urllib3/xenial,xenial 1.21.1-1+ubuntu16.04.1+certbot+1 all [upgradable from: 1.19.1-1+certbot~xenial+1]

Update your server packages

sudo apt-get update && sudo apt-get upgrade

Reboot

sudo shutdown -r now

You should now see this on startup

0 packages can be updated.
0 updates are security updates.

You can view your installed PHP configuration file and installed version by typing to following in your servers command line.

# locate php.ini
/etc/php/7.0/apache2/php.ini
/etc/php/7.0/cli/php.ini
/etc/php/7.0/fpm/php.ini

Now let’s install a package viewer

sudo apt-get install apt-show-versions

Search installed packages (or non-installed) PHP packages.

sudo apt-show-versions | grep php | more

libapache2-mod-php7.0:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
libapache2-mod-php7.0:i386 not installed
php-common:all/xenial 1:55+ubuntu16.04.1+deb.sury.org+1 uptodate
php-xdebug:amd64/xenial 2.5.5-3+ubuntu16.04.1+deb.sury.org+1 uptodate
php-xdebug:i386 not installed
php7.0:all/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-cli:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-cli:i386 not installed
php7.0-common:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-common:i386 not installed
php7.0-curl:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-curl:i386 not installed
php7.0-dev:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-dev:i386 not installed
php7.0-fpm:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-fpm:i386 not installed
php7.0-gd:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-gd:i386 not installed
php7.0-imap:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-imap:i386 not installed
php7.0-intl:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-intl:i386 not installed
php7.0-json:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-json:i386 not installed
php7.0-ldap:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-ldap:i386 not installed
php7.0-mbstring:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-mbstring:i386 not installed
php7.0-mysql:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-mysql:i386 not installed
php7.0-opcache:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-opcache:i386 not installed
php7.0-pgsql:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-pgsql:i386 not installed
php7.0-phpdbg:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-phpdbg:i386 not installed
php7.0-pspell:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-pspell:i386 not installed
php7.0-readline:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-readline:i386 not installed
php7.0-recode:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-recode:i386 not installed
php7.0-snmp:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-snmp:i386 not installed
php7.0-tidy:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-tidy:i386 not installed
php7.0-xml:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-xml:i386 not installed
php7.0-zip:amd64/xenial 7.0.25-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.0-zip:i386 not installed

Uninstall all local PHP related packages

sudo apt-get remove php* 
...
After this operation, 35.7 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 139182 files and directories currently installed.)
Removing php7.0 (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php-xdebug (2.5.5-3+ubuntu16.04.1+deb.sury.org+1) ...
Removing libapache2-mod-php7.0 (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-zip (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-xml (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-mbstring (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-dev (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-fpm (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-curl (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-gd (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-imap (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-intl (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-phpdbg (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-ldap (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-mysql (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-pgsql (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-pspell (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-recode (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-snmp (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-tidy (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-cli (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-json (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-opcache (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-readline (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php7.0-common (7.0.25-1+ubuntu16.04.1+deb.sury.org+1) ...
Removing php-common (1:55+ubuntu16.04.1+deb.sury.org+1) ...
Processing triggers for man-db (2.7.5-1) ...

Confirm packages are uninstalled

sudo apt-show-versions | grep php
>

Install PHP 7.1 and common packages

sudo apt-get install php7.1 php7.1-cli php7.1-common libapache2-mod-php7.1 php7.1-mysql php7.1-fpm php7.1-curl php7.1-gd php7.1-bz2 php7.1-mcrypt php7.1-json php7.1-tidy php7.1-mbstring php-redis php-memcached

Verify PHP 7.1 installation

apt-show-versions | grep php
libapache2-mod-php7.1:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
libapache2-mod-php7.1:i386 not installed
php-common:all/xenial 1:55+ubuntu16.04.1+deb.sury.org+1 uptodate
php-igbinary:amd64/xenial 2.0.1-1+ubuntu16.04.1+deb.sury.org+2 uptodate
php-igbinary:i386 not installed
php-memcached:amd64/xenial 3.0.3+2.2.0-1+ubuntu16.04.1+deb.sury.org+3 uptodate
php-memcached:i386 not installed
php-msgpack:amd64/xenial 2.0.2+0.5.7-1+ubuntu16.04.1+deb.sury.org+3 uptodate
php-msgpack:i386 not installed
php-redis:amd64/xenial 3.1.4-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php-redis:i386 not installed
php7.1:all/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-bz2:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-bz2:i386 not installed
php7.1-cli:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-cli:i386 not installed
php7.1-common:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-common:i386 not installed
php7.1-curl:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-curl:i386 not installed
php7.1-fpm:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-fpm:i386 not installed
php7.1-gd:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-gd:i386 not installed
php7.1-json:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-json:i386 not installed
php7.1-mbstring:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-mbstring:i386 not installed
php7.1-mcrypt:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-mcrypt:i386 not installed
php7.1-mysql:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-mysql:i386 not installed
php7.1-opcache:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-opcache:i386 not installed
php7.1-readline:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-readline:i386 not installed
php7.1-tidy:amd64/xenial 7.1.11-1+ubuntu16.04.1+deb.sury.org+1 uptodate
php7.1-tidy:i386 not installed

Reboot

sudo shutdown -r now

See if the PHP 7.1 FPM service has started

sudo systemctl | grep php
> php7.1-fpm.service

Restart PHP 7.1 FPM Service

sudo systemctl restart php7.1-fpm.service

Edit your /etc/nginx/sites-enabled/default and change the fastcgi_pass from “7.0” to “7.1”

sudo nano /etc/nginx/sites-enabled/default

Edits:

location ~ \.php$ {
    ...
    fastcgi_pass unix:/var/run/php/php7.1-fpm.sock;
    ...
}

Reload NGINX configuration and restart NGINX

sudo nginx -t && sudo nginx -s reload && sudo /etc/init.d/nginx restart
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[ ok ] Restarting nginx (via systemctl): nginx.service.

Your website should now be back up and running PHP 7.1

PHP 7.1

Post Install Tasks

View this blog post on other useful linux commands.

Run a Lynis security scan.

Edit your PHP.ini file and add required changes (e.g upload sizes).

sudo nano /etc/php/7.1/fpm/php.ini
# upload_max_filesize = 2M
+ upload_max_filesize = 8M

Troubleshooting

View PHP configuration values (add this to a debug.php and load in in a browser)

<?php

// Show all information, defaults to INFO_ALL
phpinfo();

// Show just the module information.
// phpinfo(8) yields identical results.
phpinfo(INFO_MODULES);

?>

I broke my WordPress 3.9 when I tried to update to PHP 7.1 so I rolled back to 7.0.

sudo apt-get remove php*
sudo apt-get -y install php7.0-fpm
sudo apt-get -y install php7.0-mysql php7.0-curl php7.0-gd php7.0-intl php-pear php-imagick php7.0-imap php7.0-mcrypt php-memcache  php7.0-pspell php7.0-recode php7.0-sqlite3 php7.0-tidy php7.0-xmlrpc php7.0-xsl php7.0-mbstring php-gettext
service php7.0-fpm reload

Google help had me stuck for a while when I had issues purging php 7.1.

Purge Error

Because my blog (with install steps) was down I used this site to help be find the commands to run.

Conclusion

Sometimes going with cutting edge tech you will go out on a limb, ensure you know and can restore a working site if need be.

Always have a backup and restore plan.

Hope this guide helps.

Donate and make this blog better


Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.35 WordPress 3.9 error with PHP 7.1

Filed Under: PHP, Server, Ubuntu, VM, Vultr Tagged With: 16.04, a, on, php, ubuntu, Updating, vm, vultr

Security checklist for securing a self-managed Ubuntu server in the cloud

November 2, 2017 by Simon

Below is a (perpetually updated) security checklist for securing a self-managed Ubuntu server. Recently WordPress released patch v4.8.3  that fixed some SQL injection issues.  Is your OS, Database, Web Server, OS and software up to date?

Although I have recently blogged about securing Ubuntu in the cloud, and running a server Audit with Lynus,  this new post is really about obtaining a mindset change and allocating time (each week) to ensure your self-managed servers and software is kept up to date. You can easily list down the actions you need to follow but keeping a system up to date is hard work. Sites like www.shodan.io will reveal what servers or services are vulnerable, let software updates lapse long enough and an open exploit may open a hole to your system.  It only takes minutes to set up a $2.5  a month Ubuntu server with Vultr, $5 a month Digital Ocean Server or AWS server but you need to maintain it.

I highly recommend that you watch the following video that highlights the need for even minor vulnerabilities to be patched asap. If you leave one minor vulnerability open you will give hackers a foothold into your system.

Follow @jawache on twitter.

Troy Hunt has a great post about the simplicity of hacking. Hacking is child’s play.

General Security Checklist

  • Do Setup a Firewall and only allow needed ports to accept data (use tools like Portscan and Shodan.io to find open ports).
  • Use least access permissions (on NGINX, PHP and MySQL processes).
  • Use strong unique passwords for every service (1Password and sites like Gibson Research Corp have password generators, use www.howsecureismypassword.net to check tour passwords strength)
  • Enable logging.

Find log files on your system:

cd /
find -iname "*.log"

Output (handy logs to review):

./var/log/mongodb/mongod.log
./var/log/fail2ban.log
./var/log/mysql/error.log
./var/log/ufw.log
./var/log/lynis.log
./var/log/dpkg.log
./var/log/nginx/error.log
./var/log/nginx/nginxcriterror.log
./var/log/nginx/access.log
./var/log/audit/audit.log
./var/log/php7.0-fpm.log
./var/log/mail.log
./backup/backup.log
./scripts/boot.log
etc
  • Enable brute force detection and banning (fail2ban etc) Read more here.
  • Secure folders with service accounts.
  • Do secure software (e.g WordPress Wordfence)
  • Do use SSL Certificates (and use modern cyphers and test with https://www.ssllabs.com/ssltest/)
  • Monitor SSL vulnerabilities.
  • Do a Lynis security report.
  • Install a Virus scanner (read here).
  • Secure MySQL/Databases.

First, find the version of MySQL

mysql --version
mysql  Ver 14.14 Distrib 5.7.19, for Linux (x86_64)

Read the official MySQL manual here and security guidelines here.

Read this Digital Ocean guide on securing MySQL.

  • Other: _______

Application (coding) checklist

Retain and protect information.

  • Disable errors (PHP: turn off or here)
  • Enable logging (web server, PHP and or node)
  • Sanitize data (never trust uses data) in code (see how to do this in PHP 7)
  • Do no develop on production boxes (use parameterised queries and follow OWASP application security procedures.
  • Read the OWASP Secure Coding Practices – Quick Reference Guide

Infrastructure

Plan for the worst, hope for the best.

  • Use the latest Long Term Support (LTS) version or Ubuntu
  • Update packages

View app packages (Ubuntu 16.04) with updates

sudo /usr/lib/update-notifier/apt-check -p

View app packages (Ubuntu 16.04) with updates

apt list --upgradable

To update packages type (remember to backup data and config files first)

sudo apt-get update && sudo apt-get upgrade

Among other things, you will see the following information

The following packages will be upgraded:
  binutils certbot cracklib-runtime curl distro-info-data grub-common grub-pc grub-pc-bin grub2-common initramfs-tools initramfs-tools-bin initramfs-tools-core libapache2-mod-php7.0
  libcrack2 libcurl3 libcurl3-gnutls libgnutls-openssl27 libgnutls30 libicu55 libpam-systemd libsystemd0 libudev1 linux-firmware linux-libc-dev lshw mdadm mysql-client-5.7
  mysql-client-core-5.7 mysql-common mysql-server mysql-server-5.7 mysql-server-core-5.7 nodejs php7.0 php7.0-cli php7.0-common php7.0-curl php7.0-dev php7.0-fpm php7.0-gd php7.0-imap
  php7.0-intl php7.0-json php7.0-ldap php7.0-mbstring php7.0-mysql php7.0-opcache php7.0-pgsql php7.0-phpdbg php7.0-pspell php7.0-readline php7.0-recode php7.0-snmp php7.0-tidy
  php7.0-xml php7.0-zip python-acme python-certbot python-certbot-nginx python-cffi-backend python-chardet python-idna python-six python3-chardet python3-distupgrade python3-six
  python3-update-manager systemd systemd-sysv ubuntu-release-upgrader-core udev update-manager-core wget

Show available updates

/usr/lib/update-notifier/apt-check --human-readable
0 packages can be updated.
0 updates are security updates.
  • Only work on code checked into GitHub or BitBucket (You will thank me when data or servers disappear).
  • Backup configuration files or backup to remote servers (my rsync guide here)
  • Use snapshots of VM’s.
  • Use Green/Blue server deployments (toggle one server a Prod and the other and Dev/Test and have one ready for a hot spare). Digital Ocean has a good guide here.
  • Consider forcing Content Security Polic and Public Key Pinning or at least using LetsEncrypt SSL certificates.
  • Take Snapshots of VM’s (automate)
  • Backup MySQL databases:
sudo mysqldump --all-databases > /backup/dump-$( date '+%Y-%m-%d_%H-%M-%S' ).sql -u root -p

Other Useful Linus Terminal Commands.

Mindset/Culture

Dedicate time to securing your site.

  1. Spend one day a week (or automate) the updating of the OS/Software (no excuses).
  2. Follow people on twitter and subscribe to newsletters of those that are security conscious

Don’t forget to read securing Ubuntu in the cloud blog post here.

And check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

More to come..

Donate and make this blog better


Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.2 added link to Hardening Linux Server link

v1.1 added @jawache link

Short (Article):

Filed Under: OS, Secure, Security, VM Tagged With: a, checklist, cloud, for, in, securing, security, self managed, server, the, ubuntu

Debug an offline Vultr Ubuntu server

October 22, 2017 by Simon

Having a self-managed Ubuntu server from Vultr for as low as $2.5 a month (or Digital Ocean for $5 a month (setup guides here or here respectively)) can certainly be cheap but you take on all the support and risk in keeping the services up.

Vultr has a good monitoring information but this does not show you services performance.

Monitor

Node JS monitoring service like http://docs.keymetrics.io/ is great at monitoring Node applications where webmin and PHPServerMonitor are good options for monitoring basic servers and services.

The more software you install the more complex the setup and find potential errors.

Service Interruption Notifications

How will you be notified when things are down?  Having automated monitoring scripts (or Self Service Status Pages) or external monitoring services is a good idea, the last thing you wants is to see your server or service is down based on a twitter reply.

Down

Much thanks goes to Michael Boelen on Twitter for reporting that my server was down:  Follow: @mboelen

Founder of @cisofy_is, author of rkhunter and Lynis, blogger at linux-audit.com, public speaker. #linux #security

Read more Linux troubleshooting tips at https://linux-audit.com/

You can have many different types of errors and it is going to be hard to suggest where your problem is going to be.

Provider/Networking Errors

Always assume provider errors are an issue, more often than not my servers have gone offline due to provider problems outside of my control.

I have had networking errors on Vultr with the default Dynamic DHCP IP’s (prompting changes to Static IP’s). I have also had issues with Static IP’s on Vultr and had to log a Support Tickets again when my server went offline, In this case, Vultr was able to restore my server but I am unsure of what happened?

Example Problem (DHCP IP): Dynamic IP, VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).

I logged a ticket with Vultr after I rebooted the server.

DHCP

Vultr support quickly identified the problem was my server was not picking up an IP address and suggested I set static IP and reboot.

DHCP Error

Example Problem (Static IP): VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).

Again I logged a support ticket and Vultr indicated the issues was network related.

Networking

In both cases, there was nothing I could do (a reboot did not fix each error) and a support ticket had to be logged.

Broken WordPress

Example Problem (Website Down): Error 502 bad gateway

Today I had a server stay-up buy NGINX reported error 502 bad gateway. IN this case, I rebooted the server

Know your NGINX log file location

cat /etc/nginx/nginx.conf

Show the last 22 lines of the identifies NGINX log file

tail -n 200 error_log /var/log/nginx/error.log

Error  Hints

PHP-FPM is reporting errors.

2017/10/22 19:45:02 [error] 12045#0: *100883 connect() to unix:/var/run/php/php7.0-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 62.210.215.115, server: www.fearby.com, request: "GET /article/how-to-setup-a-twitter-feed-api-endpoint-in-nodejs-with-nginx-ruby-t-etc/feed/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "fearby.com"

WordPress too is reporting errors, I could have restarted NGINX and PHP but it was just as easy to reboot the server as I forgot the right commands to restart PHP/NGINX.

2017/10/22 10:28:40 [error] 12045#0: *98242 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Maximum execution time of 30 seconds exceeded in /www/wp-content/plugins/bj-lazy-load/inc/class-bjll.php on line 329
PHP message: PHP Stack trace:
PHP message: PHP   1. {main}() /www/index.php:0
PHP message: PHP   2. require() /www/index.php:17
PHP message: PHP   3. require_once() /www/wp-blog-header.php:19
PHP message: PHP   4. include() /www/wp-includes/template-loader.php:74
PHP message: PHP   5. genesis() /www/wp-content/themes/genesis/index.php:15
PHP message: PHP   6. do_action() /www/wp-content/themes/genesis/lib/framework.php:39
PHP message: PHP   7. WP_Hook->do_action() /www/wp-includes/plugin.php:453
PHP message: PHP   8. WP_Hook->apply_filters() /www/wp-includes/class-wp-hook.php:323
PHP message: PHP   9. genesis_do_loop() /www/wp-includes/class-wp-hook.php:298
PHP message: PHP  10. genesis_standard_loop() /www/wp-content/themes/genesis/lib/structure/loops.php:41
PHP message: PHP  11. do_action() /www/wp-content/themes/genesis/lib/structure/loops.php:92
PHP message: PHP  12. WP_Hook->do_action() /www/wp-includes/plugin.php:453
PHP message: PHP  13. WP_Hook->apply_filters() /www/wp-includes/class-wp-hook.php:323
PHP message: PHP  14. genesis_do_post_content() /www/wp-includes/class-wp-hook.php:298
PHP message: PHP  15. the_content() /www/wp-content/themes/genesis/lib/structure/post.php:367
PHP message: PHP  16. apply_filters() /www/wp-includes/post-template.php:240
PHP message: PHP  17. WP_Hook->apply_filters() /www/wp-includes/plugin.php:203
PHP message: PHP  18. BJLL::filter() /www/wp-includes/class-wp-hook.php:298
PHP message: PHP  19. apply_filters() /www/wp-content/plugins/bj-lazy-load/inc/class-bjll.php:169
PHP message: PHP  20. WP_Hook->apply_filters() /www/wp-includes/plugin.php:203
PHP message: PHP  21. BJLL::filter_iframes() /www/wp-includes/class-wp-hook.php:298
PHP message: PHP  22. BJLL::_get_content_haystack() /www/wp-content/plugins/bj-lazy-load/inc/class-bjll.php:255

I rebooted the server in 30 seconds.

Reboot

How I Rebooted in Ubuntu

sudo shutdown -r now

How to prevent this error in future?

  • Extend PHP file execution time (No)
  • Daily reboots of the server (No)?
  • Daily restarts of NGINX/PHP (Yes, Short Term)
  • Check the server for errors and automatically reboot (Yes, Long Term).

How to check system uptime (verifying the server rebooted)

uptime
20:04:02 up 14 min,  1 user,  load average: 0.13, 0.09, 0.08

I will set a cronjob entry to perform diagnostics (via a Bash Script) and restart NGINX/PHP or the server. I might set up remote monitoring of the webserver content too.

I should have checked memory/CPU usage too but forgot (sometimes it is best to do a deep investigation and get more information).

Troubleshooting Network Errors

As mentioned above the providers can have network issues (with a static or Dynamic IP) so don’t feel bad if you cannot fix every networking error.

Perform External Ping

Can you ping a remote server from your server?

ping www.google.com
PING www.google.com (172.217.25.132) 56(84) bytes of data.
64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=1 ttl=57 time=0.853 ms
64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=2 ttl=57 time=0.820 ms
64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=3 ttl=57 time=0.776 ms
64 bytes from syd15s03-in-f4.1e100.net (172.217.25.132): icmp_seq=4 ttl=57 time=0.835 ms

Has your domain expired?

Check the expiry of your domain here

whois google.com | egrep -i 'Expiry Date'
Registry Expiry Date: 2020-09-14T04:00:00Z

Here is a nice post on setting up a bash script to check multiple domains expiry.

Common (Digital Ocean) Debugging commands

cat /etc/network/interfaces.d/50-cloud-init.cfg
cat /etc/network/interfaces
ip addr
ip route
uname -a
iptables -nvL --line-numbers
ls -l /lib/modules
cat /etc/udev/rules.d/70-persistent-net.rules

SSL Certificate

You can check the expiry of your SSL certificate by scanning our site with https://www.ssllabs.com/ssltest/

If you use a Let’s Encrypt SSL certificate you can reissue a certificate (see my guide here)

Web Server Config

It is a good idea to make a backup of your web server (NGINX) configuration and know what each configuration value does.

View NGINX Config

cat /etc/nginx/nginx.conf

Web Console (Vultr)

Server Setup Guide

Read more here on setting up Ubuntu on Vultr

Local Network Info

ifconfig

Read more here on setting up Ubuntu Networking on Vultr

Disable Firewall

sudo ufw disable

Read more on setting up firewalls on Ubuntu here.

Logs

Common Ubuntu Log paths: https://help.ubuntu.com/community/LinuxLogFiles

If in Doubt, Reboot

sudo shutdown -r now

Log a ticket

Remember you can log a ticket with Vultr when things go pear shape.

Monitoring

Don’t forget to have a self-serve status page to alert you when things go wrong.

Vultr Status Page

View the Vultr status page here.

Your Status Page

Read my service status page guide here.

Past Data

Do document past errors and try and prevent those errors from happening again and act upon reoccurring errors (using past documentation).

Donate and make this blog better




Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.2 added common digital ocean commands

etc

Filed Under: Server, Status, Ubuntu, VM, Vultr, Wordpress Tagged With: Debug, offline, server, ubuntu, vultr

Setup an Ubuntu Desktop GUI on a Vultr VM Remotely

September 26, 2017 by Simon

Lubuntu is a free Ubuntu desktop GUI that you can install on a Vultr or Digital Ocean VM in the cloud, more information on lubuntu here.

Introduction

When you set up a Vultr server (see my guide here on setting up a Vultr VM) you can also install an Ubuntu GUI Desktop if you prefer GUI access to your server (over SSH and the terminal command line).

After you have set up your server  (fyi: you can buy a server here on Vultr for as low as $2.5 a month or buy a Digital Ocean server for $5 a month) and secure it (perform an optional security audit).  Don’t forget to add a free SSL certificate.

Vultr allows you to log into an admin panel and access your servers web console (text or GUI)

Ubuntu GUI

Setup lubuntu

From the command line type

sudo apt-get update

TIP: If you receive failures about “Temporary failure resolving ‘archive.ubuntu.com’” edit the file “/etc/resolv.conf” and add “nameserver 8.8.8.8” Update will then work.

There are two versions of the Ubuntu Desktop

1) Light Desktop

A Light version of lubuntu is available for low memory VMs

To install a light version run

sudo apt-get install lubuntu-core

Note: lubuntu-core should work on a VM with as low as 128MB of RAM. Read more here.

or

2) Full Desktop

If you have more memory (> 1GB RAM)  you can run a richer environment

sudo apt-get install lubuntu-desktop

Note: lubuntu-desktop will take a lot longer to install over the core version.

Install the Firefox browser

apt-get install firefox

Reboot Your Server

sudo shutdown -r now

After your server reboots login to the Vultr admin panel and click the Web Console icon (computer screen icon, 5 in from the right).

Ubuntu GUI

Core Version

After you login to the web console, you will receive this screen (for the core version).

Ubuntu GUI Login

You can log in as any existing user instead of a guest if you wish.

Ubuntu GUI Login Other

You have a limited desktop and environment with the core version.

Ubuntu

When you log out you can choose from the options below.

Logout

Full Desktop version

As before open the Vultr web console for the server.

Ubuntu GUI

If your lubuntu desktop session does not load try running

sudo service lightdm start

You can then log into the Ubuntu desktop (same as the light version above).  You will be able to use FireFox, use a Terminal, install packages with the Lubuntu Software Center and browse files. The start menu is per packed with software too,

Ubuntu

The full desktop has loads more software pre-installed (and available to install via the software center).

I opened a few apps and my server only used 472MB ram (it was using about 420MB before I installed lubuntu), all CPU cores were mostly idle.

Common Folders:

/usr/share/lubuntu
/usr/share/lubuntu/wallpapers
/usr/share/backgrounds

etc

Lubuntu Sofware Center

Installing MySQL workbench

MySQL Workbench

I Installed MySQL Workbench

MySQL Workbench

More software is available in the lubuntu Software Center

Synaptic Package Manager

The Synaptic package manager is also available if you are familiar with that.

Synaptic

Troubleshooting

You may receive this error on startup.

Error

Review your: /root/.profile file

sudo nano /root/.profile

Erroring line

mesg n || true

It is safe to ignore this error message as the command is just trying to generate a suggests exit result. More on the mesg command here.

Check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

Enjoy

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.2 added link to hardening linux server.

v1.1 fixed typos (26th Sep 2017)

v1.0 Initial Post (26th Sep 2017)

etc

Filed Under: Cloud, GUI, Server, Ubuntu, VM, Vultr Tagged With: gui, Setup, ubuntu, vm, vultr

Run an Ubuntu VM system audit with Lynis

September 11, 2017 by Simon

Following on from my Securing Ubuntu in the cloud blog post I have installed Lynis open source security audit tool to check out to the security of my server in the cloud.

Lynis is an open source security auditing tool. Used by system administrators, security professionals, and auditors, to evaluate the security defences of their Linux and Unix-based systems. It runs on the host itself, so it performs more extensive security scans than vulnerability scanners. https://cisofy.com/lynis and https://github.com/CISOfy/lynis.

It is easy to setup a server in the cloud (create a server on Vultr or Digital Ocean here). Guides on setting up servers exist ( setup up a Vultr VM and configure it and digital ocean server) but how about securing it? You can install a LetsEncrypt SSL certificate in minutes or setup Content Security Policy and Public Key Pinning but don’t forget to get an external in-depth review of the security of your server(s).

Lynis Security Auditing Tool

Preparing install location (for Lynis)

cd /
mkdir utils
cd utils/

Install Lynis

sudo git clone https://www.github.com/CISOfy/lynis
Cloning into 'lynis'...
remote: Counting objects: 8357, done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 8357 (delta 28), reused 42 (delta 17), pack-reused 8295
Receiving objects: 100% (8357/8357), 3.94 MiB | 967.00 KiB/s, done.
Resolving deltas: 100% (6121/6121), done.
Checking connectivity... done.

Running a Lynus system scan

./lynis audit system -Q

Lynis Results 1/3 Output (removed sensitive output)

[ Lynis 2.5.5 ]

################################################################################
  Lynis comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
  welcome to redistribute it under the terms of the GNU General Public License.
  See the LICENSE file for details about using this software.

  2007-2017, CISOfy - https://cisofy.com/lynis/
  Enterprise support available (compliance, plugins, interface and tools)
################################################################################


[+] Initializing program
------------------------------------
- Detecting OS...  [ DONE ]
- Checking profiles... [ DONE ]

  ---------------------------------------------------
  Program version:           2.5.5
  Operating system:          Linux
  Operating system name:     Ubuntu Linux
  Operating system version:  16.04
  Kernel version:            4.4.0
  Hardware platform:         x86_64
  Hostname:                  yourservername
  ---------------------------------------------------
  Profiles:                  /linis/lynis/default.prf
  Log file:                  /var/log/lynis.log
  Report file:               /var/log/lynis-report.dat
  Report version:            1.0
  Plugin directory:          ./plugins
  ---------------------------------------------------
  Auditor:                   [Not Specified]
  Test category:             all
  Test group:                all
  ---------------------------------------------------
- Program update status...  [ NO UPDATE ]

[+] System Tools
------------------------------------
- Scanning available tools...
- Checking system binaries...

[+] Plugins (phase 1)
------------------------------------
: plugins have more extensive tests and may take several minutes to complete - Plugin pam
    [..]
- Plugin systemd
    [................]

[+] Boot and services
------------------------------------
- Service Manager [ systemd ]
- Checking UEFI boot [ DISABLED ]
- Checking presence GRUB [ OK ]
- Checking presence GRUB2 [ FOUND ]
- Checking for password protection [ OK ]
- Check running services (systemctl) [ DONE ]
: found 24 running services
- Check enabled services at boot (systemctl) [ DONE ]
: found 30 enabled services
- Check startup files (permissions) [ OK ]

[+] Kernel
------------------------------------
- Checking default run level [ RUNLEVEL 5 ]
- Checking CPU support (NX/PAE)
 support: PAE and/or NoeXecute supported [ FOUND ]
- Checking kernel version and release [ DONE ]
- Checking kernel type [ DONE ]
- Checking loaded kernel modules [ DONE ]
active modules
- Checking Linux kernel configuration file [ FOUND ]
- Checking default I/O kernel scheduler [ FOUND ]
- Checking for available kernel update [ OK ]
- Checking core dumps configuration [ DISABLED ]
- Checking setuid core dumps configuration [ PROTECTED ]
- Check if reboot is needed [ NO ]

[+] Memory and Processes
------------------------------------
- Checking /proc/meminfo [ FOUND ]
- Searching for dead/zombie processes [ OK ]
- Searching for IO waiting processes [ OK ]

[+] Users, Groups and Authentication
------------------------------------
- Administrator accounts [ OK ]
- Unique UIDs [ OK ]
- Consistency of group files (grpck) [ OK ]
- Unique group IDs [ OK ]
- Unique group names [ OK ]
- Password file consistency [ OK ]
- Query system users (non daemons) [ DONE ]
- NIS+ authentication support [ NOT ENABLED ]
- NIS authentication support [ NOT ENABLED ]
- sudoers file [ FOUND ]
- Check sudoers file permissions [ OK ]
- PAM password strength tools [ OK ]
- PAM configuration files (pam.conf) [ FOUND ]
- PAM configuration files (pam.d) [ FOUND ]
- PAM modules [ FOUND ]
- LDAP module in PAM [ NOT FOUND ]
- Accounts without expire date [ OK ]
- Accounts without password [ OK ]
- Checking user password aging (minimum) [ DISABLED ]
- User password aging (maximum) [ DISABLED ]
- Checking expired passwords [ OK ]
- Checking Linux single user mode authentication [ OK ]
- Determining default umask
- umask (/etc/profile) [ NOT FOUND ]
- umask (/etc/login.defs) [ SUGGESTION ]
- umask (/etc/init.d/rc) [ SUGGESTION ]
- LDAP authentication support [ NOT ENABLED ]
- Logging failed login attempts [ ENABLED ]

[+] Shells
------------------------------------
- Checking shells from /etc/shells
: found 6 shells (valid shells: 6).
- Session timeout settings/tools [ NONE ]
- Checking default umask values
- Checking default umask in /etc/bash.bashrc [ NONE ]
- Checking default umask in /etc/profile [ NONE ]

[+] File systems
------------------------------------
- Checking mount points
- Checking /home mount point [ SUGGESTION ]
- Checking /tmp mount point [ SUGGESTION ]
- Checking /var mount point [ SUGGESTION ]
- Query swap partitions (fstab) [ NONE ]
- Testing swap partitions [ OK ]
- Testing /proc mount (hidepid) [ SUGGESTION ]
- Checking for old files in /tmp [ OK ]
- Checking /tmp sticky bit [ OK ]
- ACL support root file system [ ENABLED ]
- Mount options of / [ NON DEFAULT ]
- Checking Locate database [ FOUND ]
- Disable kernel support of some filesystems
- Discovered kernel modules: cramfs freevxfs hfs hfsplus jffs2 udf 

[+] Storage
------------------------------------
- Checking usb-storage driver (modprobe config) [ NOT DISABLED ]
- Checking USB devices authorization [ ENABLED ]
- Checking firewire ohci driver (modprobe config) [ DISABLED ]

[+] NFS
------------------------------------
- Check running NFS daemon [ NOT FOUND ]

[+] Name services
------------------------------------
- Searching DNS domain name [ UNKNOWN ]
- Checking /etc/hosts
- Checking /etc/hosts (duplicates) [ OK ]
- Checking /etc/hosts (hostname) [ OK ]
- Checking /etc/hosts (localhost) [ SUGGESTION ]
- Checking /etc/hosts (localhost to IP) [ OK ]

[+] Ports and packages
------------------------------------
- Searching package managers
- Searching dpkg package manager [ FOUND ]
- Querying package manager
- Query unpurged packages [ NONE ]
- Checking security repository in sources.list file [ OK ]
- Checking APT package database [ OK ]
- Checking vulnerable packages [ OK ]
- Checking upgradeable packages [ SKIPPED ]
- Checking package audit tool [ INSTALLED ]

[+] Networking
------------------------------------
- Checking IPv6 configuration [ ENABLED ]
 method [ AUTO ]
 only [ NO ]
- Checking configured nameservers
- Testing nameservers
: 108.xx.xx.xx [ OK ]
: 2001:xxx:xxx:xxx::6 [ OK ]
- Minimal of 2 responsive nameservers [ OK ]
- Checking default gateway [ DONE ]
- Getting listening ports (TCP/UDP) [ DONE ]
* Found 18 ports
- Checking promiscuous interfaces [ OK ]
- Checking waiting connections [ OK ]
- Checking status DHCP client [ NOT ACTIVE ]
- Checking for ARP monitoring software [ NOT FOUND ]

[+] Printers and Spools
------------------------------------
- Checking cups daemon [ NOT FOUND ]
- Checking lp daemon [ NOT RUNNING ]

[+] Software: e-mail and messaging
------------------------------------
- Sendmail status [ RUNNING ]

[+] Software: firewalls
------------------------------------
- Checking iptables kernel module [ FOUND ]
- Checking iptables policies of chains [ FOUND ]
- Checking for empty ruleset [ OK ]
- Checking for unused rules [ FOUND ]
- Checking host based firewall [ ACTIVE ]

[+] Software: webserver
------------------------------------
- Checking Apache (binary /usr/sbin/apache2) [ FOUND ]
: No virtual hosts found
* Loadable modules [ FOUND (106) ]
- Found 106 loadable modules 
- anti-DoS/brute force [ OK ]
- web application firewall [ OK ]
- Checking nginx [ FOUND ]
- Searching nginx configuration file [ FOUND ]
- Found nginx includes [ 2 FOUND ]
- Parsing configuration options
- /etc/nginx/nginx.conf
- /etc/nginx/sites-enabled/default
- SSL configured [ YES ]
- Ciphers configured [ YES ]
- Prefer server ciphers [ YES ]
- Protocols configured [ YES ]
- Insecure protocols found [ NO ]
- Checking log file configuration
- Missing log files (access_log) [ NO ]
- Disabled access logging [ NO ]
- Missing log files (error_log) [ NO ]
- Debugging mode on error_log [ NO ]

[+] SSH Support
------------------------------------
- Checking running SSH daemon [ FOUND ]
- Searching SSH configuration [ FOUND ]
- SSH option: AllowTcpForwarding [ SUGGESTION ]
- SSH option: ClientAliveCountMax [ SUGGESTION ]
- SSH option: ClientAliveInterval [ OK ]
- SSH option: Compression [ SUGGESTION ]
- SSH option: FingerprintHash [ OK ]
- SSH option: GatewayPorts [ OK ]
- SSH option: IgnoreRhosts [ OK ]
- SSH option: LoginGraceTime [ OK ]
- SSH option: LogLevel [ SUGGESTION ]
- SSH option: MaxAuthTries [ SUGGESTION ]
- SSH option: MaxSessions [ SUGGESTION ]
- SSH option: PermitRootLogin [ SUGGESTION ]
- SSH option: PermitUserEnvironment [ OK ]
- SSH option: PermitTunnel [ OK ]
- SSH option: Port [ SUGGESTION ]
- SSH option: PrintLastLog [ OK ]
- SSH option: Protocol [ OK ]
- SSH option: StrictModes [ OK ]
- SSH option: TCPKeepAlive [ SUGGESTION ]
- SSH option: UseDNS [ OK ]
- SSH option: VerifyReverseMapping [ NOT FOUND ]
- SSH option: X11Forwarding [ SUGGESTION ]
- SSH option: AllowAgentForwarding [ SUGGESTION ]
- SSH option: AllowUsers [ NOT FOUND ]
- SSH option: AllowGroups [ NOT FOUND ]

[+] SNMP Support
------------------------------------
- Checking running SNMP daemon [ NOT FOUND ]

[+] Databases
------------------------------------
- MySQL process status [FOUND ]

[+] LDAP Services
------------------------------------
- Checking OpenLDAP instance [ NOT FOUND ]

[+] PHP
------------------------------------
- Checking PHP [ FOUND ]
- Checking PHP disabled functions [ FOUND ]
- Checking expose_php option [ OFF ]
- Checking enable_dl option [ OFF ]
- Checking allow_url_fopen option [ ON ]
- Checking allow_url_include option [ OFF ]
- Checking PHP suhosin extension status [ OK ]
- Suhosin simulation mode status [ OK ]

[+] Squid Support
------------------------------------
- Checking running Squid daemon [ NOT FOUND ]

[+] Logging and files
------------------------------------
- Checking for a running log daemon [ OK ]
- Checking Syslog-NG status [ NOT FOUND ]
- Checking systemd journal status [ FOUND ]
- Checking Metalog status [ NOT FOUND ]
- Checking RSyslog status [ FOUND ]
- Checking RFC 3195 daemon status [ NOT FOUND ]
- Checking minilogd instances [ NOT FOUND ]
- Checking logrotate presence [ OK ]
- Checking log directories (static list) [ DONE ]
- Checking open log files [ DONE ]
- Checking deleted files in use [ FILES FOUND ]

[+] Insecure services
------------------------------------
- Checking inetd status [ NOT ACTIVE ]

[+] Banners and identification
------------------------------------
- /etc/issue [ FOUND ]
- /etc/issue contents [ OK ]
- /etc/issue.net [ FOUND ]
- /etc/issue.net contents [ OK ]

[+] Scheduled tasks
------------------------------------
- Checking crontab/cronjob [ DONE ]
- Checking atd status [ RUNNING ]
- Checking at users [ DONE ]
- Checking at jobs [ NONE ]

[+] Accounting
------------------------------------
- Checking accounting information [ NOT FOUND ]
- Checking sysstat accounting data [ NOT FOUND ]
- Checking auditd [ NOT FOUND ]

[+] Time and Synchronization
------------------------------------
- NTP daemon found: ntpd [ FOUND ]
- NTP daemon found: systemd (timesyncd) [ FOUND ]
- Checking for a running NTP daemon or client [ OK ]
- Checking valid association ID's [ FOUND ]
- Checking high stratum ntp peers [ OK ]
- Checking unreliable ntp peers [ FOUND ]
- Checking selected time source [ OK ]
- Checking time source candidates [ OK ]
- Checking falsetickers [ OK ]
- Checking NTP version [ FOUND ]

[+] Cryptography
------------------------------------
- Checking for expired SSL certificates [0/1] [ NONE ]

[+] Virtualization
------------------------------------

[+] Containers
------------------------------------

[+] Security frameworks
------------------------------------
- Checking presence AppArmor [ FOUND ]
- Checking AppArmor status [ ENABLED ]
- Checking presence SELinux [ NOT FOUND ]
- Checking presence grsecurity [ NOT FOUND ]
- Checking for implemented MAC framework [ OK ]

[+] Software: file integrity
------------------------------------
- Checking file integrity tools
- Checking presence integrity tool [ NOT FOUND ]

[+] Software: System tooling
------------------------------------
- Checking automation tooling
- Automation tooling [ NOT FOUND ]
- Checking presence of Fail2ban [ FOUND ]
- Checking Fail2ban jails [ ENABLED ]
- Checking for IDS/IPS tooling [ FOUND ]

[+] Software: Malware
------------------------------------

[+] File Permissions
------------------------------------
- Starting file permissions check
/root/.ssh [ OK ]

[+] Home directories
------------------------------------
- Checking shell history files [ OK ]

[+] Kernel Hardening
------------------------------------
- Comparing sysctl key pairs with scan profile
- fs.protected_hardlinks (exp: 1) [ OK ]
- fs.protected_symlinks (exp: 1) [ OK ]
- fs.suid_dumpable (exp: 0) [ DIFFERENT ]
- kernel.core_uses_pid (exp: 1) [ DIFFERENT ]
- kernel.ctrl-alt-del (exp: 0) [ OK ]
- kernel.dmesg_restrict (exp: 1) [ DIFFERENT ]
- kernel.kptr_restrict (exp: 2) [ DIFFERENT ]
- kernel.randomize_va_space (exp: 2) [ OK ]
- kernel.sysrq (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.all.accept_redirects (exp: 0) [ OK ]
- net.ipv4.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv4.conf.all.bootp_relay (exp: 0) [ OK ]
- net.ipv4.conf.all.forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.log_martians (exp: 1) [ DIFFERENT ]
- net.ipv4.conf.all.mc_forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.proxy_arp (exp: 0) [ OK ]
- net.ipv4.conf.all.rp_filter (exp: 1) [ OK ]
- net.ipv4.conf.all.send_redirects (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.default.accept_redirects (exp: 0) [ OK ]
- net.ipv4.conf.default.accept_source_route (exp: 0) [ OK ]
- net.ipv4.conf.default.log_martians (exp: 1) [ DIFFERENT ]
- net.ipv4.icmp_echo_ignore_broadcasts (exp: 1) [ OK ]
- net.ipv4.icmp_ignore_bogus_error_responses (exp: 1) [ OK ]
- net.ipv4.tcp_syncookies (exp: 1) [ DIFFERENT ]
- net.ipv4.tcp_timestamps (exp: 0) [ DIFFERENT ]
- net.ipv6.conf.all.accept_redirects (exp: 0) [ OK ]
- net.ipv6.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv6.conf.default.accept_redirects (exp: 0) [ OK ]
- net.ipv6.conf.default.accept_source_route (exp: 0) [ OK ]

[+] Hardening
------------------------------------
- Installed compiler(s) [ FOUND ]
- Installed malware scanner [ NOT FOUND ]

[+] Custom Tests
------------------------------------
- Running custom tests...  [ NONE ]

[+] Plugins (phase 2)
------------------------------------
- Plugins (phase 2) [ DONE ]

================================================================================

...

Lynis Results 2/3 – Warnings

  Warnings (1):
  ----------------------------
  ! Found one or more vulnerable packages. [REMOVED-FIXED] 
      https://cisofy.com/controls/REMOVED-FIXED/
...

I resolved the only warning by typing

apt-get update
apt-get upgrade
shutdown -r now

After updating the Lynis system scan I re-ran the text and got

 -[ Lynis 2.5.5 Results ]-

  Great, no warnings

Lynis Results 3/3 – Suggestions

  Suggestions (44):
  ----------------------------
  * Set a password on GRUB bootloader to prevent altering boot configuration (e.g. boot in single user mode without password) [BOOT-5122] 
      https://cisofy.com/controls/BOOT-5122/

  * Configure minimum password age in /etc/login.defs [AUTH-9286] 
      https://cisofy.com/controls/AUTH-9286/

  * Configure maximum password age in /etc/login.defs [AUTH-9286] 
      https://cisofy.com/controls/AUTH-9286/

  * Default umask in /etc/login.defs could be more strict like 027 [AUTH-9328] 
      https://cisofy.com/controls/AUTH-9328/

  * Default umask in /etc/init.d/rc could be more strict like 027 [AUTH-9328] 
      https://cisofy.com/controls/AUTH-9328/

  * To decrease the impact of a full /home file system, place /home on a separated partition [FILE-6310] 
      https://cisofy.com/controls/FILE-6310/

  * To decrease the impact of a full /tmp file system, place /tmp on a separated partition [FILE-6310] 
      https://cisofy.com/controls/FILE-6310/

  * To decrease the impact of a full /var file system, place /var on a separated partition [FILE-6310] 
      https://cisofy.com/controls/FILE-6310/

  * Disable drivers like USB storage when not used, to prevent unauthorized storage or data theft [STRG-1840] 
      https://cisofy.com/controls/STRG-1840/

  * Check DNS configuration for the dns domain name [NAME-4028] 
      https://cisofy.com/controls/NAME-4028/

  * Split resolving between localhost and the hostname of the system [NAME-4406] 
      https://cisofy.com/controls/NAME-4406/

  * Install debsums utility for the verification of packages with known good database. [PKGS-7370] 
      https://cisofy.com/controls/PKGS-7370/

  * Update your system with apt-get update, apt-get upgrade, apt-get dist-upgrade and/or unattended-upgrades [PKGS-7392] 
      https://cisofy.com/controls/PKGS-7392/

  * Install package apt-show-versions for patch management purposes [PKGS-7394] 
      https://cisofy.com/controls/PKGS-7394/

  * Consider running ARP monitoring software (arpwatch,arpon) [NETW-3032] 
      https://cisofy.com/controls/NETW-3032/

  * Check iptables rules to see which rules are currently not used [FIRE-4513] 
      https://cisofy.com/controls/FIRE-4513/

  * Install Apache mod_evasive to guard webserver against DoS/brute force attempts [HTTP-6640] 
      https://cisofy.com/controls/HTTP-6640/

  * Install Apache modsecurity to guard webserver against web application attacks [HTTP-6643] 
      https://cisofy.com/controls/HTTP-6643/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : AllowTcpForwarding (YES --> NO)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : ClientAliveCountMax (3 --> 2)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : Compression (DELAYED --> NO)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : LogLevel (INFO --> VERBOSE)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : MaxAuthTries (2 --> 1)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : MaxSessions (10 --> 2)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : PermitRootLogin (YES --> NO)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : Port (22 --> )
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : TCPKeepAlive (YES --> NO)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : X11Forwarding (YES --> NO)
      https://cisofy.com/controls/SSH-7408/

  * Consider hardening SSH configuration [SSH-7408] 
    - Details  : AllowAgentForwarding (YES --> NO)
      https://cisofy.com/controls/SSH-7408/

  * Change the allow_url_fopen line to: allow_url_fopen = Off, to disable downloads via PHP [PHP-2376] 
      https://cisofy.com/controls/PHP-2376

  * Check what deleted files are still in use and why. [LOGG-2190] 
      https://cisofy.com/controls/LOGG-2190/

  * Add a legal banner to /etc/issue, to warn unauthorized users [BANN-7126] 
      https://cisofy.com/controls/BANN-7126/

  * Add legal banner to /etc/issue.net, to warn unauthorized users [BANN-7130] 
      https://cisofy.com/controls/BANN-7130/

  * Enable process accounting [ACCT-9622] 
      https://cisofy.com/controls/ACCT-9622/

  * Enable sysstat to collect accounting (no results) [ACCT-9626] 
      https://cisofy.com/controls/ACCT-9626/

  * Enable auditd to collect audit information [ACCT-9628] 
      https://cisofy.com/controls/ACCT-9628/

  * Check ntpq peers output for unreliable ntp peers and correct/replace them [TIME-3120] 
      https://cisofy.com/controls/TIME-3120/

  * Install a file integrity tool to monitor changes to critical and sensitive files [FINT-4350] 
      https://cisofy.com/controls/FINT-4350/

  * Determine if automation tools are present for system management [TOOL-5002] 
      https://cisofy.com/controls/TOOL-5002/

  * One or more sysctl values differ from the scan profile and could be tweaked [KRNL-6000] 
      https://cisofy.com/controls/KRNL-6000/

  * Harden compilers like restricting access to root user only [HRDN-7222] 
      https://cisofy.com/controls/HRDN-7222/

  * Harden the system by installing at least one malware scanner, to perform periodic file system scans [HRDN-7230] 
    - Solution : Install a tool like rkhunter, chkrootkit, OSSEC
      https://cisofy.com/controls/HRDN-7230/

  Follow-up
  ----------------------------
  - Show details of a test (lynis show details TEST-ID)
  - Check the logfile for all details (less /var/log/lynis.log)
  - Read security controls texts (https://cisofy.com)
  - Use --upload to upload data to central system (Lynis Enterprise users)

================================================================================

  Lynis security scan details

  Hardening index : 64 [############        ]
  Tests performed : 255
  Plugins enabled : 2

  Components
  - Firewall               [V]
  - Malware scanner        [X]

  Lynis Modules
  - Compliance Status      [?]
  - Security Audit         [V]
  - Vulnerability Scan     [V]

  Files
  - Test and debug information      : /var/log/lynis.log
  - Report data                     : /var/log/lynis-report.dat

================================================================================

  Lynis 2.5.5

  Auditing, system hardening, and compliance for UNIX-based systems
  (Linux, macOS, BSD, and others)

  2007-2017, CISOfy - https://cisofy.com/lynis/
  Enterprise support available (compliance, plugins, interface and tools)

================================================================================

  [TIP] Enhance Lynis audits by adding your settings to custom.prf (see /linis/lynis/default.prf for all settings)

Installing a Malware Scanner

Install ClamAV

sudo apt-get install clamav

Download virus and malware definitions (this takes about 30 min)

sudo freshclam

Output:

sudo freshclam
> ClamAV Update process started at Wed Nov 15th 20:44:55 2017
> Downloading main.cvd [10%]

I had an issue on some boxes with clamav reporting I could not run freshclam

sudo freshclam
ERROR: /var/log/clamav/freshclam.log is locked by another process
ERROR: Problem with internal logger (UpdateLogFile = /var/log/clamav/freshclam.log).

This was fixed by typing

rm -rf /var/log/clamav/freshclam.log
sudo freshclam

Troubleshooting clamav

Clam AV does not like low ram boxes and may produce this error

Downloading main.cvd [100%]
ERROR: Database load killed by signal 9
ERROR: Failed to load new database

It looks like the solution is to increase your total ram.

fyi: Scan with ClamAV

sudo clamscan --max-filesize=3999M --max-scansize=3999M --exclude-dir=/www/* -i -r /

Re-running Lynis gave me the following malware status

- Malware scanner        [V]

Lynis Security rating

Hardening index : 69 [##############      ]

Installed

sudo apt-get install apt-show-versions
sudo apt-get install arpwatch
sudo apt-get install arpon

After re-running the test I got this Lynis security rating score (an improvement of 1)

Hardening index : 70 [#############       ]

Installed and configured debsums and auditd

sudo apt-get install debsums
sudo apt-get install audit

Now I get the following Lynis security rating score.

Hardening index : 71 [##############      ]

Conclusion

Lynis is great at performing an audit and recommending areas of work to allow you to harden your system (brute force protection, firewall, etc)

Security Don’ts

  • Never think you are done securing a system.

Security Do’s

  • Update Software (and remove software you do not use.)
  • Check Lynis Suggestions and try and resolve.
  • Security is an ongoing process, Do install a firewall, do ban bad IP’s, Do whitelist good IP’s, Do review Logs,
  • Do limit port access, make backups and keep on securing.

I will keep on securing and try and get remove all issues.

Read my past post on Securing Ubuntu in the cloud.

Scheduling an auto system updates is not enough in Ubuntu (as it is not recommended as the administrator should make decisions, not a scheduled job).

apt-get update
apt-get upgrade

fyi: CISOFY/Lynis do have paid subscriptions to have external scans of your servers: https://cisofy.com/pricing. (why upgrade?)

Lynis Plans

I will look into this feature soon.

Updating Lynis

I checked the official documentation and ran an update check

./lynis --check-update
This option is deprecated
Use: lynis update info

./lynis update info

 == Lynis ==

  Version            : 2.5.5
  Status             : Outdated
  Installed version  : 255
  Latest version     : 257
  Release date       : 2017-09-07
  Update location    : https://cisofy.com/lynis/


2007-2017, CISOfy - https://cisofy.com/lynis/

Not sure how to update?

./lynis update
Error: Need a target for update

Examples:
lynis update check
lynis update info

./lynis update check
status=outdated

I opened an issue about updating v2.5.5 here. I asked Twiter for help.

Twitter

Official Response: https://packages.cisofy.com/community/#debian-ubuntu

Git Response

Waiting..

I ended up deleting Lynis 2.5.5

ls -al
rm -R *
rm -rf *
rm -rf .git
rm -rf .gitignore
rm -rf .travis.yml
cd ..
rm -R lynis/
ls -al

Updated

./lynis update check
status=up-to-date

And reinstalled to v2.5.8

sudo git clone https://www.github.com/CISOfy/lynis

Output:

sudo git clone https://www.github.com/CISOfy/lynis
Cloning into 'lynis'...
remote: Counting objects: 8538, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 8538 (delta 0), reused 0 (delta 0), pack-reused 8534
Receiving objects: 100% (8538/8538), 3.96 MiB | 2.01 MiB/s, done.
Resolving deltas: 100% (6265/6265), done.
Checking connectivity... done.

More actions post upgrade to 2.5.8

  • Added a legal notice to “/etc/issues”, “/etc/issues.net” file’s.

Installing Lynis via apt-get instead of git clone

The official steps can be located here: https://packages.cisofy.com/community/#debian-ubuntu

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C80E383C3DE9F082E01391A0366C67DE91CA5D5F
apt install apt-transport-https
echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/99disable-translations
echo "deb https://packages.cisofy.com/community/lynis/deb/xenial main" > /etc/apt/sources.list.d/cisofy-lynis.list
apt update
apt install lynis
lynis show version

Unfortunately, I had an error with “apt update”

Error:

E: Malformed entry 1 in list file /etc/apt/sources.list.d/cisofy-lynis.list (Component)
E: The list of sources could not be read.

Complete install output

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C80E383C3DE9F082E01391A0366C67DE91CA5D5F
Executing: /tmp/tmp.Dz9g9nKV6i/gpg.1.sh --keyserver
keyserver.ubuntu.com
--recv-keys
C80E383C3DE9F082E01391A0366C67DE91CA5D5F
gpg: requesting key 91CA5D5F from hkp server keyserver.ubuntu.com
gpg: key 91CA5D5F: public key "CISOfy Software (signed software packages) <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

# apt install apt-transport-https
Reading package lists... Done
Building dependency tree
Reading state information... Done
apt-transport-https is already the newest version (1.2.24).
The following packages were automatically installed and are no longer required:
  gamin libfile-copy-recursive-perl libgamin0 libglade2-0 libpango1.0-0 libpangox-1.0-0 openbsd-inetd pure-ftpd-common update-inetd
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.

# echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/99disable-translations

# echo "deb https://packages.cisofy.com/community/lynis/deb/ xenial main" > /etc/apt/sources.list.d/cisofy-lynis.list

# apt update
E: Malformed entry 1 in list file /etc/apt/sources.list.d/cisofy-lynis.list (Component)
E: The list of sources could not be read.

I reopened Github issue 491. A quick reply revealed that I did not put a space before “xenial” (oops)

fyi: I removed the dead keystore from apt by typing…

apt-key list
apt-key del 91CA5D5F
rm -rf /etc/apt/sources.list.d/cisofy-lynis.list

I can now install and update other packages with apt and not have the following error

E: Malformed entry 1 in list file /etc/apt/sources.list.d/cisofy-lynis.list (Component)
E: The list of sources could not be read.
E: Malformed entry 1 in list file /etc/apt/sources.list.d/cisofy-lynis.list (Component)
E: The list of sources could not be read.

I will remove the git clone and re-run the apt version later and put in more steps to get to a High 90’s Lynis score.

More

Read the official documentation https://cisofy.com/documentation/lynis/

Next: This guide will investigate the enterprise version of https://cisofy.com/pricing/ soon.

Hope this helps. If I have missed something please let me know on Twitter at @FearbySoftware

Donate and make this blog better



Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.46 Git hub response.

Filed Under: Advice, Cloud, Computer, Firewall, OS, Security, Server, Software, ssl, Ubuntu, VM, Vultr Tagged With: Audit, Lynis, secure, security, ubuntu

  • « Go to Previous Page
  • Go to page 1
  • Go to page 2
  • Go to page 3
  • Go to Next Page »

Primary Sidebar

Poll

What would you like to see more posts about?
Results

Support this Blog

Create your own server today (support me by using these links

Create your own server on UpCloud here ($25 free credit).

Create your own server on Vultr here.

Create your own server on Digital Ocean here ($10 free credit).

Remember you can install the Runcloud server management dashboard here if you need DevOps help.

Advertisement:

Tags

2FA (9) Advice (17) Analytics (9) App (9) Apple (10) AWS (9) Backup (21) Business (8) CDN (8) Cloud (49) Cloudflare (8) Code (8) Development (26) Digital Ocean (13) DNS (11) Domain (27) Firewall (12) Git (7) Hosting (18) HTTPS (6) IoT (9) LetsEncrypt (7) Linux (20) Marketing (11) MySQL (24) NGINX (11) NodeJS (11) OS (10) PHP (13) Scalability (12) Scalable (14) Security (44) SEO (7) Server (26) Software (7) SSH (7) ssl (17) Tech Advice (9) Ubuntu (39) Uncategorized (23) UpCloud (12) VM (44) Vultr (24) Website (14) Wordpress (25)

Disclaimer

Terms And Conditions Of Use All content provided on this "www.fearby.com" blog is for informational purposes only. Views are his own and not his employers. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. Never make changes to a live site without backing it up first.

Advertisement:

Footer

Popular

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Add Google AdWords to your WordPress blog

Security

  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • Setting up DNSSEC on a Namecheap domain hosted on UpCloud using CloudFlare
  • Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx
  • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
  • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
  • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
  • Beyond SSL with Content Security Policy, Public Key Pinning etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Run an Ubuntu VM system audit with Lynis
  • Securing Ubuntu in the cloud
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Code

  • How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains
  • Useful Java FX Code I use in a project using IntelliJ IDEA and jdk1.8.0_161.jdk
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider
  • How to setup PHP FPM on demand child workers in PHP 7.x to increase website traffic
  • Installing Android Studio 3 and creating your first Kotlin Android App
  • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
  • How to use Sublime Text editor locally to edit code files on a remote server via SSH
  • Creating your first Java FX app and using the Gluon Scene Builder in the IntelliJ IDEA IDE
  • Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

Tech

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Is OSX Mojave on a 2014 MacBook Pro slower or faster than High Sierra
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • The case of the overheating Mac Book Pro and Occam’s Razor
  • Useful Linux Terminal Commands
  • Useful OSX Terminal Commands
  • Useful Linux Terminal Commands
  • What is the difference between 2D, 3D, 360 Video, AR, AR2D, AR3D, MR, VR and HR?
  • Application scalability on a budget (my journey)
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

Wordpress

  • Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution
  • Setting web push notifications in WordPress with OneSignal
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Wordfence Security Plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
  • Moving WordPress to a new self managed server away from CPanel
  • Moving WordPress to a new self managed server away from CPanel

General

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Using the WinSCP Client on Windows to transfer files to and from a Linux server over SFTP
  • Connecting to a server via SSH with Putty
  • Setting web push notifications in WordPress with OneSignal
  • Infographic: So you have an idea for an app
  • Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl
  • Building faster web apps with google tools and exceed user expectations
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in

Some ads on this site use cookies. You can opt-out if of local analytics tracking by scrolling to the bottom of the front page or any article and clicking "You are not opted out. Click here to opt out.". Accept Reject Read More
GDPR, Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT