Code, Security and Server Stuff

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View All Posts.

ubuntu

Ubuntu 16.04: Spectre, Meltdown Security Vulnerabilities (and how to patch).

by

Below is a post about the Spectre and Meltdown Security vulnerabilities and mostly how it relates to Ubuntu.

Advertisement:



Spectre and Meltdown Background

Google Project Zero found a server-side hardware bug (undocumented feature) that allows reading of privileged memory by leveraging a CPU (and possibly any GPU and SOC) feature to execute code ahead of time in “if” code branches before the result of the “if” case is known. This execute code ahead of demand feature was added to speed up processors to assists the FETCH, DECODE, EXECUTE and WRITE-BACK stages in the execution pipeline preparation.

Processors hate reading from main memory (it is too slow) so if data can be PREFETCHED or CACHED before being executed in the CPU allowing the CPU can do more work. This bug/flaw is not really a bug/flaw IMHO but an insecure efficiency feature.

Read more on the Spectre and Meltdown bug here at Wired.

CPU History

Aside: Check out the Red Hill Hardware guide and the evolution (documentation) of early CPU’s.

You can read more about the Pentium 4’s cache, rapid execution engine and instruction set additions to learn more about the evolution of CPU efficiency here.

Making processors faster (adding more MHZ) may be futile if the cache is too small or slow, and simply adding more cache can increase costs. Branch prediction was a way to increase performance (by using idle clock cycles or saving clock cycles) without adding extra cache or silicone (extra cost). I suspect in the future branch prediction and read ahead features may be locked down or processor manufacturers may swing back to adding more MHZ/Cores/Cache.

Anandtech https://www.anandtech.com have a great article on branch prediction (I can’t find the article now but will add it when I find it later) but this guide gives the gist.

CPU 101

A CPU is much like a checkout area at a grocery store, and a multi-core CPU is like a grocery store with multiple checkouts.

  • Things (processing and reading to/from memory) happen sequentially (per core).
  • Only one item can be scanned (processed) at a time (per core).
  • Customers trolleys and items are like program threads and items to scan (to be calculated in the CPU).
  • Customers trolleys (programs with things to calculate) line up and wait for the CPU (attendant) to scan (execute) items. PRE-FETCH and other CPU tasks help organize data related to instructions.
  • One checkout line (core) cannot read or affect items at another checkout (thread safety).

When a price check is called on an item (causing huge delays while the price is being checked by a runner (reading from main memory)) the checkout attendant (CPU core) processes the next items at the checkout (items in the processor execution pipeline). Branch predicting will read ahead in idle times to prevent idle delays or cache-misses to prevent slowdown. Processors usually make sure things are in the processors L3, L2 or L1 memory before they are executed but some commands with pre-requisite data cannot be pre-cached.

CPU instruction information

Here is a list of x86 instructions

Troy Hunt in Weekly Update 68 https://www.troyhunt.com/weekly-update-68/ mentioned a twitter thread by Graham Sutherland (@gsuberland) https://twitter.com/gsuberland/status/948907452786933762 that summaries speculative execution more succinctly. Meltdown and Spectre bugs are due to the speculative execution in the processor.

Official Information on Spectre and Meltdown

Spectre (Security Vulnerability Wikipedia Article)

Meltdown (Security Vulnerability Wikipedia Article)

Proof of concepts exploits in the wide

Proof of concept and exploits are no doubt in the wild (as reported by Michael Schwarz – @misc0110)

Ubuntu Impact

I have a number of Ubuntu servers and I have updated them to fix Spectre and Meltdown issues.

UpCloud is my favourite cloud provider.

    Ubuntu said here that is has been notified by Intel of this issue since November 09 2017.

    Ubuntu Timeline (16.04 related snip from here)

      • 2017 Nov 09: the Ubuntu Security team is notified by Intel under NDA.
      • 2017 Nov 20: the CRD is established as 2018-01-09.
      • 2017 Dec: the Ubuntu Security team receives notifications from additional silicon vendors about the impact to their products.
      • 2018 Jan 03: issue becomes public a few days before the CRD.
      • 2018 Jan 04: Canonical publicly communicates the planned update schedule.
      • 2018 Jan 04: Mozilla releases timing attack mitigations.
      • 2018 Jan 05: Ubuntu Firefox updates are made available in USN 3516-1.
      • 2018 Jan 07: Candidate kernels are beginning to be made available for testing at ppa:canonical-kernel-team/pti. This initial round will address CVE-2017-5754 (aka Meltdown or Variant 3) for x86_64. We will address CVE-2017-5715 and CVE-2017-5753 (aka Spectre or Variant 1 & 2) in a subsequent round. We will also address additional architectures in subsequent rounds. Kernels currently available are as follows. We will continue to update this table as more become available:
      • Package: linux, Version: 4.4.0-108.131, Series: Xenial 16.04
      • -2018 Jan 09: NVIDIA driver updates published, see USN-3521-1.
      • Cloud image updates.
      • Core image updates.

      At this time it looks like this has been fixed on Ubuntu 16.04 LTS (Xenial Xerus) with released (57.0.4+build1-0ubuntu0.16.04.1). Consider updating your Ubuntu servers.

      You can follow the Ubuntu CVE listing here to be ahead of future security issues.
      https://people.canonical.com/~ubuntu-security/cve/main.html

      Spectre and Meltdown related Ubuntu CVE’s

      Spectre – CVE-2017-5715

      Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.

      Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5715.html

      Spectre – CVE-2017-5753

      Description: Systems with microprocessors utilizing speculative execution and branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis.

      Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Spectre. A local attacker could use this to expose sensitive information, including kernel memory.

      Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5753.html

      Meltdown – CVE-2017-5754

      Description: Systems with microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized disclosure of information to an attacker with local user access via a side-channel analysis of the data cache.

      Ubuntu-Description: Jann Horn discovered that microprocessors utilizing speculative execution and indirect branch prediction may allow unauthorized memory reads via side-channel attacks. This flaw is known as Meltdown. A local attacker could use this to expose sensitive information, including kernel memory.

      Link: https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5754.html

      Links

      Ubuntu Security News https://usn.ubuntu.com/usn/

      Subscribe to the Ubuntu Security Announcement Distribution List https://lists.ubuntu.com/mailman/listinfo/ubuntu-security-announce

      Ubuntu CVE Tracker (Main) http://people.canonical.com/~ubuntu-security/cve/main.html

      Links from CVE articles

      https://spectreattack.com/
      https://meltdownattack.com/
      https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715
      https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html
      https://googleprojectzero.blogspot.co.uk/2018/01/reading-privileged-memory-with-side.html
      https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr
      http://www.amd.com/en/corporate/speculative-execution
      https://developer.arm.com/support/security-update
      https://www.qemu.org/2018/01/04/spectre/
      https://usn.ubuntu.com/usn/usn-3516-1
      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753
      http://nvidia.custhelp.com/app/answers/detail/a_id/4611
      https://usn.ubuntu.com/usn/usn-3521-1
      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754
      https://github.com/IAIK/KAISER
      https://gruss.cc/files/kaiser.pdf
      https://security.googleblog.com/2018/01/todays-cpu-vulnerability-what-you-need.html
      https://googleprojectzero.blogspot.co.uk/2018/01/reading-privileged-memory-with-side.html

      FYI: Ubuntu 17.04 will not be getting the Spectre and Meltdown fixes, this is a good reason why not to use a non-LTS (long time support) release of Ubuntu (abandoned after 9 months):
      https://lists.ubuntu.com/archives/ubuntu-announce/2018-January/000227.html

      How to update Ubuntu

      As always backup your server and configuration first (consider taking a snapshot). I run the following command to update my system and reboot.

      Warning: Some packages may overwrite in-production configuration files (or break production servers) so take your time updating, use test servers (green and blue or dev, test and prod) and only upgrade production when you are ready.

      fyi: AWS related Speculative Execution post: https://aws.amazon.com/security/security-bulletins/AWS-2018-013/

      Impact on Future Program Build Times

      Twitter user Peter Czanik (@PCzanik https://twitter.com/PCzanik) reports that compile times that fix speculative execution have increased his build times from 4 minutes to 21 minutes.

      Windows Impacts

      Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems

      https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the-performance-impact-of-spectre-and-meltdown-mitigations-on-windows-systems/

      OSX Impacts
      Report: Intel CPUs suffer from major security flaw, fix could bring notable performance hit to macOS

      Web Browser and JavaScript Impacts

      General

      Here’s what every Chrome user should do in the wake of #Spectre

      http://mashable.com/2018/01/04/google-chrome-spectre-precaution-meltdown/

      Microsoft reveals how Spectre updates can slow your PC down

      https://www.theverge.com/2018/1/9/16868290/microsoft-meltdown-spectre-firmware-updates-pc-slowdown

      Microsoft halts AMD Meltdown and Spectre patches after reports of unbootable PCs

      Review: https://twitter.com/search?q=spectre%20meltdown

      Viewing the Change log of updatable packages

      View the change log of updatable packages for a certain Cve.

      sudo apt-get update

      sudo apt-get changelog ntp | grep CVE-2017-5715

      The output will show matches of updatable packages that match.

      Ubuntu Cloud Tips

      Read my guide on Useful Linux Terminal Commands https://fearby.com/article/useful-linux-terminal-commands/

      Read my guide on how to setting up a Vultr VM (Ubuntu) and configuring it https://fearby.com/article/setting-vultr-vm-configuring/

      Good luck.

      Scott Manleys breakdown of Spectre and Meltdown

      More Reading

      Anandtech – Understanding Meltdown & Spectre: What To Know About New Exploits That Affect Virtually All CPUs.

      More Fearby.com Reading

      Donate and make this blog better

      Ask a question or recommend an article

      [contact-form-7 404 "Not Found"]

      Revision History

      v1.4 Scott Manleys link

      v1.3 Added Anandtech article.

      v1.2 Wired link.

      v1.1 view the changelog of updatable packages.

      v1.0 Initial Copy.

      Hope this helps someone.

      Updating PHP 7.0 to 7.1 on an Ubuntu 16.04 Vultr VM

      by

      Here is how you can quickly update PHP 7.0 to 7.1 on a Vultr Ubuntu domain.

      Advertisement:



      I have configured a number of Vultr domains with NGINX and PHP 7.1 FPM and today I realised I need to update PHP 7.0 to 7.1 to fix a  few security exploits (read more here and here on securing Ubuntu in the cloud). PHP has a good page where you can keep up to date with PHP news here https://secure.php.net/. You can also view the PHP bug tracker to view bugs here. PHP aggregation user @php_net on twitter is good to follow, the official PHP twitter account is @official_php.

      I have not noticed in daily Ubuntu package updates no option to update PHP 7.0 to 7.1, I must have to update manually.

      WARNING: Backup your site and test this on a non production server before doing it on a live server.  I had an issue with PHP 7.1 breaking WordPress 3.9 (mysql issues with some plugins) and I had to roll back to 7.0 (see rollback tips in troubleshooting below). WordPress says it is PHP 7.1 compatible but issues exist. WordPress 3.9 ditches “mysql” and used “mysqli” and when instead PHP 7.1 WordPress could not find “mysqli”?

      List packages with updates

      You can run the following to view upgradable packages (TIP: Backup NGINX and other configuration files before any upgrades).

      Update your server packages

      Reboot

      You should now see this on startup

      You can view your installed PHP configuration file and installed version by typing to following in your servers command line.

      Now let’s install a package viewer

      Search installed packages (or non-installed) PHP packages.

      Uninstall all local PHP related packages

      Confirm packages are uninstalled

      Install PHP 7.1 and common packages

      Verify PHP 7.1 installation

      Reboot

      See if the PHP 7.1 FPM service has started

      sudo systemctl | grep php
> php7.1-fpm.service

      Restart PHP 7.1 FPM Service

      Edit your /etc/nginx/sites-enabled/default and change the fastcgi_pass from “7.0” to “7.1”

      Edits:

      Reload NGINX configuration and restart NGINX

      Your website should now be back up and running PHP 7.1

      PHP 7.1

      Post Install Tasks

      View this blog post on other useful linux commands.

      Run a Lynis security scan.

      Edit your PHP.ini file and add required changes (e.g upload sizes).

      sudo nano /etc/php/7.1/fpm/php.ini
# upload_max_filesize = 2M
+ upload_max_filesize = 8M

      Troubleshooting

      View PHP configuration values (add this to a debug.php and load in in a browser)

      <?php

// Show all information, defaults to INFO_ALL
phpinfo();

// Show just the module information.
// phpinfo(8) yields identical results.
phpinfo(INFO_MODULES);

?>

      I broke my WordPress 3.9 when I tried to update to PHP 7.1 so I rolled back to 7.0.

      Google help had me stuck for a while when I had issues purging php 7.1.

      Purge Error

      Because my blog (with install steps) was down I used this site to help be find the commands to run.

      Conclusion

      Sometimes going with cutting edge tech you will go out on a limb, ensure you know and can restore a working site if need be.

      Always have a backup and restore plan.

      Hope this guide helps.

      Donate and make this blog better


      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      v1.35 WordPress 3.9 error with PHP 7.1

      Security checklist for securing a self-managed Ubuntu server in the cloud

      by

      Below is a (perpetually updated) security checklist for securing a self-managed Ubuntu server. Recently WordPress released patch v4.8.3  that fixed some SQL injection issues.  Is your OS, Database, Web Server, OS and software up to date?

      Advertisement:



      Although I have recently blogged about securing Ubuntu in the cloud, and running a server Audit with Lynus,  this new post is really about obtaining a mindset change and allocating time (each week) to ensure your self-managed servers and software is kept up to date. You can easily list down the actions you need to follow but keeping a system up to date is hard work. Sites like www.shodan.io will reveal what servers or services are vulnerable, let software updates lapse long enough and an open exploit may open a hole to your system.  It only takes minutes to set up a $2.5  a month Ubuntu server with Vultr, $5 a month Digital Ocean Server or AWS server but you need to maintain it.

      I highly recommend that you watch the following video that highlights the need for even minor vulnerabilities to be patched asap. If you leave one minor vulnerability open you will give hackers a foothold into your system.

      Follow @jawache on twitter.

      Troy Hunt has a great post about the simplicity of hacking. Hacking is child’s play.

      General Security Checklist

      Find log files on your system:

      Output (handy logs to review):

      ./var/log/mongodb/mongod.log
./var/log/fail2ban.log
./var/log/mysql/error.log
./var/log/ufw.log
./var/log/lynis.log
./var/log/dpkg.log
./var/log/nginx/error.log
./var/log/nginx/nginxcriterror.log
./var/log/nginx/access.log
./var/log/audit/audit.log
./var/log/php7.0-fpm.log
./var/log/mail.log
./backup/backup.log
./scripts/boot.log
etc
      • Enable brute force detection and banning (fail2ban etc) Read more here.
      • Secure folders with service accounts.
      • Do secure software (e.g WordPress Wordfence)
      • Do use SSL Certificates (and use modern cyphers and test with https://www.ssllabs.com/ssltest/)
      • Monitor SSL vulnerabilities.
      • Do a Lynis security report.
      • Install a Virus scanner (read here).
      • Secure MySQL/Databases.

      First, find the version of MySQL

      mysql --version
mysql  Ver 14.14 Distrib 5.7.19, for Linux (x86_64)

      Read the official MySQL manual here and security guidelines here.

      Read this Digital Ocean guide on securing MySQL.

      • Other: _______

      Application (coding) checklist

      Retain and protect information.

      Infrastructure

      Plan for the worst, hope for the best.

      • Use the latest Long Term Support (LTS) version or Ubuntu
      • Update packages

      View app packages (Ubuntu 16.04) with updates

      View app packages (Ubuntu 16.04) with updates

      To update packages type (remember to backup data and config files first)

      Among other things, you will see the following information

      Show available updates

      • Only work on code checked into GitHub or BitBucket (You will thank me when data or servers disappear).
      • Backup configuration files or backup to remote servers (my rsync guide here)
      • Use snapshots of VM’s.
      • Use Green/Blue server deployments (toggle one server a Prod and the other and Dev/Test and have one ready for a hot spare). Digital Ocean has a good guide here.
      • Consider forcing Content Security Polic and Public Key Pinning or at least using LetsEncrypt SSL certificates.
      • Take Snapshots of VM’s (automate)
      • Backup MySQL databases:
      sudo mysqldump --all-databases > /backup/dump-$( date '+%Y-%m-%d_%H-%M-%S' ).sql -u root -p

      Other Useful Linus Terminal Commands.

      Mindset/Culture

      Dedicate time to securing your site.

      1. Spend one day a week (or automate) the updating of the OS/Software (no excuses).
      2. Follow people on twitter and subscribe to newsletters of those that are security conscious

      Don’t forget to read securing Ubuntu in the cloud blog post here.

      And check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

      More to come..

      Donate and make this blog better


      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      v1.2 added link to Hardening Linux Server link

      v1.1 added @jawache link

      Short (Article):

      Debug an offline Vultr Ubuntu server

      by

      Having a self-managed Ubuntu server from Vultr for as low as $2.5 a month (or Digital Ocean for $5 a month (setup guides here or here respectively)) can certainly be cheap but you take on all the support and risk in keeping the services up.

      Advertisement:



      Vultr has a good monitoring information but this does not show you services performance.

      Monitor

      Node JS monitoring service like http://docs.keymetrics.io/ is great at monitoring Node applications where webmin and PHPServerMonitor are good options for monitoring basic servers and services.

      The more software you install the more complex the setup and find potential errors.

      Service Interruption Notifications

      How will you be notified when things are down?  Having automated monitoring scripts (or Self Service Status Pages) or external monitoring services is a good idea, the last thing you wants is to see your server or service is down based on a twitter reply.

      Down

      Much thanks goes to Michael Boelen on Twitter for reporting that my server was down:  Follow: @mboelen

      Founder of @cisofy_is, author of rkhunter and Lynis, blogger at linux-audit.com, public speaker. #linux #security

      Read more Linux troubleshooting tips at https://linux-audit.com/

      You can have many different types of errors and it is going to be hard to suggest where your problem is going to be.

      Provider/Networking Errors

      Always assume provider errors are an issue, more often than not my servers have gone offline due to provider problems outside of my control.

      I have had networking errors on Vultr with the default Dynamic DHCP IP’s (prompting changes to Static IP’s). I have also had issues with Static IP’s on Vultr and had to log a Support Tickets again when my server went offline, In this case, Vultr were able to restore my server but I am unsure of what happened?

      Example Problem (DHCP IP): Dynamic IP, VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).

      I logged a ticket with Vultr after I rebooted the server.

      DHCP

      Vultr support quickly identified the problem was my server was not picking up an IP address and suggested I set static IP and reboot.

      DHCP Error

      Example Problem (Static IP): VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).

      Again I logged a support ticket and Vultr indicated the issues was network related.

      Networking

      In both cases, there was nothing I could do (a reboot did not fix each error) and a support ticket had to be logged.

      Broken WordPress

      Example Problem (Website Down): Error 502 bad gateway

      Today I had a server stay-up buy NGINX reported error 502 bad gateway. IN this case I rebooted teh server

      Know your NGINX log file location

      Show the last 22 lines of the identifies NGINX log file

      tail -n 200 error_log /var/log/nginx/error.log

      Error  Hints

      PHP-FPM is reporting errors.

      WordPress too is reporting errors, I could have restarted NGINX and PHP but it was just as easy to reboot the server as I forgot the right commands to restart PHP/NGINX.

      I rebooted the server in 30 seconds.

      Reboot

      How I Rebooted in Ubuntu

      How to prevent this error in future?

      • Extend PHP file execution time (No)
      • Daily reboots of the server (No)?
      • Daily restarts of NGINX/PHP (Yes, Short Term)
      • Check the server for errors and automatically reboot (Yes, Long Term).

      How to check system uptime (verifying the server rebooted)

      I will set a cronjob entry to perform diagnostics (via a Bash Script) and restart NGINX/PHP or the server. I might set up remote monitoring of the web server content too.

      I should have checked memory/CPU usage too but forgot (sometimes it is best to do a deep investigation and get more information).

      Troubleshooting Network Errors

      As mentioned above the providers can have network issues (with a static or Dynamic IP) so don’t feel bad if you cannot fix every networking error.

      Perform External Ping

      Can you ping a remote server from your server?

      Has your domain expired?

      Check the expiry of your domain here

      Here is a nice post on setting up a bash script to check multiple domains expiry.

      Common (Digital Ocean) Debugging commands

       

      SSL Certificate

      You can check the expiry of your SSL certificate by scanning our site with https://www.ssllabs.com/ssltest/

      If you use a Let’s Encrypt SSL certificate you can reissue a certificate (see my guide here)

      Web Server Config

      It is a good idea to make a backup of your web server (NGINX) configuration and know what each configuration value does.

      View NGINX Config

      Web Console (Vultr)

      Server Setup Guide

      Read more here on setting up Ubuntu on Vultr

      Local Network Info

      Read more here on setting up Ubuntu Networking on Vultr

      Disable Firewall

      Read more on setting up firewalls on Ubuntu here.

      Logs

      Common Ubuntu Log paths: https://help.ubuntu.com/community/LinuxLogFiles

      If in Doubt, Reboot

      Log a ticket

      Remember you can log a ticket with Vultr when things go pear shape.

      Monitoring

      Don’t forget to have a self-serve status page to alert you when things go wrong.

      Vultr Status Page

      View the Vultr status page here.

      Your Status Page

      Read my service status page guide here.

      Past Data

      Do document past errors and try and prevent those errors from happening again and act upon reoccurring errors (using past documentation).

       

      Donate and make this blog better




      Ask a question or recommend an article

      [contact-form-7 404 "Not Found"]

      Revision History

      v1.2 added common digital ocean commands

      etc

      Setup an Ubuntu Desktop GUI on a Vultr VM Remotely

      by

      Lubuntu is a free Ubuntu desktop GUI that you can install on a Vultr or Digital Ocean VM in the cloud, more information on lubuntu here.

      Advertisement:



      Introduction

      When you set up a Vultr server (see my guide here on setting up a Vultr VM) you can also install an Ubuntu GUI Desktop if you prefer GUI access to your server (over SSH and the terminal command line).

      After you have set up your server  (fyi: you can buy a server here on Vultr for as low as $2.5 a month or buy a Digital Ocean server for $5 a month) and secure it (perform an optional security audit).  Don’t forget to add a free SSL certificate.

      Vultr allows you to log into an admin panel and access your servers web console (text or GUI)

      Ubuntu GUI

      Setup lubuntu

      From the command line type

      TIP: If you receive failures about “Temporary failure resolving ‘archive.ubuntu.com’” edit the file “/etc/resolv.conf” and add “nameserver 8.8.8.8” Update will then work.

      There are two versions of the Ubuntu Desktop

      1) Light Desktop

      A Light version of lubuntu is available for low memory VMs

      To install a light version run

      Note: lubuntu-core should work on a VM with as low as 128MB of RAM. Read more here.

      or

      2) Full Desktop

      If you have more memory (> 1GB RAM)  you can run a richer environment

      Note: lubuntu-desktop will take a lot longer to install over the core version.

      Install the Firefox browser

      apt-get install firefox

      Reboot Your Server

      sudo shutdown -r now

      After your server reboots login to the Vultr admin panel and click the Web Console icon (computer screen icon, 5 in from the right).

      Ubuntu GUI

      Core Version

      After you login to the web console, you will receive this screen (for the core version).

      Ubuntu GUI Login

      You can log in as any existing user instead of a guest if you wish.

      Ubuntu GUI Login Other

      You have a limited desktop and environment with the core version.

      Ubuntu

      When you log out you can choose from the options below.

      Logout

      Full Desktop version

      As before open the Vultr web console for the server.

      Ubuntu GUI

      If your lubuntu desktop session does not load try running

      You can then log into the Ubuntu desktop (same as the light version above).  You will be able to use FireFox, use a Terminal, install packages with the Lubuntu Software Center and browse files. The start menu is per packed with software too,

      Ubuntu

      The full desktop has loads more software pre-installed (and available to install via the software center).

      I opened a few apps and my server only used 472MB ram (it was using about 420MB before I installed lubuntu), all CPU cores were mostly idle.

      Common Folders:

      etc

      Lubuntu Sofware Center

      Installing MySQL workbench

      MySQL Workbench

      I Installed MySQL Workbench

      MySQL Workbench

      More software is available in the lubuntu Software Center

      Synaptic Package Manager

      The Synaptic package manager is also available if you are familiar with that.

      Synaptic

      Troubleshooting

      You may receive this error on startup.

      Error

      Review your: /root/.profile file

      Erroring line

      It is safe to ignore this error message as the command is just trying to generate a suggests exit result. More on the mesg command here.

      Check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

      Enjoy

      Donate and make this blog better




      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      Revision History

      v1.2 added link to hardening linux server.

      v1.1 fixed typos (26th Sep 2017)

      v1.0 Initial Post (26th Sep 2017)

      etc

      Run an Ubuntu VM system audit with Lynis

      by

      Following on from my Securing Ubuntu in the cloud blog post I have installed Lynis open source security audit tool to check out to the security of my server in the cloud.

      Advertisement:


      Lynis is an open source security auditing tool. Used by system administrators, security professionals, and auditors, to evaluate the security defences of their Linux and Unix-based systems. It runs on the host itself, so it performs more extensive security scans than vulnerability scanners. https://cisofy.com/lynis and https://github.com/CISOfy/lynis.

      It is easy to setup a server in the cloud (create a server on Vultr or Digital Ocean here). Guides on setting up servers exist ( setup up a Vultr VM and configure it and digital ocean server) but how about securing it? You can install a LetsEncrypt SSL certificate in minutes or setup Content Security Policy and Public Key Pinning but don’t forget to get an external in-depth review of the security of your server(s).

      Lynis Security Auditing Tool

      Preparing install location (for Lynis)

      Install Lynis

      Running a Lynus system scan

      ./lynis audit system -Q

      Lynis Results 1/3 Output (removed sensitive output)

      [ Lynis 2.5.5 ]

################################################################################
  Lynis comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
  welcome to redistribute it under the terms of the GNU General Public License.
  See the LICENSE file for details about using this software.

  2007-2017, CISOfy - https://cisofy.com/lynis/
  Enterprise support available (compliance, plugins, interface and tools)
################################################################################


[+] Initializing program
------------------------------------
- Detecting OS...  [ DONE ]
- Checking profiles... [ DONE ]

  ---------------------------------------------------
  Program version:           2.5.5
  Operating system:          Linux
  Operating system name:     Ubuntu Linux
  Operating system version:  16.04
  Kernel version:            4.4.0
  Hardware platform:         x86_64
  Hostname:                  yourservername
  ---------------------------------------------------
  Profiles:                  /linis/lynis/default.prf
  Log file:                  /var/log/lynis.log
  Report file:               /var/log/lynis-report.dat
  Report version:            1.0
  Plugin directory:          ./plugins
  ---------------------------------------------------
  Auditor:                   [Not Specified]
  Test category:             all
  Test group:                all
  ---------------------------------------------------
- Program update status...  [ NO UPDATE ]

[+] System Tools
------------------------------------
- Scanning available tools...
- Checking system binaries...

[+] Plugins (phase 1)
------------------------------------
: plugins have more extensive tests and may take several minutes to complete - Plugin pam
    [..]
- Plugin systemd
    [................]

[+] Boot and services
------------------------------------
- Service Manager [ systemd ]
- Checking UEFI boot [ DISABLED ]
- Checking presence GRUB [ OK ]
- Checking presence GRUB2 [ FOUND ]
- Checking for password protection [ OK ]
- Check running services (systemctl) [ DONE ]
: found 24 running services
- Check enabled services at boot (systemctl) [ DONE ]
: found 30 enabled services
- Check startup files (permissions) [ OK ]

[+] Kernel
------------------------------------
- Checking default run level [ RUNLEVEL 5 ]
- Checking CPU support (NX/PAE)
 support: PAE and/or NoeXecute supported [ FOUND ]
- Checking kernel version and release [ DONE ]
- Checking kernel type [ DONE ]
- Checking loaded kernel modules [ DONE ]
active modules
- Checking Linux kernel configuration file [ FOUND ]
- Checking default I/O kernel scheduler [ FOUND ]
- Checking for available kernel update [ OK ]
- Checking core dumps configuration [ DISABLED ]
- Checking setuid core dumps configuration [ PROTECTED ]
- Check if reboot is needed [ NO ]

[+] Memory and Processes
------------------------------------
- Checking /proc/meminfo [ FOUND ]
- Searching for dead/zombie processes [ OK ]
- Searching for IO waiting processes [ OK ]

[+] Users, Groups and Authentication
------------------------------------
- Administrator accounts [ OK ]
- Unique UIDs [ OK ]
- Consistency of group files (grpck) [ OK ]
- Unique group IDs [ OK ]
- Unique group names [ OK ]
- Password file consistency [ OK ]
- Query system users (non daemons) [ DONE ]
- NIS+ authentication support [ NOT ENABLED ]
- NIS authentication support [ NOT ENABLED ]
- sudoers file [ FOUND ]
- Check sudoers file permissions [ OK ]
- PAM password strength tools [ OK ]
- PAM configuration files (pam.conf) [ FOUND ]
- PAM configuration files (pam.d) [ FOUND ]
- PAM modules [ FOUND ]
- LDAP module in PAM [ NOT FOUND ]
- Accounts without expire date [ OK ]
- Accounts without password [ OK ]
- Checking user password aging (minimum) [ DISABLED ]
- User password aging (maximum) [ DISABLED ]
- Checking expired passwords [ OK ]
- Checking Linux single user mode authentication [ OK ]
- Determining default umask
- umask (/etc/profile) [ NOT FOUND ]
- umask (/etc/login.defs) [ SUGGESTION ]
- umask (/etc/init.d/rc) [ SUGGESTION ]
- LDAP authentication support [ NOT ENABLED ]
- Logging failed login attempts [ ENABLED ]

[+] Shells
------------------------------------
- Checking shells from /etc/shells
: found 6 shells (valid shells: 6).
- Session timeout settings/tools [ NONE ]
- Checking default umask values
- Checking default umask in /etc/bash.bashrc [ NONE ]
- Checking default umask in /etc/profile [ NONE ]

[+] File systems
------------------------------------
- Checking mount points
- Checking /home mount point [ SUGGESTION ]
- Checking /tmp mount point [ SUGGESTION ]
- Checking /var mount point [ SUGGESTION ]
- Query swap partitions (fstab) [ NONE ]
- Testing swap partitions [ OK ]
- Testing /proc mount (hidepid) [ SUGGESTION ]
- Checking for old files in /tmp [ OK ]
- Checking /tmp sticky bit [ OK ]
- ACL support root file system [ ENABLED ]
- Mount options of / [ NON DEFAULT ]
- Checking Locate database [ FOUND ]
- Disable kernel support of some filesystems
- Discovered kernel modules: cramfs freevxfs hfs hfsplus jffs2 udf 

[+] Storage
------------------------------------
- Checking usb-storage driver (modprobe config) [ NOT DISABLED ]
- Checking USB devices authorization [ ENABLED ]
- Checking firewire ohci driver (modprobe config) [ DISABLED ]

[+] NFS
------------------------------------
- Check running NFS daemon [ NOT FOUND ]

[+] Name services
------------------------------------
- Searching DNS domain name [ UNKNOWN ]
- Checking /etc/hosts
- Checking /etc/hosts (duplicates) [ OK ]
- Checking /etc/hosts (hostname) [ OK ]
- Checking /etc/hosts (localhost) [ SUGGESTION ]
- Checking /etc/hosts (localhost to IP) [ OK ]

[+] Ports and packages
------------------------------------
- Searching package managers
- Searching dpkg package manager [ FOUND ]
- Querying package manager
- Query unpurged packages [ NONE ]
- Checking security repository in sources.list file [ OK ]
- Checking APT package database [ OK ]
- Checking vulnerable packages [ OK ]
- Checking upgradeable packages [ SKIPPED ]
- Checking package audit tool [ INSTALLED ]

[+] Networking
------------------------------------
- Checking IPv6 configuration [ ENABLED ]
 method [ AUTO ]
 only [ NO ]
- Checking configured nameservers
- Testing nameservers
: 108.xx.xx.xx [ OK ]
: 2001:xxx:xxx:xxx::6 [ OK ]
- Minimal of 2 responsive nameservers [ OK ]
- Checking default gateway [ DONE ]
- Getting listening ports (TCP/UDP) [ DONE ]
* Found 18 ports
- Checking promiscuous interfaces [ OK ]
- Checking waiting connections [ OK ]
- Checking status DHCP client [ NOT ACTIVE ]
- Checking for ARP monitoring software [ NOT FOUND ]

[+] Printers and Spools
------------------------------------
- Checking cups daemon [ NOT FOUND ]
- Checking lp daemon [ NOT RUNNING ]

[+] Software: e-mail and messaging
------------------------------------
- Sendmail status [ RUNNING ]

[+] Software: firewalls
------------------------------------
- Checking iptables kernel module [ FOUND ]
- Checking iptables policies of chains [ FOUND ]
- Checking for empty ruleset [ OK ]
- Checking for unused rules [ FOUND ]
- Checking host based firewall [ ACTIVE ]

[+] Software: webserver
------------------------------------
- Checking Apache (binary /usr/sbin/apache2) [ FOUND ]
: No virtual hosts found
* Loadable modules [ FOUND (106) ]
- Found 106 loadable modules 
- anti-DoS/brute force [ OK ]
- web application firewall [ OK ]
- Checking nginx [ FOUND ]
- Searching nginx configuration file [ FOUND ]
- Found nginx includes [ 2 FOUND ]
- Parsing configuration options
- /etc/nginx/nginx.conf
- /etc/nginx/sites-enabled/default
- SSL configured [ YES ]
- Ciphers configured [ YES ]
- Prefer server ciphers [ YES ]
- Protocols configured [ YES ]
- Insecure protocols found [ NO ]
- Checking log file configuration
- Missing log files (access_log) [ NO ]
- Disabled access logging [ NO ]
- Missing log files (error_log) [ NO ]
- Debugging mode on error_log [ NO ]

[+] SSH Support
------------------------------------
- Checking running SSH daemon [ FOUND ]
- Searching SSH configuration [ FOUND ]
- SSH option: AllowTcpForwarding [ SUGGESTION ]
- SSH option: ClientAliveCountMax [ SUGGESTION ]
- SSH option: ClientAliveInterval [ OK ]
- SSH option: Compression [ SUGGESTION ]
- SSH option: FingerprintHash [ OK ]
- SSH option: GatewayPorts [ OK ]
- SSH option: IgnoreRhosts [ OK ]
- SSH option: LoginGraceTime [ OK ]
- SSH option: LogLevel [ SUGGESTION ]
- SSH option: MaxAuthTries [ SUGGESTION ]
- SSH option: MaxSessions [ SUGGESTION ]
- SSH option: PermitRootLogin [ SUGGESTION ]
- SSH option: PermitUserEnvironment [ OK ]
- SSH option: PermitTunnel [ OK ]
- SSH option: Port [ SUGGESTION ]
- SSH option: PrintLastLog [ OK ]
- SSH option: Protocol [ OK ]
- SSH option: StrictModes [ OK ]
- SSH option: TCPKeepAlive [ SUGGESTION ]
- SSH option: UseDNS [ OK ]
- SSH option: VerifyReverseMapping [ NOT FOUND ]
- SSH option: X11Forwarding [ SUGGESTION ]
- SSH option: AllowAgentForwarding [ SUGGESTION ]
- SSH option: AllowUsers [ NOT FOUND ]
- SSH option: AllowGroups [ NOT FOUND ]

[+] SNMP Support
------------------------------------
- Checking running SNMP daemon [ NOT FOUND ]

[+] Databases
------------------------------------
- MySQL process status [FOUND ]

[+] LDAP Services
------------------------------------
- Checking OpenLDAP instance [ NOT FOUND ]

[+] PHP
------------------------------------
- Checking PHP [ FOUND ]
- Checking PHP disabled functions [ FOUND ]
- Checking expose_php option [ OFF ]
- Checking enable_dl option [ OFF ]
- Checking allow_url_fopen option [ ON ]
- Checking allow_url_include option [ OFF ]
- Checking PHP suhosin extension status [ OK ]
- Suhosin simulation mode status [ OK ]

[+] Squid Support
------------------------------------
- Checking running Squid daemon [ NOT FOUND ]

[+] Logging and files
------------------------------------
- Checking for a running log daemon [ OK ]
- Checking Syslog-NG status [ NOT FOUND ]
- Checking systemd journal status [ FOUND ]
- Checking Metalog status [ NOT FOUND ]
- Checking RSyslog status [ FOUND ]
- Checking RFC 3195 daemon status [ NOT FOUND ]
- Checking minilogd instances [ NOT FOUND ]
- Checking logrotate presence [ OK ]
- Checking log directories (static list) [ DONE ]
- Checking open log files [ DONE ]
- Checking deleted files in use [ FILES FOUND ]

[+] Insecure services
------------------------------------
- Checking inetd status [ NOT ACTIVE ]

[+] Banners and identification
------------------------------------
- /etc/issue [ FOUND ]
- /etc/issue contents [ OK ]
- /etc/issue.net [ FOUND ]
- /etc/issue.net contents [ OK ]

[+] Scheduled tasks
------------------------------------
- Checking crontab/cronjob [ DONE ]
- Checking atd status [ RUNNING ]
- Checking at users [ DONE ]
- Checking at jobs [ NONE ]

[+] Accounting
------------------------------------
- Checking accounting information [ NOT FOUND ]
- Checking sysstat accounting data [ NOT FOUND ]
- Checking auditd [ NOT FOUND ]

[+] Time and Synchronization
------------------------------------
- NTP daemon found: ntpd [ FOUND ]
- NTP daemon found: systemd (timesyncd) [ FOUND ]
- Checking for a running NTP daemon or client [ OK ]
- Checking valid association ID's [ FOUND ]
- Checking high stratum ntp peers [ OK ]
- Checking unreliable ntp peers [ FOUND ]
- Checking selected time source [ OK ]
- Checking time source candidates [ OK ]
- Checking falsetickers [ OK ]
- Checking NTP version [ FOUND ]

[+] Cryptography
------------------------------------
- Checking for expired SSL certificates [0/1] [ NONE ]

[+] Virtualization
------------------------------------

[+] Containers
------------------------------------

[+] Security frameworks
------------------------------------
- Checking presence AppArmor [ FOUND ]
- Checking AppArmor status [ ENABLED ]
- Checking presence SELinux [ NOT FOUND ]
- Checking presence grsecurity [ NOT FOUND ]
- Checking for implemented MAC framework [ OK ]

[+] Software: file integrity
------------------------------------
- Checking file integrity tools
- Checking presence integrity tool [ NOT FOUND ]

[+] Software: System tooling
------------------------------------
- Checking automation tooling
- Automation tooling [ NOT FOUND ]
- Checking presence of Fail2ban [ FOUND ]
- Checking Fail2ban jails [ ENABLED ]
- Checking for IDS/IPS tooling [ FOUND ]

[+] Software: Malware
------------------------------------

[+] File Permissions
------------------------------------
- Starting file permissions check
/root/.ssh [ OK ]

[+] Home directories
------------------------------------
- Checking shell history files [ OK ]

[+] Kernel Hardening
------------------------------------
- Comparing sysctl key pairs with scan profile
- fs.protected_hardlinks (exp: 1) [ OK ]
- fs.protected_symlinks (exp: 1) [ OK ]
- fs.suid_dumpable (exp: 0) [ DIFFERENT ]
- kernel.core_uses_pid (exp: 1) [ DIFFERENT ]
- kernel.ctrl-alt-del (exp: 0) [ OK ]
- kernel.dmesg_restrict (exp: 1) [ DIFFERENT ]
- kernel.kptr_restrict (exp: 2) [ DIFFERENT ]
- kernel.randomize_va_space (exp: 2) [ OK ]
- kernel.sysrq (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.all.accept_redirects (exp: 0) [ OK ]
- net.ipv4.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv4.conf.all.bootp_relay (exp: 0) [ OK ]
- net.ipv4.conf.all.forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.log_martians (exp: 1) [ DIFFERENT ]
- net.ipv4.conf.all.mc_forwarding (exp: 0) [ OK ]
- net.ipv4.conf.all.proxy_arp (exp: 0) [ OK ]
- net.ipv4.conf.all.rp_filter (exp: 1) [ OK ]
- net.ipv4.conf.all.send_redirects (exp: 0) [ DIFFERENT ]
- net.ipv4.conf.default.accept_redirects (exp: 0) [ OK ]
- net.ipv4.conf.default.accept_source_route (exp: 0) [ OK ]
- net.ipv4.conf.default.log_martians (exp: 1) [ DIFFERENT ]
- net.ipv4.icmp_echo_ignore_broadcasts (exp: 1) [ OK ]
- net.ipv4.icmp_ignore_bogus_error_responses (exp: 1) [ OK ]
- net.ipv4.tcp_syncookies (exp: 1) [ DIFFERENT ]
- net.ipv4.tcp_timestamps (exp: 0) [ DIFFERENT ]
- net.ipv6.conf.all.accept_redirects (exp: 0) [ OK ]
- net.ipv6.conf.all.accept_source_route (exp: 0) [ OK ]
- net.ipv6.conf.default.accept_redirects (exp: 0) [ OK ]
- net.ipv6.conf.default.accept_source_route (exp: 0) [ OK ]

[+] Hardening
------------------------------------
- Installed compiler(s) [ FOUND ]
- Installed malware scanner [ NOT FOUND ]

[+] Custom Tests
------------------------------------
- Running custom tests...  [ NONE ]

[+] Plugins (phase 2)
------------------------------------
- Plugins (phase 2) [ DONE ]

================================================================================

...

      Lynis Results 2/3 – Warnings

      I resolved the only warning by typing

      After updating the Lynis system scan I re ran the text and got

      Lynis Results 3/3 – Suggestions

      Installing a Malware Scanner

      Install ClamAV

      Download virus and malware definitions (this takes about 30 min)

      Output:

      sudo freshclam
> ClamAV Update process started at Wed Nov 15th 20:44:55 2017
> Downloading main.cvd [10%]

      I had an issue on some boxes with clamav reporting I could not run freshclam

      sudo freshclam
ERROR: /var/log/clamav/freshclam.log is locked by another process
ERROR: Problem with internal logger (UpdateLogFile = /var/log/clamav/freshclam.log).

      This was fixed by typing

      Troubleshooting clamav

      Clam AV does not like low ram boxes and may produce this error

      It looks like the solution is to increase your total ram.

      fyi: Scan with ClamAV

      Re-running Lynis gave me the following malware status

      Lynis Security rating

      Installed

      After re-running the test I got this Lynis security rating score (an improvement of 1)

      Installed and configured debsums and auditd

      Now I get the following Lynis security rating score.

      Conclusion

      Lynis is great great at performing an audit and reccomending areas of work to allow you to harden your system (brute force protection, firewall, etc)

      Security Don’ts

      • Never think you are done securing a system.

      Security Do’s

      • Update Software (and remove software you do not use.)
      • Check Lynis Suggestions and try and resolve.
      • Security is an ongoing process, Do install a firewall, do ban bad IP’s, Do whitelist good IP’s, Do review Logs,
      • Do limit port access, make backups and keep on securing.

      I will keep on securing and try and get remove all issues.

      Read my past post on Securing Ubuntu in the cloud.

      Scheduling a auto system updates is not enough in Ubuntu (as it is not recommended as the administrator should make decisions, not a scheduled job).

      apt-get update
apt-get upgrade

      fyi: CISOFY/Lynis do have paid subscriptions to have external scans of your servers: https://cisofy.com/pricing. (why upgrade?)

      Lynis Plans

      I will look into this feature soon.

      Updating Lynis

      I checked the official documentation and ran an update check

      ./lynis --check-update
This option is deprecated
Use: lynis update info

./lynis update info

 == Lynis ==

  Version            : 2.5.5
  Status             : Outdated
  Installed version  : 255
  Latest version     : 257
  Release date       : 2017-09-07
  Update location    : https://cisofy.com/lynis/


2007-2017, CISOfy - https://cisofy.com/lynis/

      Not sure how to update?

      ./lynis update
Error: Need a target for update

Examples:
lynis update check
lynis update info

./lynis update check
status=outdated

      I opened an issue about updating v2.5.5 here. I asked Twiter for help.

      Twitter

      Official Response: https://packages.cisofy.com/community/#debian-ubuntu

      Git Response

      Waiting..

      I ended up deleting Lynis 2.5.5

      Updated

      And reinstalled to v2.5.8

      Output:

      More actions post upgrade to 2.5.8

      • Added a legal notice to “/etc/issues”, “/etc/issues.net” file’s.

      Installing Lynis via apt-get instead of git clone

      The official steps can be located here: https://packages.cisofy.com/community/#debian-ubuntu

      apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C80E383C3DE9F082E01391A0366C67DE91CA5D5F
apt install apt-transport-https
echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/99disable-translations
echo "deb https://packages.cisofy.com/community/lynis/deb/xenial main" > /etc/apt/sources.list.d/cisofy-lynis.list
apt update
apt install lynis
lynis show version

      Unfortunately, I had an error with “apt update”

      Error:

      Complete install output

      I reopened Github issue 491. A quick reply revealed that I did not put a space before “xenial” (oops)

      fyi: I removed the dead keystore from apt by typing…

      I can now install and update other packages with apt and not have the following error

      I will remove the git clone and re-run the apt version later and put in more steps to get to a High 90’s Lynis score.

      More

      Read the official documentation https://cisofy.com/documentation/lynis/

      Next: This guide will investigate the enterprise version of https://cisofy.com/pricing/ soon.

      Hope this helps. If I have missed something please let me know on Twitter at @FearbySoftware

      Donate and make this blog better



      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      v1.46 Git hub response.

      Installing Redis 3.x onto Ubuntu 16.04

      by

      This post will show you how to install Redis 3.x onto Ubuntu 16.04

      Advertisement:



      Redis is a server-side (Lua based) schema-free open source (in Memory) NoSQL Key/Value store database that supports replication between servers. Redis has an Eventual Consistency replication between servers (over Immediate Consistency) between servers. Eventual consistency is evil in some peoples minds, eventual consistency does require different coding considerations (to guaranteed valid data but does offer speed benefits).

      Redis is the worlds 9th most popular database behind Oracle, MySQL, Microsoft SQL Server, PostgreSQL, MongoDB, DB2, Microsoft Access, Cassandra. Redis is the most popular key-value store database. View the database trend chart here.

      When to use Redis over MySQL or MongoDB

      Here is a great guide on when to use Redis over MongoDB. Read my previous guide on building a scalable and secure MySQL Cluster.

      Redis has a place where a relational database or NoSQL document stores do not (but it is more work). Read more here.

      Redis is great in situations where caching or where memory is available on the server.

      Free Memory

       

      Installing Redis on Ubuntu

      Backup the conf file

      Open the redis cli (and test it).

      Show Redis info (from the redis-cli)

      Benchmarking your Redis (quick)

      Benchmarking your Redis (slow)

      How to save and query a simple key/value integer

      How to save and query a simple key/value string

      Saving and querying multiple values

      Clear all

      Add to list (new items at the top)

      127.0.0.1:6379> lpush testlist "a"
(integer) 1
127.0.0.1:6379> lpush testlist "b"
(integer) 2
127.0.0.1:6379> lpush testlist "c"
(integer) 3
127.0.0.1:6379> lrange testlist 0 -1
1) "c"
2) "b"
3) "a"

      Add to list (new items at the bottom)

      Redis Documentation

      redis.io

      redis.io/­documentation

      https://redis.io/commands/

      Redis Crash Course Tutorial

      More

      5 uses of redis as a database.

      Using Redis at scale at Twitter

      Digital Ocean guide on Redis

      Coming soon PHP access to Redis.

       

      Donate and make this blog better




      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      v1.2 Removed reference to Redis 4 in the blurb.

      v1.1 Oops, this was Redis 3 and not 4 (thanks, Patrik)

      v1.0 Initial version

      How to setup and secure MongoDB on Ubuntu 16.04 and verify with Studio 3T

      by

      Here is how you can install MongoDB 3.4.x on Ubuntu 16.04 and secure it. I use the Studio 3T software form Studio 3T.

      Advertisement:



      Before you install MongoDB ensure you have secured your server and installed it and installed an SSL certificate. Click the links here to set up a Digital Ocean, Vultr or AWS Server. Read my old guide to installing MongoDB on Ubuntu 14.04  and using Studio 3T ( https://studio3t.com/ ).

      Install MongoDB 3.6

      Official guide here

      Add the keyserver

      Update

      Install the latest stable version of MongoDB

      sudo apt-get install -y mongodb-org

      MongoDB.conf options

      FYI: Secure MongoDB: https://docs.mongodb.com/manual/security/ and Security Checklist https://docs.mongodb.com/manual/administration/security-checklist/

      MongoDB Hardening Info: https://docs.mongodb.com/manual/core/security-hardening/

      Start Mongodb

      You should see startup activity

      In a new terminal window open a MongoDB process

      mongo --port 27017 --authenticationDatabase 'admin' --username username --password password
>...
>

      Create a user

      Create a test db

      Run MongoDB at Startup

      Note: I tried setting up a service but it failed so I added the following command to /etc/rc.local

       

      Todo: Service Setup.

      Ignore the follwoing….

      Create a service

      I added

      TIP: Bing to local 127.0.0.1 and also your external IP (if you have hardened and setup IP whitelists for the port).

      Make /etc/init.d/mongod executable

      Firewall

      Configure your firewall (tip: whitelist your local development IP to allow your IP access etc)

      Reload the firewall and show the status. Add port 27017 (or your custom port) to your TCP and UDP firewall. http://icanhazip.com/ will display your public IP.

      Display the MongoDB version

      Make MongoDB configuration changes

      – Consider saving your database data somewhere else (mongodb.conf)

      – Consider redirecting your log file (mongodb.conf)

      systemLog:
  destination: file
  logAppend: true
  path: "/folder_to_store_mongodb_logs/mongo.log"

      – Consider changing the default MongoDB port (mongodb.conf)

      net:
  port: 27123

      – Allow MongoDB to talk locally and globally if need be and optionally enable IPV6 (binding IP’s in mongodb.conf)

      TIP: Ensure you bind you localhost port (127.0.0.1) AND your public IP (e.g 123.123.123.123) as you will need to bind to public IP too if you want to connect to MongoDB externally.  I did not bing my eternal IP and was blocked for a few days.

      If you allow external access then consider whitelisting your IP and disabling local admin login – more here  (mongodb.conf)

      Official configuration documentation can be found here.

      At this stage, MongoDB is open to the world and if you connect to your server with no username or password you will see it is open.

      mongodb openI created an admin user with Studio 3T for MongoDB in the IntelliShell.

      Create MOngoDB USERI typed the following to create a user (I added a root role after creating the screenshot above).

      Verify that the user was created

      Now you can use these credentials to log in to the database.

      use login details

      You can see the new credentials are working and now we need to remove anonymous and empty connections.

      Add the following to mongodb.conf

      Restart MongoDB

      or

      /usr/bin/mongod --config /etc/mongod.conf

      Now when you connect to your database with no login details you will see no databases.

      all secure

      Show the status of MongoDB

      View the last 20 lines of the MongoDB log file.

      (or replace the path with your log location)

      tail -n 20 /yourmongodb_logs/mongod.log

      Viewing MongoDB files

      MongoDB Users and Roles

      More on securing MongoDB here and here.

      Remove MongoDB

      Remove MongoDB files

      cd /mongodb_data_folder/
rm -R *.*
rm -R WiredTiger
rm -R journal

      Remove MongoDB

      I had these warnings with MongoDB 3.2

      I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
I CONTROL  [initandlisten] **        We suggest setting it to 'never'

      I ran this fix.

      Installing MongoDB on Ubuntu 16.04 guide here.  More on SSL and MongoDB here.

      todo: MongoDB Startup at reboot troubleshooting steps.

      Good luck.

       

      Donate and make this blog better




      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      v1.6 Added MongoDB 3.6 install info and strat at startup

      v1.5 MongoDB troubleshooting

      How to be alerted after system boot on Ubuntu 16.04 with an email via Gmail

      by

      This will allow you to sent an email at startup on Ubuntu boot. You will need to ensure sendmail is setup and working (read my guide on How to send email via G Suite from Ubuntu in the cloud, setup an Ubuntu server in the cloud here (guide here)).

      Advertisement:



      Create a  scripts folder

      mkdir /scripts/

      Create a file called /scripts/emailstartup.sh and add..

      optional: Uncomment lines above to attach a zip file instead of a log file (don’t forget to attach the zip instead of the log file in sendmail.)

      Make the script file executable

      sudo chmod +X /scripts/emailstartup.sh

      Test the script

      Add the following to crontab -e to ensure the script is executed 5 minutes after startup.

       

      On reboot, you will be emailed desired start-up information.

      email capture

       

       

      Donate and make this blog better




      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      v1.0 initial post

      Moving a CPanel domain with email to a self managed VPS and Gmail

      by

      Below is my guide for moving away from NetRegistry CPanel domain to a self-managed server and GSuite email.

      Advertisement:


      I have had www.fearby.com since 1999 on three CPanel hosts (superwerbhost in the US, Jumba in Australia, Uber in Australia (NetRegistry have acquired Uber and performance at the time of writing is terrible)). I was never informed by Uber of the sale but my admin portal was moved from one host to another and each time performance degraded. I tried to speed up WordPress by optimizing images, installing cache plugins but nothing worked, pages were loading in around 24 seconds on https://www.webpagetest.org.

      Buy a domain name from Namecheap here.

      Domain names for just 88 cents!

      I had issues with a CPanel domain on the hosts (Uber/Netregistry) as they were migrating domains and the NetRegstry chat rep said I needed to phone Uber for support. No thanks, I’m going self-managed and saving a dollar.

      Advertisement:



      I decided to take ownership of my slow domain and setup my own VM and direct web traffic to it and redirect email to GMail (I have done this before).  I have setup Digital Ocean VM’s (Ubuntu and Centos), Vultr VM’s and AWS VM’s.

      I have also had enough of Resource Limit Reached messages with CPanel and I can’t wait to…

      • not have a slow WordPress.
      • setup my own server (not a slow hosts server).
      • spend $5 less (we currently pay $25 for a CPanel website with 20GB storage total)
      • get a faster website (sub 24 seconds load time).
      • larger email mailboxes (30GB each).
      • Generate my own “SSL Labs A+ rated” certificate for $10 a year instead of $150 a year for an “SSL Labs C rated” SSL certificate from my existing hosts.

      Backup

      I have about 10 email accounts on my CPanel domain (using 14GB) and 2x WordPress sites.  I want to backup my emails with (Outlook Export and Thunderbird Profile backup) and backup my domain file(s) a few times before I do anything.  Once DNS is set in motion no server waits.

      The Plan

      Once everything is backed up I intend to setup a $5 a month Vulr VM and redirect all mail to Google G Suite (I have redirected mail before).

      I will setup a Vultr web server in Sydney (following my guide here), buy an  SSL certificate from Namecheap and move my WordPress sites.

      Rough Plan

      • Reduce email accounts from 10x to 3x
      • Backup emails (twice with ThunderBird and Outlook).
      • Setup A Ubuntu V on Vultr.
      • Signup for Google G Suite Trial.
      • Transfer my domain to Namecheap.
      • Link to domain DNS to Vultr
      • Link to domain MX records to Google Email.
      • Transfer website.
      • Setup emails on google.
      • Restore WordPress.
      • Go live.
      • Downgrade to personal G Suite before the trial expires
      • Close down the old server.

      Signing up for Google G Suite

      I visited https://gsuite.google.com/ and started creating an account.

      Get 20% off your first year by signing up for Google G Suite using this link: https://goo.gl/6vpuMm

       

      Screenshots of Google G Suite setup

      I created a link between G Suite and an existing GMail account.

      More screenshots of Google G suite setup

      Now I can create the admin account.

      Picture of G suite asking how i will log in

      Tip: Don’t use any emails that are linked as secondary emails with any Google services (this won’t be allowed). It’s s a well-known issue that you cannot add users emails who are linked to Google services (even as backup emails for Gmail, detach the email before adding it). Read more here.

      Google G suite did not like my email provided

       

      Final setup steps.

      Final G suite setup screenshots.

      Now I can add email accounts to G Suite.

      G Suite said im all ready to add users

      Adding email users to G Suite.

      G Suite adding users

      The next thing I had to do was upload a file to my domain to verify I own the domain (DNS verification is also an option).

      I must say the setup and verify steps are quite easy to follow on G Suite.

      Time to backup our existing CPanel site.

      Screenshot of Cpanel users

       

      Backup Step 1 (hopefully I won’t need this)

      I decided to grab a complete copy of my CPanel domain with domains, databases and email accounts. This took 24 hours.

      CPanel backup screenshot

      Backup Step 2 (hopefully I won’t need this)

      I download all mail via IMAP in Outlook and Mozilla Thunderbird and export it (Outlook Export and Thunderbird Profile backup). Google have IMAP instructions here.

      DNS Changes at Namecheap

      I obtained my domain EPP code from my CPanel hosts and transferred the domain name to Namecheap.

      Namecheap was even nice enough to set my DNS point to my existing domain so I did not have to rush a move before DNS propagation.

      P.S The Namecheap Chat Staff and Namecheap  Mobile App is awesome.

      NameCheap DNS

      Having backed up everything I logged into Namecheap and set my DNS to “NameCheap BasicDNS” and then went “Advanced DNS” and set appropriate DNS records for my domain. This assumes you have setup a VM with IPV4 and IPV6 (follow my guide here).

      • A Record @ IPV4_OF_MY_VULTR_SERVER
      • A Record www IPV4_OF_MY_VULTR_SERVER
      • A Record ftp IPV4_OF_MY_VULTR_SERVER
      • AAAA Record @ IPV6_OF_MY_VULTR_SERVER
      • AAAA Record www IPV6_OF_MY_VULTR_SERVER
      • AAAA Record ftp IPV6_OF_MY_VULTR_SERVER
      • C Name www fearby.com

      The Google G Suite also asked me to add these following MX records to the DNS records.

      • MX Record @ ASPMX.L.GOOGLE.COM. 1
      • MX Record @ ASPMX1.L.GOOGLE.COM. 5
      • MX Record @ ASPMX2.L.GOOGLE.COM. 5
      • MX Record @ ASPMX3.L.GOOGLE.COM. 10
      • MX Record @ ASPMX4.L.GOOGLE.COM. 10

      Then it was a matter of telling Google DNS changes were made (once DNS has replicated across the US).

      My advice is to set DNS changes before bed as it can take 12 hours.

      Sites like https://www.whatsmydns.net/ are great for keeping track of DNS replication.

      Transferring WordPress

      I logged into the CPanel and exported my WordPress Database (34MB SQL file).

      I had to make the following PHP.ini changes to allow the larger file size restore uploads with the Adminer utility (default is 2mb). I could not get the server side adminer.sls.gz option to restore the database?

      I had to make the following changes to nginx.conf (to prevent 404 errors on the database upload)

      I also had to make these changes to NGINX (sites-available/default) to allow WordPress to work

      I had a working MySQL (I followed my guide here).

      Adminer is the best PHP MySQL management utility (beats PhpMyAdmin hands down).

      Restart NGINX and PHP

      I had an error on database import, a non-descript error in script line 1 (error hint here).

      A simple search and replace in the SQL fixed it.

      Once I had increased PHP uploads to 50M and Nginx I was able to upload my database backup with Adminer  (just remember to import to the created database that matches. the wp-config.php. Also, ensure your WordPress content is in place too.

      The only other problem I had was WordPress gave an “Error 500” so moved   few plugins an all was good.

      Importing Old Email

      I was able to use the Google G Suite tools to import my old Mail (CPanel IMAP to Google IMAP).

      Import IMAP mail to GMail

      I love root access on my own server now, goodbye CPanel “Usage Limit Exceeded” errors (I only had light traffic on my site).

      My self-hosted WordPress is a lot snappier now, my server has plenty of space (and only costs $0.007c and hour for 1x CPU, 1GB ram, 25GB SSD storage and 1000GB data transfer quota). I use the htop command to view system processor and disk space usage.

      I can now have more space for content and not be restricted by tight hosts disk quotas or slow shared servers.  I use the pydf command to view dis space.

      I use ncdu to view folder usage.

      Installing ncdu

      Type ncdu in the folder you want to browse under.

      You can arrow up and down folder structures and view folder/file usage.

      SSL Certificate

      I am setting up a new multi year SS cert now, I will update this guide later.  I had to read my SSL guide with Digital Ocean here.

      I generated some certificate on my server

      Sample output for  a new certificate

      openssl req -new -newkey rsa:4096 -nodes -keyout dummy.key -out dummy.csr
Generating a 4096 bit RSA private key
.................................................................................................++
......++
writing new private key to 'dummy.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]: AU
State or Province Name (full name) [Some-State]: NSW
Locality Name (eg, city) []:Tamworth
Organization Name (eg, company) [Internet Widgits Pty Ltd]: Dummy Org
Organizational Unit Name (eg, section) []: Dummy Org Dept
Common Name (e.g. server FQDN or YOUR name) []: DummyOrg
Email Address []: [email protected]

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []: password
An optional company name []: DummyCO
[email protected]:~/sslcsrmaster4096# cat dummy.csr
-----BEGIN CERTIFICATE REQUEST-----
MIIFAjCCAuoCAQAwgYsxCzAJBgNVBAYTAkFVMQwwCgYDVQQIDANOU1cxETAPBgNV
BAcMCFRhbXdvcnRoMRIwEAYDVQQKDAlEdW1teSBPcmcxFzAVBgNVBAsMDkR1bW15
IE9yZyBEZXB0MREwDwYDVQQDDAhEdW1teU9yZzEbMBkGCSqGSIb3DQEJARYMbWVA
ZHVtbXkub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA6PUtWkRl
+gL0Hx354YuJ5Sul2Xh+ljILSlFoHAxktKlE+OJDJAtUtVQpo3/F2rGTJWmmtef+
shortenedoutput
swrUzpBv8hjGziPoVdd8qdAA2Gh/Y5LsehQgyXV1zGgjsi2GN4A=
-----END CERTIFICATE REQUEST-----

       

      I then uploaded the certificate to Namecheap for an SSL cert registration.

      I selected DNS C Name record as a way to verify I own my domain.

      I am now waiting for Namecheap to verify my domain

      End of the Google G Suite Business Trial

      Before the end of the 14-day trial, you will need to add billing details to keep the email working.

      At this stage, you can downgrade from a $10/m business account per user to a $5/m per user account if you wish. The only loss would be storage and google app access.

      Get 20% off your first year by signing up for Google G Suite using this link: https://goo.gl/6vpuMm

      Before your trial ends, add your payment details and downgrade from $10/user a month business prices to $5/iser a month individual if needed.

      G Suite Troubleshooting

      I was able to access new G Suite email account via gmail.com but not via Outlook 2015? I reset the password, followed the google troubleshooting guide and used the official incoming and outgoing settings but nothing worked.

      troubleshooting 1

      Google phone support suggested I enable less secure connection settings as Google firewall may be blocking Outlook. I know the IMAP RFC is many years old but I doubt Microsoft are talking to G Suite in a lazy manner.

      Now I can view my messages and I can see one email that said I was blocked by the firewall. Google phone support and faqs don’t say why Outlook 2015 SSL based IMAP was blocked?

      past email

      Conclusion

      Thanks to my wife who put up with my continual updates over the entire domain move. Voicing the progress helped me a lot.

      Donate and make this blog better




      Ask a question or recommend an article
      [contact-form-7 404 "Not Found"]

      V1.8 added ad link