IoT, Code, Security, Server Stuff etc

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

Analytics

Setting web push notifications in WordPress with OneSignal

by

Note: I will update this guide to include full WordPress push integration soon (this method works with any HTML website)

When visiting a friends website ( https://markontech.com/ ) I noticed he had push notifications setup.

Screenshot of push notification ability at https://markontech.com/

I was able to click a bell notification allow push notifications to be sent when he posts new content. Nice

When I clicked subscribe Google Chrome prompted me to allow or block notifications.

Screenshot showing Chrome asking for permissions to receive push notifications

A floating icon allows me to unsubscribe at anytime and manage subscription details.

Screehshot showing an unsubscribe icon for removing push notifications

Mark from https://markontech.com/ said he was using a WordPress plugin from OneSignal.

I checked out One Signal from a coding point of view in 2015 (read here) and apparently they now have a WordPress Plugin.

Who knew meme (girl with amazed look and hands up)

Price?

The One Signal blog post says is free under 30,000 subscribers then it jumps to $99/m for above 30,000 subscribers

Pricing table screenshot from https://onesignal.com/pricing

You will need to pay if you want full GDPR compliance (read more here).

Installing the OneSignal Plugin

I visited the OneSignal Plugin page at https://wordpress.org/plugins/onesignal-free-web-push-notifications/

Screenshot of https://onesignal.com Push plugin page at WordPress

I noted the latest plugin URL path

https://downloads.wordpress.org/plugin/onesignal-free-web-push-notifications.1.16.16.zip

I downloaded the zip file (from the command line)

wget https://downloads.wordpress.org/plugin/onesignal-free-web-push-notifications.1.16.16.zip
--2019-03-19 19:24:33--  https://downloads.wordpress.org/plugin/onesignal-free-web-push-notifications.1.16.16.zip
Resolving downloads.wordpress.org (downloads.wordpress.org)... 198.143.164.250
Connecting to downloads.wordpress.org (downloads.wordpress.org)|198.143.164.250|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6764683 (6.5M) [application/octet-stream]
Saving to: 'onesignal-free-web-push-notifications.1.16.16.zip'

onesignal-free-web-push-notifications.1.1 100%[=====================================================================================>]   6.45M  --.-KB/s    in 0.1s

2019-03-19 19:24:33 (54.2 MB/s) - 'onesignal-free-web-push-notifications.1.16.16.zip' saved [6764683/6764683]

I unzipped the plugin

unzip onesignal-free-web-push-notifications.1.16.16.zip

I could now activate the One Signal Plugin in WordPress

Scrreenshot of the non activated WordPress plugin

A One Signal menu appeared in my WordPress Dashboard. Nice

Screenshot of the OneSignal menu in WordPress

I was prompted to add the API key etc to the plugins configuration.

Screesnhot of OneSignal asking for setup to continue.

I logged into OneSignal and created an app.

Screenshot of https://onesignal.com to create new app screen

I selected Web Push

Screenshot of https://onesignal.com choosing a push notification template (web push)

Major Issue

Long story, I at first selected WordPress Plugin or Website builder and finished the setup but my CDN was picking up the java script files and moving it to my CDN automatically and the One Push code was invoking a Web Service error. Upon investigation web services need to be run from the same origin.

I need top setup the One Signal separate to WordPress.

Later I will see if I can exclude my CDN from caching the JavaScript but for now I will delete the WordPress plugin I just downloaded (above) and instead setup a Typical Site (with HTML). My CDN (read here) is very fast so I will keep it.

I following on from the wizard above (after selecting Web Push) I selected Typical site for the OneSignal app. Having a typical site setup will force me to create a push notification manually after each post but that’s ok (gives me time to proof read).

Screenshot of a typical site template chosen

During the app creation process at One Signal I set my site name and URL.

Screenshot of me choosing a site name and description at https://onesignal.com

Now I need to create a prompt for the app.

Screenmshot of me setting up a prompt at https://onesignal.com

I set the prompt style (floating button)

Screenshot of me choosing the prompt style (floating button)

Permission prompt created.

Permission prompt created and saved

Now I copied the generated HTML code that One Signal created and added it to the <head> section of my WordPress theme.

Screnshot of html code to make the push notification work.

I opened the WordPress Dashboard and opened my theme “Header and Footer scripts” section and pasted the code from OneSignal.

Screenshot of me pasting code from OneSignal into WordPress head area.

I published the changes and loaded my site.

The button was not working and when I opened the Google Chrome Developer console I noticed I needed to upload some java script files to my site.

Screenshot of errors complaining of missing recources.

Oops: I should have read all the documentation (here)

I followed the prompts and downloaded the One Signal SDK files (from Github) to place on my site from https://github.com/OneSignal/OneSignal-Website-SDK/releases/download/https-integration-files/OneSignal-Web-SDK-HTTPS-Integration-Files.zip

Screenshot of OneSignal prompting me to download https://github.com/OneSignal/OneSignal-Website-SDK/releases/download/https-integration-files/OneSignal-Web-SDK-HTTPS-Integration-Files.zip

I downloaded the zip file on my server (over SSH command Line), I extracted the zip file contents into the root of my site (don’t forget to chown the files if the downloaded was from another account (I forgot and my nginx web server would not deliver the files)).

I reloaded my site and I now have Push notification subscription features for my readers.

My browser asked to to allow the notifications 🙂

Clicking the bell again allows you to unsubscribe.

Testing Push Notifications

Ok, lets send a notification for the last post I blogged about.

I logged into onesignal.com and clicked Messages. I clicked New Message and specified a Message title and body.

Screenshot of https://app.onesignal.com/apps

I set an icon, image, URL (when clicked) and priority. The icon needed to be 192×192 pixels.

Screen shot showing the ability to set an icon, image and priority.

I can sent instantly or at a scheduled time

Send Instantly or at a scheduled time and submit button.

A confirmation box appears before sending.

A confirmation box appears before sending.

A notification took a few seconds to appear on my Windows 10 machine.

Windows 10 push notification received

The notification also appeared on my android

Statistics of real time read receipts started coming in. I don’t think this is accurate as the android read was not shown

Statistics showing reads and not read

Nice

Adding an Auto Prompt

To enable auto prompt (to send notifications) on visitors page load I added a second prompt alongside the bell.

I re edited the app in One Signal and added an auto prompt (prompt).

One Signal can also ask the user if they want to receive prompts automatically.

Done

One Signal prompts listing a) bell icon and b) auto push

This is what happens on a new Browser (Firefox), the prompt was not triggered on Google Chrome because I was already subscribed and unsubscribing would not trigger it.

Auto push notification prompt screenshot in Firefox.

Mobile App

I did check for an Android App for One Signal but only poor third party apps were found.

Troubleshooting

Update History

v1.1 Added “I will update this guide to include full WordPress push integration soon”

v1.2 added auto prompt information v1.0 Initial version

Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution

by

This is how I replacing Google Analytics with Piwik/Matomo for a locally hosted privacy-focused open source analytics solution

Aside

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. PHP is my programming language of choice.

Now on with the post

Google Analytics

I will fully admit Google Analytics is good. I posted this a while ago on how you can set up Google Analytics on your site.

Google Analytics has some great charts and graphs. Simple to set up and easy to use.

Analitics Home

My site traffic is growing and I would prefer to hold my own analytics on user data. Matomo is an analytics solution that stays on my server and not in the hands of Google.

Blog Growth

Google Analytics can be Slow

Sometimes the Google Analytics server is slow (affecting the speed of my server). I blogged recently about speeding up a WordPress site here and Google Servers were not adding expiry headers on assets.

I did log a ticket with Google to fix this and the experience was terrible.

Support for Google Analytics is terrible

Gogole Analytics support of terrible

GT Metrix scores show poor delivery of tracking assets.

Google Slow Assets

Privacy

After the Cambridge Analytica fiasco (that made me decide to delete facebook) sending analytics to Google is not a good idea.

I am not saying Google is evil but I want my site’s visitors tracking data to remain local.

Website Speed Benchmark before installing Matomo

I can load my site in 1.3 seconds at best, 1.5 seconds on average and 2.0 seconds at worst. My site is loading 11 assets.

GTmetrix 1.3 second page load time

Page Speed Scores

GTMerix page speed load times

Y Slow Scores, Gogol Assets are reporting no expiry headers (slowing down scores)

GTMetrix yslow load times

Google Analytics tracking assets are slow.

Gmetrix waterfall list

Optimizations to be made

Browser caching is not possible with Google Analytics.

Gogole lacking browser caching

Missing Expiry Headers (I can see a Google Tag Manager server is slowing down my servers benchmark score)

Google lacking Expiry Headers

Why Mamoto (instead of Google Analytics)

I came across

Mamoto was mentioned

I visited https://matomo.org/

Mamoto webpage

Snip

> Take care of running Matomo yourself by installing it on your own server. There is no cost for Matomo itself but you need a server and update Matomo & your server regularly to keep it fast and secure. Need help? The Matomo team provides free help resources and paid support.

Mamoto Setup Instruction Guide

Source Code

Source code is available.

> Matomo is the leading open alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites, apps & the IoT and visualise this data and extract insights. Privacy is built-in. We love Pull Requests! https://matomo.org/

https://github.com/matomo-org/matomo

Installation Guide

I read the installation guide here https://matomo.org/docs/installation/

You can view the changelog here https://matomo.org/changelog/

Downloading Mamoto

I logged into my server via SSH and downloaded the 18MB download to the desired folder

I unzipped the zip file

I loaded the URL where Matoto was installed (e.g “https://fearby.com/folder/subfolder/matomo/”)

I received this well-crafted error.

Matomo File Permission Error

Raw Output

I refreshed the page after running the commands above on my site (via SSH)

Matomo Setup Step 1

A system check was performed. I installed when PHP 7.2.11 was the latest, PHP 7.2.12 or higher might be available. Follow my guide to update PHP on Ubuntu.

System Check

I had one Issue with Freetype not being installed.

Install Freetype

I solved this error by installing FreeType

Output

Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'freetype-tools' for glob 'freetype*'
Note, selecting 'freetype2-demos' for glob 'freetype*'
The following NEW packages will be installed:
  freetype2-demos
0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded.
Need to get 123 kB of archives.
After this operation, 728 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 freetype2-demos amd64 2.8.1-2ubuntu2 [123 kB]
Fetched 123 kB in 0s (965 kB/s)
Selecting previously unselected package freetype2-demos.
(Reading database ... 122574 files and directories currently installed.)
Preparing to unpack .../freetype2-demos_2.8.1-2ubuntu2_amd64.deb ...
Unpacking freetype2-demos (2.8.1-2ubuntu2) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up freetype2-demos (2.8.1-2ubuntu2) ...

Then I installed “php-gd”

Output:

I refreshed the Matomo setup wizard page, Freetype is now installed 🙂

FreeType is installed

Database Settings

For the life of me, I could not get Matomo to talk to a database on another server so I set it up on my localhost.

I used this guide to help in mysql CLI to create the database and users.

Enter Matomo Database settings

Commands in mysql to create a database and user and assign the user to the database. If you are not comfortable with MySql CLI you can use Adminder GUI.

I used this PHP code to test connecting to the dedicated server before using the localhost

<?php
$servername = "localhost";
$username = "databaseuser";
$password = "#################";
$dbname = "tbdatabasename";

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);

// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
} else {
        echo "Connection Success";
}

$conn->close();
?>

Database created ok

Database OK

I created a Matomo user then I grabbed the javascript tracking ID code so I could paste this into WordPress.

Matomo Tracking ID

I opened my WordPress theme settings and deleted the Google tracking tags and added the Matomo tracking code.

Delete Google Tracking Tags

I added the Matomo tracking javascript in the head section.

The dashboard is up and collecting data.

Matomo Dashboard

Some reports are missing data so I will come back later.

After 1 week I could see data

Matomo is not collecting daya

Securing Mamoto

I read this guide here to secure Matomo

Opt Out Tracking

I enabled Opt Out Tracking in the Mamoto settings and added the generated opt-out code to my front page and at the bottom or all existing articles.

I had to allow iframe tags on my site by adding this header in NGINX (previously I blocked iframes)

add_header X-Frame-Options sameorigin

Add Opt Out Tracking Code to WordPress.

Matomo Opt Out Added to WordPRess widgets

I updated my privacy page and my GDPR notification bar. Now visitors will see a opt-out of tracking on the front page and all article pages.

Opt out of tracking enabled

SMTP Settings

I added my GSuite mail server settings to enable sending of reports via email. I loaded my old guide here to get the GSuite SMTP settings.

GSuite SMTP Settings Added

I enabled force https on the Mamoto application (edited: config/config.ini.php file)

[General]
...
force_ssl = 1

Matomo Plugins (Marketplace)

I opened the System then Plugins section of Matomo to open the Marketplace

Plugins

I installed these plugins

  • Force SSL
  • HidePasswordReset
  • Google Authenticator
  • Device Pixel Ratio
  • Bandwidth
  • Js Tracker Force Async
  • Treemap Visualization
  • Security Info
  • Custom Alerts
  • IP Reports
  • Live Tab
  • etc

Updating PHP

Matomo Admin (Panel – Security/Diagnostics) section will report if your PHP gets out of date.

Matomo warning of PHP being out of date

Hardening Advice

I enabled 2fA Authorisation at logins (Google Analytics Plugin).

Matomo 2fa Login screenshot

Read my guide here on hardware 2FA YubiCo YubiKeys here.

php.ini hardening changes

Matomo also recommended some php.ini file changes.

> open_basedir – open_basedir is disabled. When this is enabled, only files that are in the given directory/directories and their subdirectories can be read by PHP scripts. You should consider turning this on. Keep in mind that other web applications not written in PHP will not be restricted by this setting.

> upload_tmp_dir – upload_tmp_dir is disabled, or is set to a common world-writable directory. This typically allows other users on this server to access temporary copies of files uploaded via your PHP scripts. You should set upload_tmp_dir to a non-world-readable directory

This may break your WordPress so enable at your own risk. I might move Mamoto to a dedicated “analytics” subdomain then enable these options.

Troubleshooting

I had to run this command when installing Device Pixel Ratio, Device Network Information, Bandwidth plugins

Output:

GTMetrix (After)

GT Metrix reports that my site is not slower (still 1.5 seconds)

GTMetrix After Pagespeed

I can see that some JavaScript is not being picked up by CDN.

GTMetrix After YSlow

Also 2 More files loading (when compared to Google Analytics)

2 More Files

Time to add the Mamoto files to my CDN.

Adding Matomo Resources to a CDN

I read this Matomo forum post.

I copied these 2 assets to my WordPress wp-content folder (my WordPress CDN ewww.io will then upload them to the CDN).

I have cache everything enabled in ewww.io and this will copy the javascript assets ot my CDN.  I will need to manually update these js files each time a Matomo update is installed.

I change my Matomo tracker code to include the new CDN location

<!-- Matomo -->
<script type="text/javascript">
  var _paq = _paq || [];
  /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
  _paq.push(['trackPageView']);
  _paq.push(['enableLinkTracking']);
  (function() {
    var u="//fearby.com/utils/matomo/";
    _paq.push(['setTrackerUrl', u+'piwik.php']);
    _paq.push(['setSiteId', '1']);
    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript'; g.async=true; g.defer=true; g.src='https://fearby-com.exactdn.com/wp-content/piwik.js'; s.parentNode.insertBefore(g,s);
  })();
</script>
<!-- End Matomo Code -->

I could not find out how to change the location of my (now CDN cached https://fearby-com.exactdn.com/wp-content/optOut.js) so I temporarily disabled the opt-out form on my front page.

todo: Find out how to change the CDN location of optOut.js and re-enabled the form.

All assets are loading from CDN.

GT Metrix shows my site loads in 1.4 seconds

Analytics Reporting

Graphs are not as pretty as Google Analytics but they are working.

Matomo is not collecting daya

Mobile Reporting

Mobile reporting is good too.

Screenshot of the Matomo Mobile app

Updating Matomo Plugins

Don’t forget to update your plugins from the Matomo dashboard.

Updating Matomo (Core)

Matomo has an official guide on how to update Matomo here.

I do not have FTP so I will perform the manual three step update.

But before I do that I will manually backup my web server and database server just in case.

I backed up my Matomo config (I SSH”ed to the server)

$ cd /www-root/matomo-root/

$ cp ./config.ini.php ./config.ini.3.x.x.php

I navigated to the folder above my Matomo folder

$ cd ..

$ cd ..

I downloaded Matomo

$ wget https://builds.matomo.org/matomo.zip

I unzipped the zip file

$ unzip -o matomo.zip

I removed the matomo.zip file

$ rm matomo.zip

I loaded the Matomo Login page again and was prompted to update the database.

Matomo Database Update Required

Matomo reported it was updated Successfully.

Matomo was updated message

Oops, and error in config error appeared when I tried to log in.

Matomo Error in config

Oh, Do I need to replace the config file with my backed up config file?

(edit: Yes Matomo say to do this, my bad)

Ten seconds later I accidentally deleted all my config files (I had zero backups), the next 2 minutes were spent shutting down my servers (web and db) and restoring them from backup. Thank goodness UpCloud are awesome hosts.

I now had to restore my servers and repeat the steps but this time restore my config file before logging back in.

I did this but had the same error

> An error occurred
> Authentication object cannot be found in the container. Maybe the Login plugin is not activated?
> You can activate the plugin by adding:
> Plugins[] = Login under the [Plugins] section in your config/config.ini.php

I checked my replaced config.ini.php and it did have

> [PluginsInstalled]
> PluginsInstalled[] = “Login”

I googled and found this page that said reset your password (this was not an option as Matomo was not loading)

I logged into mysql with my Matomo user

> mysql -u matomodbusername -p
> Enter password:
> Welcome to the MySQL monitor. Commands end with ; or \g.
> Server version: 5.7.xxxx

> Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

> Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

> Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

> mysql> show databases;
+——————–+
| Database |
+——————–+
| information_schema |
| matomodb |
+——————–+
> 2 rows in set (0.00 sec)

The account and database seem ok.

I tried “FLUSH PRIVILEGES;” with no luck

I tried to sop mysql but it was locked

It was late so I rebooted my server (it did not come back up after a few minutes, I forced a reboot)

I still had an “Authentication object cannot be found in the container.” error when trying to login to Matomo???

I re-checked the “config.ini.php” file after reding threads at the Matomo Forums

$ sudo nano /www-root/matomo-root/config.ini.php

“Plugins[] = “Login”” was not in the “[Plugins]” area of the file???  I added it, saved the change and was able to reload the Matomo GUI.

I checked some key reports.

Visitors over time:

Visitors over time report

Visitor Location Map

Visitor Location Map

Visitor Overview

Visitor Overview

Out links Clicked

Out links Clicked

Nice

I subscribed to the Matomo newsletter here to keep up to date with Matomo update releases: https://matomo.org/newsletter/

Good luck and I hope this guide helps someone

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.2 Hardening info

v1.1 Updating Matomo

v1.0 Initial post

Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc

by

This is a draft post showing how you can monitor the performance of a server (or servers) with NixStats and receive alerts by SMS, Push, Email, Telegram etc

fyi: This is not a paid post, this is just me using the NixStats software to monitor my servers and send alerts.

Finding a good host

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Domain names for just 88 cents!

Monitoring Servers

The post below will show you how you can monitor servers online with https://nixstats.com/ and send alerts when resources reach limits or servers fail.

https://nixstats.com/

I signed up and started a Nixstats (14 day free trial).

Start Nixstats Trial

After I created an account I was emailed by Nixtsats with agent install instructions for Linux (1 line). I was also advised to add contacts and to set up alerts.

I logged into the Nixstats settings and set up…

  • My Timezone
  • Default reporting period
  • First name and Surname
  • Reporting emails
  • etc

Nixstats Subscription Upgrade

Subscription options

  • Free (5 monitors, 1 server, 24-hour data retention etc)
  • Founder (25 monitors, 10 servers, 30-day data retention etc)
  • Business (100 monitors, 15 servers, 60-day data retention etc)

Subscription Options

I enabled the limited founder subscription so I can monitor 10x servers (this deal is too good to miss). I tried creating a status page myself last year and it is terribly hard.

Subscriptiosn page

I am now out of the free trial period 🙂 Let’s start monitoring many servers.

Subscription Activated

I enabled two factor Auth to Nixstats logins

Nixstats Two Factor Auth

I created a Nixstats API key for future use (watch this space)

Create API Key

I installed the Nixstats agent (the dashboard gave a 1 line command you can run as root to install the agent (on Linux)).

Instal Nixstats Agent

FYI: Command (######################## is a number linked to your account)

Output

nixstatsagent.sh 100%[====================================================>] 37.80K –.-KB/s in 0.1s

2018-10-02 09:53:56 (338 KB/s) – ‘nixstatsagent.sh’ saved [38708/38708]

Found Ubuntu …
Installing …
Installing Python2-PIP …
Installing nixstatsagent …
Generation a server id …
Got server_id: ######################
Creating and starting service
Created symlink /etc/systemd/system/multi-user.target.wants/nixstatsagent.service -> /etc/systemd/system/nixstatsagent.service.
Created the nixstatsagent service

Server Dashboard
below is a summary of all connected servers ( https://nixstats.com/dashboard/servers ).

Server Sumamry

Monitor Setup

I set up a number of monitors to monitor ping replies and https traffic

Monitors

Advanced Monitoring

I can also set the monitor credentials, timeouts, retries, auth methods, max redirects and frequency. If you server blocks login or resource GET attempts you may need to whitelist IP’s. IP’s of monitoring servers are located here https://nixstats.com/whitelist.php

Monitoring advanced options

Monitor Summary

The default dashboard is very informative. Feel free to create your own dashboards that focus on your own infrastructure or apps.

Monitor Summary

Individual Server Reports

You can click on a server and monitor it in detail.

Nixstats Graphs

Server Memory Graphs

Long-term memory graphs.

Memory Graph

Install Optional Nixstats Plugins

Nixstats offers many plugins to monitor software that is installed on your server (e.g NGINX, MySQL, PHP etc).

1) NGINX Monitoring (Plugin)

To enable NGINX monitoring I read https://help.nixstats.com/en/article/monitoring-nginx-50nu7f/

I edited my NGINX sites-enabled config.

I added the following

server {
    listen 127.0.0.1:8080;
    server_name localhost;
    location /status_page {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }
}

I tested, reloaded and restarted NGINX

The status page will only be available on the local machine, I tested the page on the local machine

It’s Working.

I edit /etc/nixstats.ini

I remove comments before these lines to enable the plugin

I ran the following command to see if NGINX monitoring is possible

Output

nginx:
{
    "accepts": 39,
    "accepts_per_second": 0.0,
    "active_connections": 6,
    "handled": 39,
    "handled_per_second": 0.0,
    "reading": 0,
    "requests": 119,
    "requests_per_second": 0.0,
    "waiting": 5,
    "writing": 1
}

It’s Working

I restarted the nixstatsagent

I can now view NGINX properties like active_connections in my dashboard. 🙂

2) Enable PHP-FPM Monitoring (Plugin)

Looks like a PHP-FPM monitoring was recently added lets set that up too. Read my guide on setting up PHP child workers here.

To enable PHP-FPM monitoring I read https://help.nixstats.com/en/article/monitoring-php-fpm-1tlyur6/

I edited my PHP-FPM ini file

I added the following line

Restart PHP

I added the following to /etc/nginx/sites-enabled/default localhost server block added above Note I use php 7.2 below (read more here).

server {
    listen 127.0.0.1:8080;
    server_name localhost;

location /status_phpfpm {
access_log off;
allow 127.0.0.1;
deny all;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass unix:/run/php/php7.2-fpm.sock;
log_not_found off;
}
}

I tested, reloaded and restarted NGINX

I restart PHP-FPM

Enabled the plugin in /etc/nixstats.ini

I tested the status page

Output:

{
	"pool":"www",
	"process manager":"static",
	"start time":1538654543,
	"start since":178,
	"accepted conn":28,
	"listen queue":0,
	"max listen queue":0,
	"listen queue len":0,
	"idle processes":49,
	"active processes":1,
	"total processes":50,
	"max active processes":2,
	"max children reached":0,
	"slow requests":0
}

I tested the agent

Output:

phpfpm:
{
    "accepted_conn": 51,
    "accepted_conn_per_second": 0.0,
    "active_processes": 1,
    "idle_processes": 49,
    "listen_queue": 0,
    "listen_queue_len": 0,
    "max_active_processes": 2,
    "max_children_reached": 0,
    "max_listen_queue": 0,
    "pool": "www",
    "process_manager": "static",
    "slow_requests": 0,
    "start_since": 318,
    "start_time": 1538654543,
    "total_processes": 50
}

I can now query PHP-FPM status values in Nixstats 🙂

Query PHP-FPM

Enable MySQL Monitoring (Plugin)

To enable MySQL monitoring I read https://help.nixstats.com/en/article/monitoring-mysql-1frskd8/

I edited the nixstats.ini

I enabled the mysql section in nixstats.ini and added my mysql credentials

[mysql]
enabled=yes
username=mysqluser
password=#######################
host=127.0.0.1
database=mysql
port=3306
socket=null

I ran this command to test MySQL querying

nixstatsagent --test mysql
mysql:
Load error: No module named MySQLdb

I had an error.

A quick Google revealed I had to install a mysql python module.

Allow localhost to connect to MySQL

Edit /etc/mysql.cnf and allow all localhost and external connections (I could not bind to localhost and an external IP at the same time)

bind-address    = 0.0.0.0

TIP: Ensure you have firewalled access to your MySQL server, never open it up without protection.

Let’s try again

nixstatsagent --test mysql

Output

Nice,

I restart MySQL

I restart my Nixstats service

service nixstatsagent restart

Now let’s monitor MySQL in Nixstats

I can now view MySQL metrix

MySQL MEtrix

Status Page

Nixstats allows you to create a status page ( https://nixstats.com/pages/overview ) where you can add any servers or monitors to that page. This stats page is truly awesome, it builds a live status page based on data coming from your installed agents.

You can even set up a custom subdomain that points to a Nixstats hosted status page too (e.g https://status.yourdomain.com).

FYI: An SSL certificate on your staus page may take a few hours to set up. Don’t panic if it is not instantatly available.

Custom Status Page

Nice.

This saves doing it yourself. The status page will look like it running on your server.

Status Page

You can create a status page that automatically aggregates collected data from monitors and displays them in a nice layout.

Status Page

This is great, I used to do my own status pages but not anymore.

Alerts

I added a contact so I could receive alerts. I could then add my mobile, email and PushOver key (to receive push notifications) and Telegram Bot API token.

Contact

Test Alerts

I sent a test alert to each service against the contact.

Test Alerts

I activated a Pushover licence on my Android device for about $7.49 AUD (one off) to ensure I keep getting Push Notifications.

Bought licence for PushOver

Nixstats have links that show you how you can create a Telegram Bot and Pushover.net account.

Pushover will cost about $5 USD one off per device (see faq).

I created the following alerts

Alert: Disk Usage higher than 90%

Alert Disk Usage higher than 90%

Alert: Load greater than 90 per cent for 1 minute

Alert load greater than 90 percent for 1 minute

Alert: Less than 5 percent memory free.

Alert less than 5 percent memory free

Summary of alerts.

Alert Sumamry

I also added a CPU reached 95% one for 5 mins alert too (but it’s not pictured above)

I forgot to specify alert recipients and methods for each alert so I edited each alert and added the contact.

Selected Alert Recipients and methods

Now it’s time to test the alerts.

I shut down a server to test alerts

Alerts to my defined Email, SMS, Push and Telegram are working a treat 🙂

Alerts Working

After I rebooted the server I also received alerts about the server being back up.

The status page showed the server that was offline too.

Server Offline

Nice

Troubleshooting

I had an issue instaling the agent on Debian

I ran the following command

nixstatsagent.sh: line 508: [: Installer exited with error code 0. See nixstatsagent.log for details.: integer expression expected

An error occurred, please check the install log file (nixstatsagent.log)!

Contents of nixstatsagent.log

I asked the Nixstats chat help and I was advised I had a dead repository (I removed this (editing the dead repo in the appropriate file in /etc/apt/) and all was ok)

I had trouble testing my Telegram alerts but it was my fault as I forgot to follow the bot account I created. Telegram does not allow message from a user (bot) unless you follow them.

A chat with the Nixstats staff sorted me out. Thanks, Nixstats chat team.

Nixstats chat

I had an issue with a missing python mysql package

I solved it by instaling python-mysqldb

Nixstats Help

Nixstats have a help subdomain: https://help.nixstats.com/en/

Nixstats Help

Error Logs Plugin

I did ask Nixstats on Twitter and they said they are working on a logging plugin, I can’t wait for that.

I now have access to beta log features and can see log tabs in Nixstats

I had or check the version of my rsyslogd

I edited: /etc/rsyslog.d/31-nixstats.conf

I pasted

##########################################################
### Rsyslog Template for Nixstats ###
##########################################################

$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

template(name=”NixFormat” type=”string”
string=”<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [[email protected] tag=\”rsyslog\”] %msg%\n”
)

action(type=”omfwd” protocol=”tcp” target=”log.nixstats.com” port=”514″ template=”NixFormat”)
#################END CONFIG FILE#########################

I restarted the rsyslog service

Live Log Output

I can see a live log from (unknown) logs.

I can see the firewall blocking access to certain ports.

Live Log

Search logs

Search

Blacklist Checking (Beta)

Nixstats tweeted “We just launched a new blacklist check feature. Monitor your IP and hostname reputation. Free during the Beta!”

I enabled it.

I received a Blacklist notification

I requested removal at junkmailfilter.com

Thanks, Nixstats

Conclusion

This is one of the best software packages I have seen in a while. I have developed status

pages in the past (no more). Great work Nixstats.

Please check out Nixstats today, they are awesome. Signup for a free account and consider the limited time founder plan (it’s a bargain).

Nixstats Live chat support is awesome

Server Plug

If you need a server, consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.3 Blacklist beta

V1.2 Logs beta

v1.1 Logging Tweet

v1.0 Initial Post

Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap

by

This guide will show how you can set up a website to use Cloudflare on a VM hosted on Vultr and Namecheap

I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. This post will show how to let Cloudflare handle the DNS for the domain.

Update 2018: For the best performing VM host (UpCloud) read my guide on the awesome UpCloud VM hosts (get $25 free credit by signing up here).

Snip from here “Cloudflare’s enterprise-class web application firewall (WAF) protects your Internet property from common vulnerabilities like SQL injection attacks, cross-site scripting, and cross-site forgery requests with no changes to your existing infrastructure.”

Buy a Domain 

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Cloudflare Benefits (Free Plan)

  • DDoS Attack Protection (Huge network to absorb attacks DDoS attacks over 600Gbps are no problem for our 15 Tbps networks)
  • Global CDN
  • Shared SSL certificate (I disabled this and opted to use my own)
  • Access to audit logs
  • 3 page rules (maximum)

View paid plan options here.

Cloudflare CDN map

Cloudflare CDN says it can load assets up to 2x faster, 60% less bandwidth from your servers by delivering assets from 127 data centres.

Cloudflare Global Network

Setup

You will need to sign up at cloudflare.com

Cloudflare

After you create an account you will be prompted to add a siteAdd SiteCloudflare will pull your public DNS records to import.

Query DNS

You will be prompted to select a plan (I selected free)

Plan Select

Verify DNS settings to import.

DNS Import

You will now be asked to change your DNS nameservers with your domain reseller

DNS Nameservers

TIP: If you have an SSL cert (e.g Lets Encrypt) already setup head to the crypto section and select ” Full (Strict)” to prevent ERR_TOO_MANY_REDIRECTS errors.

Strict SSL

Cloudflare UI

I asked Twitter if they could kindly load my site so I could see if Cloudflare dashboard/stats were loading.

The Cloudflare CTO responded.  🙂

Confirm Cloudflare link to a domain from the OSX Comand line

Caching Rule

I set up the following caching rule to cache everything for 8 hours instead of WordPress pages

Page Rules

“fearby.com.com/wp-*” Cache level: Bypass

“fearby.com.com/wp-admin/post.php*” Cache level: Bypass

“fearby.com/*” Cache Everything, Edge Cache TTL: 8 Hours

Cache Results

Cache appears to be sitting at 50% after 12 hours.  having cache os dynamic pages out there is ok unless I need to fix a typo, then I need to login to Cloudflare and clear the cache manually (or wait 8 hours)

Performance after a few hours

DNS times in gtmetrix have now fallen to a sub 200ms (Y Slow is now a respectable A, it was a C before).  I just need to wait for caching and minification to kick in.

DNS Improved

webpagetest.org results are awesome

See here: https://www.webpagetest.org/result/180314_PB_7660dfbe65d56b94a60d7a604ca250b3/

  • Load Time: 1.80s
  • First Byte 0.176s
  • Start Render 1.200s

webpagetest

Google Page Speed Insights Report

Mobile: 78/100

Desktop: 87/100

Check with https://developers.google.com/speed/pagespeed/insights/

Update 24th March 2018 Attacked?

I noticed a spike in and traffic (incoming and threats) on the 24th of March 2018.

I logged into Cloudflare on my mobile device and turned on Under Attack Mode.

Under Attack Flow

Cloudflare was now adding a delay screen in the middle of my initial page load. Read more here.  A few hours after the Attach started it was over.

After the Attack

I looked at the bandwidth and found no increase in traffic from my initial host VM. Nice.

cloudflare-attack-001

Thanks, Cloudflare.

Cloudflare Pros

  • Enabling Attack mode was simple.
  • Soaked up an attack.
  • Free Tier
  • Many Reports
  • Option to force HTTPS over HTTP
  • Option to ban/challenge suspicious IP’s and set challenge timeframes.
  • Ability to setup IP firewall rules and Application Firewalls.
  • User-agent blocking
  • Lockdown URL’s to IP’s (pro feature)
  • Option to minify Javascript, CSS and HTML
  • Option to accelerate mobile links
  • Brotli compression on assets served.
  • Optio to enable BETA Rocket loader for Javascript performance tweaks.
  • Run Javascript service workers from the 120+ CDN’s
  • Page/URL rules o perform custom actions (redirects, skip cache, Encryption etc)
  • HTTP/2 on, IPV6 ON
  • Option to setup load balancing/failover
  • CTO of Cloudflare responded in Twitter 🙂
  • Option to enable rate limiting (charged at 10,000 hits for $0.05c)
  • Option to block countries (pro feature)
  • Option to install apps in Cloudflare like(Goole Analytics,

Cloudflare Cons

  • No more logging into NameCheap to perform DNS management (I now goto Cloudflare, Namecheap are awesome).
  • Cloudflare Support was slow/confusing (I ended up figuring out the redirect problem myself).
  • Some sort of verify Cloudflare Setup/DNS/CDN access would be nice. After I set this up my gtmetrix load times were the same and I was not sure if DNS needs to replicate? Changing minify settings in Cloudflare did not seem to happen.
  • WordPress draft posts are being cached even though page riles block wp-admin page caching.
  • Would be nice to have ad automatic Under Attack mode
  • Now all sub-domains were transferred in the setup ( id did not know for weeks)

Cloudflare status

Check out https://www.cloudflarestatus.com/ for status updates.

Don’t forget to install the CloudFlare Plugin for WordPress if you use WordPress.

More Reading

Check out my OWASP Zap and Kali Linux self-application Penetration testing posts.

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.8 host Command from the OSX CLI

v1.7 Subdomain error

v1.6 Cloudflare Attack

v1.5 WordPress Plugin

v1.4 More Reading

v1.3 added WAF snip

v1.2 Added Google Page Speed Insights and webpage rest results

v1.1 Added Y-Slow

v1.0 Initial post

Manage Social Media posts with Buffer

by

Here is a quick setup guide for Buffer.com where you can connect to and post (manually or scheduled) to multiple social media platforms.

You can view pricing here. You can signup for a  free Buffer Individual plan: https://buffer.com/signup. Signup to Buffer (Free, Limited)

Post Signup Setup

Connect Buffer to Social Media Platforms

Buffer SIgnup

Type post content

Schedule

Change the default image

Define an Image

Choose the platforms and images

Choose Platforms

Schedule the Post

Schedule

You can manually share to a platform at any time.

Share Now

TIP: If you share now you will need to manually share on each platform separately.

Results

Buffer Results

Buffer features I like

Buffer features I Don’t like

  • Manual Share to all feature missing.
  • Timezones earlier than US Timezones appear to be untouchable (my Timezone is set)
    Timezone

Buffer FAQ’s: https://faq.buffer.com/

Tip: Create custom posting slots

Custom Slots

More soon (reply automation and image creation).

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.1 Added Posting Slots Info

etc

How to optimize your sites Search Engine Optimization (SEO) and grow customers without paying for Ads

by

How to optimize your sites Search Engine Optimization (SEO) and grow customers without paying for Ads.

This guide is a shorter post around setting up SEO (Search Engine Optimization) and driving more traffic to your site without buying ADs.  In a nutshell, to have better SEO you need to jump some technical hurdles in order to drive more traffic to your site from search engines along with understanding your customer’s needs and making things easier for them.

I have blogged about these topics before but these posts are too long in reflection.

Buying Ad’s?

Facebook, Google, Bing and advertising agencies will recommend you set goals around growth and site traffic and pay for those goals to succeed (usually by advertisements).

Don’t get me wrong Advertising works but it is a competitive market, Online sites can easily setup the display of Ad’s on their site (my guide here Add Google AdWords to your WordPress blog, https://fearby.com/article/add-google-adwords-wordpress-blog/ ). You can buy physical billboard ad’s on the side of roads (e.g http://www.buythisspace.com.au/). I tried to enquire about the costs of a physical billboard but the agencies robot verification rejected my enquiry submission so I gave up.  Advertising is buying peoples times and people now how to avoid ad’s and not interact with them (7 Marketing Lessons from Eye-Tracking Studies https://blog.kissmetrics.com/eye-tracking-studies/)

Do more of what works

Spoiler: This guide will recommend you do more of what works over buying millions of ad’s and hoping for new and engaged customers and customer growth.

SEO Tip’s

This older article on  How to boost your site’s SEO  attempts to mention what you need to do it to get better SEO.

Do run a modern great site

I am a big fan of word of mouth over free/organic traffic over paid customers via advertising (Mostly because I am tight and realize advertising can be a bottomless pit). The single biggest thing you can do to have more organic traffic from search engines is run a modern and fast website, have valuable content and make it as easy for the customer as possible. This is why I moved my site and setup an SSL certificate (link to article).

Search engines like your site to be fast, updated frequently, have sitemaps to make their jobs easier and have an SSL certificate to keep the web safe etc.

Google, Bing and other search engines will not send traffic your way if you do not satisfy them that your site is liked or has valuable content.  Google makes money from Google Analytics by helping people understand their site’s visitors then recommend you pay for ad’s to use on sites that have AdWords on their site ( WordPress to a new self-managed server away from CPanel ).

Traffic Source types

  • Organic – An organic visitor to your site is one who found your site by searching something that was relevant to their search term and not by clicking on an advertisement.
  • Paid – A paid user is someone who has clicked an ad to come to your site.
  • Social – A social visitor is one who is known to come from a social media site, using social media sites like Twitter, Facebook or Instagram is a must to driving organic traffic (go where the people are).

Engagement

How engaged are your customers?  Have you asked your customers recently what they value or appreciate about your business or product? Have you asked for feedback recently?

User Engagement Levels

  • None – Do you have landing pages that quickly inform customers of your products or services?
  • Low – What do they need to know about your product or service?
  • Medium – Aware (engaged)
  • High – Can this person be an advocate for your business?
  • Gone – Did you get exit Feedback?

Ways to engage already engaged customers.

What can you do to help understand your customer’s needs and make their purchase processes easier?

Why are your customers leaving?

Understand more about your customers reasons for leaving and act upon preventing others from leaving.

  • Trying something new (Does your website need to be simpler?)
  • Are your products too expensive?.
  • Your site (or ordering) is not convenient (Do you need to setup online ordering/subscriptions and delivery?)
  • etc

Who are your customers

Are your customers.

  • Engaged
  • Informed
  • Advocates

Feedback

  • Do you have feedback loops (A simple feedback form can solve this)?

What do you know about your customers?

  • Product Satisfaction
  • Product Loyalty
  • Product Awareness

Paid Traffic (Ad’s)

Free Traffic (SEO + Organic Ad’s)

  • Blog Posts (Sharing value/passion)
  • Social Media Posts (use hashtags)
  • Instagram (Post value/passion)

Most importantly Do what works (Measure and replicate).

Focus on Business Value

Generate a  SWOT Analysis ( Free tool here https://xtensio.com/ )

  • What are your Strengths?
  • What are your Weaknesses?
  • What are your Opportunities?
  • What are your Threats?

Goals

Goals allow you to investigate, learn, act and measure I order to improve.

  • Investigate – Data.
  • Learn/Insight – Make Assumptions.
  • Act – Act and measure.

Read more about customer engagement here https://en.wikipedia.org/wiki/Customer_engagement

Bonus

 Do ensure your website is compliant with accessibility and technical standards

Conclusion

The more you know the better you can connect, Do set goals and as a minimum setup Google Analytics, SSL certificate and submit your site to search engines, then focus on a fast site that makes things simple for your customers.

Donate and make this blog better



Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.0 Initial version

Setting up Google Analytics on your website

by

Google Analytics is a popular easy to install and use statistics and reporting tool that you can add to your website (and it’ free)

To setup, Google Analytics go to https://analytics.google.com/analytics/web/ and create an account. From here you can add a site and generate a tracking ID.

Google Analitics Geenrate ID

The website tracking code was (I changed the code to 555555555).

I opened WordPress and went to Appearance then Editor and selected header.php and added the tracking code under the <head> HTML tag.

edit file

This tracking ID allows Google to generate stats from your visitors.

I was unable to update the file in WordPress until I set permissions in Ubuntu in (Read these guide to setup an Ubuntu Server on Vultr for as low as $2.5 a month of setup a $5 a month with Digital Ocean or AWS).  I have guides on moving WordPress here or setting up WordPress from the command line here). If you update WordPress you may need to re add the tracking ID.

sudo chmod 666 /www/wp-content/themes/twentyseventeen/header.php

I loaded my WordPress website and verified that the tracking code was loading in the HTML source.  You can also embed the tracking code in static HTML websites.

html source

After a few days, you can view your sites statistics. from the Googe Analytics home portal.  This will allow you know when to publish, know how popular your content is, know what new content to create etc.

Page Hits

The best feature of Google Analytics is page hit information. To me, the total number of hits is less important than Avg. Time on Page and Bounce Rate.

page hits

Dashboard

The Google Analytics dashboard home is very informative.

Analitics Home

Google Analytics Terms

Google has a glossary for terms here.

  • Users – The unique user that visited your site.
  • Bounce Rate – The percentage or users who loaded your site and left after viewing the initial page.
  • Active Users – The total number of active users reading your site.
  • User Retention – The percentage of users who have returned to your site.
  • Device – The device (Desktop, tablet or mobile device) that was used to read your site.
  • Organic Search – The number of users who found your site via a search engine.  Having a highly efficient SEO will see a higher Organic search percentage.
  • Sessions – The number of unique sessions that your users have accessed your site.
  • Direct – The times a user has directly typed your website URL (or have visited your site in incognito/privacy mode).
  • Referral – The percentage or know referrals from other websites.
  • Social – Known number of visits to our site from social media platforms.

Overview

You can watch in real-time users accessing your site. This is important when you send out mailing list to users when new content is posted, will 1,000 visitors take down your site? Are you posting at the right time for your sites visitors timezone?

active usersAudience Overview

This report will tell you a lot about who and where people are visiting your site form and what language they speak, OS they use, what browser they use and what city they are from.

audience stats

Google Analytics allows you to drill down on most captured data.

City breakdown

I can see Apple devices are the most popular mobile devices accessing my site (but mobile devices in total only take up 12 % of my site’s traffic).

Devices

The User Flow report is a great way to see how people interact with your site (where they come from, what they do and where they drop out).

user flow

Google Analytics has a handy page speed tool that you can use to identify what you need to do to speed up your site.

Page speed

Google Analytics have goals that allow you to set targets to meet. Usually, Google encourages you to assign a monetary value to a goal then suggest you buy Google Ad’s to achieve these goals (this is why Google Analytics is free). Read my guide on setting up Google AdWords on your WordPress blog.

goals

You can set email alerts on key stats.

alerts

More to come later.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.1 added page hits information

Securing Ubuntu in the cloud

by

It is easy to deploy servers to the cloud within a few minutes, you can have a cloud-based server that you (or others can use). ubuntu has a great guide on setting up basic security issues but what do you need to do.

If you do not secure your server expects it to be hacked into. Below are tips on securing your cloud server.

First, read more on scanning your server with Lynis security scan.

Always use up to date software

Always use update software, malicious users can detect what software you use with sites like shodan.io (or use port scan tools) and then look for weaknesses from well-published lists (e.g WordPress, Windows, MySQL, node, LifeRay, Oracle etc). People can even use Google to search for login pages or sites with passwords in HTML (yes that simple).  Once a system is identified by a malicious user they can send automated bots to break into your site (trying millions of passwords a day) or use tools to bypass existing defences (Security researcher Troy Hunt found out it’s child’s play).

Portscan sites like https://mxtoolbox.com/SuperTool.aspx?action=scan are good for knowing what you have exposed.

You can also use local programs like nmap to view open ports

Instal nmap

Find open ports

Limit ssh connections

Read more here.

Use ufw to set limits on login attempts

Only allow known IP’s access to your valuable ports

sudo ufw allow from 123.123.123.123/32 to any port 22

Delete unwanted firewall rules

Only allow known IP’s to certain ports

Also, set outgoing traffic to known active servers and ports

Don’t use weak/common Diffie-Hellman key for SSL certificates, more information here.

More info on generating SSL certs here and setting here and setting up Public Key Pinning here.

Intrusion Prevention Software

Do run fail2ban: Guide here https://www.linode.com/docs/security/using-fail2ban-for-security

I use iThemes Security to secure my WordPress and block repeat failed logins from certain IP addresses.

iThemes Security can even lock down your WordPress.

You can set iThemes to auto lock out users on x failed logins

Remember to use allowed whitelists though (it is so easy to lock yourself out of servers).

Passwords

Do have strong passwords and change the root password provided by the hosts. https://howsecureismypassword.net/ is a good site to see how strong your password is from brute force password attempts. https://www.grc.com/passwords.htm is a good site to obtain a strong password.  Do follow Troy Hunt’s blog and twitter account to keep up to date with security issues.

Configure a Firewall Basics

You should install a firewall on your Ubuntu and configure it and also configure a firewall with your hosts (e.g AWS, Vultr, Digital Ocean).

Configure a Firewall on AWS

My AWS server setup guide here. AWS allow you to configure the firewall here in the Amazon Console.

Type Protocol Port Range Source Comment
HTTP TCP 80 0.0.0.0/0 Opens a web server port for later
All ICMP ALL N/A 0.0.0.0/0 Allows you to ping
All traffic ALL All 0.0.0.0/0 Not advisable long term but OK for testing today.
SSH TCP 22 0.0.0.0/0 Not advisable, try and limit this to known IP’s only.
HTTPS TCP 443 0.0.0.0/0 Opens a secure web server port for later

Configure a Firewall on Digital Ocean

Configuring a firewall on Digital Ocean (create a $5/m server here).  You can configure your Digital Ocean droplet firewall by clicking Droplet, Networking then Manage Firewall after logging into Digital Ocean.

Configure a Firewall on Vultr

Configuring a firewall on Vultr (create a $2.5/m server here).

Don’t forget to set IP rules for IPV4 and IPV6, Only set the post you need to allow and ensure applications have strong passwords.

Ubuntu has a firewall built in (documentation).

Enable the firewall

Adding common ports

sudo ufw allow ssh/tcp
sudo ufw logging on
sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 53
sudo ufw allow 443
sudo ufw allow 873
sudo ufw enable
sudo ufw status
sudo ufw allow http
sudo ufw allow https

Add a whitelist for your IP (use http://icanhazip.com/ to get your IP) to ensure you won’t get kicked out of your server.

sudo ufw allow from 123.123.123.123/24 to any port 22

More help here.  Here is a  good guide on ufw commands. Info on port numbers here.

If you don’t have a  Digital Ocean server for $5 a month click here and if a $2.5 a month Vultr server here.

Backups

rsync is a good way to copy files to another server or use Bacula

Basics

Initial server setup guide (Digital Ocean).

Sudo (admin user)

Read this guide on the Linux sudo command (the equivalent if run as administrator on Windows).

Users

List users on an Ubuntu OS (or compgen -u)

Common output

Add User

e.g

Add user to a group

Show users in a group

This will show users in a group

Remove a user

Rename user

Change user password

Groups

Show all groups

compgen -ug

Common output

compgen -g
root
daemon
bin
sys
adm
tty
disk
lp
mail
proxy
sudo
www-data
backup
irc
etc

You can create your own groups but first, you must be aware of group ids

cat /etc/group

Then you can see your systems groups and ids.

Create a group

Permissions

Read this https://help.ubuntu.com/community/FilePermissions

How to list users on Ubuntu.

Read more on setting permissions here.

Chmod help can be found here.

Install Fail2Ban

I used this guide on installing Fail2Ban.

Check Fail2Ban often and add blocks to the firewall of known bad IPs

Best practices

Ubuntu has a guide on basic security setup here.

Startup Processes

It is a good idea to review startup processes from time to time.

Accounts

  • Read up on the concept of least privilege access for apps and services here.
  • Read up on chmod permissions.

Updates

Do update your operating system often.

Minimal software

Only install what software you need

Exploits and Keeping up to date

Do keep up to date with exploits and vulnerabilities

Secure your applications

  • NodeJS: Enable logging in applications you install or develop.

Ban repeat Login attempts with FailBan

Fail2Ban config

Hosts File Hardening

Add

Add a whitelist with your ip on /etc/fail2ban/jail.conf (see this)

Restart the service

sudo service fail2ban restart
sudo service fail2ban status

Intrusion detection (logging) systems

Tripwire will not block or prevent intrusions but it will log and give you a heads up with risks and things of concern

Install Tripwire.

Running Tripwire

This will scan your system for issues of note

My Output.

sudo nano /var/log/tiger/security.report.username.170809-18:42

Security scripts *** 3.2.3, 2008.09.10.09.30 ***
Wed Aug  9 18:42:24 AEST 2017
20:42> Beginning security report for username (x86_64 Linux 4.4.0-89-generic).

# Performing check of passwd files...
# Checking entries from /etc/passwd.
--WARN-- [pass014w] Login (bob) is disabled, but has a valid shell.
--WARN-- [pass014w] Login (root) is disabled, but has a valid shell.
--WARN-- [pass015w] Login ID sync does not have a valid shell (/bin/sync).
--WARN-- [pass012w] Home directory /nonexistent exists multiple times (3) in
         /etc/passwd.
--WARN-- [pass012w] Home directory /run/systemd exists multiple times (2) in
         /etc/passwd.
--WARN-- [pass006w] Integrity of password files questionable (/usr/sbin/pwck
         -r).

# Performing check of group files...

# Performing check of user accounts...
# Checking accounts from /etc/passwd.
--WARN-- [acc021w] Login ID dnsmasq appears to be a dormant account.
--WARN-- [acc022w] Login ID nobody home directory (/nonexistent) is not
         accessible.

# Performing check of /etc/hosts.equiv and .rhosts files...

# Checking accounts from /etc/passwd...

# Performing check of .netrc files...

# Checking accounts from /etc/passwd...

# Performing common access checks for root (in /etc/default/login, /securetty, and /etc/ttytab...
--WARN-- [root001w] Remote root login allowed in /etc/ssh/sshd_config

# Performing check of PATH components...
--WARN-- [path009w] /etc/profile does not export an initial setting for PATH.
# Only checking user 'root'

# Performing check of anonymous FTP...

# Performing checks of mail aliases...
# Checking aliases from /etc/aliases.

# Performing check of `cron' entries...
--WARN-- [cron005w] Use of cron is not restricted

# Performing check of 'services' ...
# Checking services from /etc/services.
--WARN-- [inet003w] The port for service ssmtp is also assigned to service
         urd.
--WARN-- [inet003w] The port for service pipe-server is also assigned to
         service search.

# Performing NFS exports check...

# Performing check of system file permissions...
--ALERT-- [perm023a] /bin/su is setuid to `root'.
--ALERT-- [perm023a] /usr/bin/at is setuid to `daemon'.
--ALERT-- [perm024a] /usr/bin/at is setgid to `daemon'.
--WARN-- [perm001w] The owner of /usr/bin/at should be root (owned by daemon).
--WARN-- [perm002w] The group owner of /usr/bin/at should be root.
--ALERT-- [perm023a] /usr/bin/passwd is setuid to `root'.
--ALERT-- [perm024a] /usr/bin/wall is setgid to `tty'.

# Checking for known intrusion signs...
# Testing for promiscuous interfaces with /bin/ip
# Testing for backdoors in inetd.conf

# Performing check of files in system mail spool...

# Performing check for rookits...
# Running chkrootkit (/usr/sbin/chkrootkit) to perform further checks...
--WARN-- [rootkit004w] Chkrootkit has detected a possible rootkit installation
Possible Linux/Ebury - Operation Windigo installetd

# Performing system specific checks...
# Performing checks for Linux/4...

# Checking boot loader file permissions...
--WARN-- [boot02] The configuration file /boot/grub/menu.lst has group
         permissions. Should be 0600
--FAIL-- [boot02] The configuration file /boot/grub/menu.lst has world
         permissions. Should be 0600
--WARN-- [boot06] The Grub bootloader does not have a password configured.

# Checking for vulnerabilities in inittab configuration...

# Checking for correct umask settings for init scripts...
--WARN-- [misc021w] There are no umask entries in /etc/init.d/rcS

# Checking Logins not used on the system ...

# Checking network configuration
--FAIL-- [lin013f] The system is not protected against Syn flooding attacks
--WARN-- [lin017w] The system is not configured to log suspicious (martian)
         packets

# Verifying system specific password checks...

# Checking OS release...
--WARN-- [osv004w] Unreleased Debian GNU/Linux version `stretch/sid'

# Checking installed packages vs Debian Security Advisories...

# Checking md5sums of installed files

# Checking installed files against packages...
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/udev/hwdb.bin' does not belong to any package.

# Performing check of root directory...

# Checking device permissions...
--WARN-- [dev003w] The directory /dev/block resides in a device directory.
--WARN-- [dev003w] The directory /dev/char resides in a device directory.
--WARN-- [dev003w] The directory /dev/cpu resides in a device directory.
--FAIL-- [dev002f] /dev/fuse has world permissions
--WARN-- [dev003w] The directory /dev/hugepages resides in a device directory.
--FAIL-- [dev002f] /dev/kmsg has world permissions
--WARN-- [dev003w] The directory /dev/lightnvm resides in a device directory.
--WARN-- [dev003w] The directory /dev/mqueue resides in a device directory.
--FAIL-- [dev002f] /dev/rfkill has world permissions
--WARN-- [dev003w] The directory /dev/vfio resides in a device directory.

# Checking for existence of log files...
--FAIL-- [logf005f] Log file /var/log/btmp permission should be 660
--FAIL-- [logf007f] Log file /var/log/messages does not exist

# Checking for correct umask settings for user login shells...
--WARN-- [misc021w] There is no umask definition for the dash shell
--WARN-- [misc021w] There is no umask definition for the bash shell

# Checking symbolic links...

# Performing check of embedded pathnames...
20:47> Security report completed for username.

More on Tripwire here.

Hardening PHP

Hardening PHP config (and backing the PHP config it up), first create an info.php file in your website root folder with this info

Now look for what PHP file is loadingPHP Config

Back that your PHP config file

TIP: Delete the file with phpinfo() in it as it is a security risk to leave it there.

TIP: Read the OWASP cheat sheet on using PHP securely here and securing php.ini here.

Some common security changes

Don’t forget to review logs, more config changes here.

Antivirus

Yes, it is a good idea to run antivirus in Ubuntu, here is a good list of antivirus software

I am installing ClamAV as it can be installed on the command line and is open source.

ClamAV help here.

Scan a folder

Setup auto-update antivirus definitions

I set auto updates 24 times a day (every hour) via daemon updates.

tip: Download manual antivirus update definitions. If you only have a 512MB server your update may fail and you may want to stop fresh claim/php/nginx and mysql before you update to ensure the antivirus definitions update. You can move this to a con job and set this to update at set times over daemon to ensure updates happen.

Manual scan with a bash script

Create a bash script

Bash script contents to update antivirus definitions.

Edit the crontab to run the script every hour

Uninstalling Clam AV

You may need to uninstall Clamav if you don’t have a lot of memory or find updates are too big.

Setup Unattended Ubuntu Security updates

At login, you should receive

Other

  • Read this awesome guide.
  • install Fail2Ban
  • Do check your log files if you suspect suspicious activity.

Check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.92 added hardening a linux server link