• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Create a VM ($25 Credit)
  • Buy a Domain
  • 1 Month free Back Blaze Backup
  • Other Deals
    • Domain Email
    • Nixstats Server Monitoring
    • ewww.io Auto WordPress Image Resizing and Acceleration
  • About
  • Links

IoT, Code, Security, Server Stuff etc

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

  • Cloud
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to buy a new domain and SSL cert from NameCheap, a Server from Digital Ocean and configure it.
    • Setting up a Vultr VM and configuring it
    • All Cloud Articles
  • Dev
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to setup pooled MySQL connections in Node JS that don’t disconnect
    • NodeJS code to handle App logins via API (using MySQL connection pools (1000 connections) and query parameters)
    • Infographic: So you have an idea for an app
    • All Development Articles
  • MySQL
    • Using the free Adminer GUI for MySQL on your website
    • All MySQL Articles
  • Perf
    • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap
    • All Performance Articles
  • Sec
    • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
    • Using OWASP ZAP GUI to scan your Applications for security issues
    • Setting up the Debian Kali Linux distro to perform penetration testing of your systems
    • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
    • PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API
    • Setting strong SSL cryptographic protocols and ciphers on Ubuntu and NGINX
    • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
    • All Security Articles
  • Server
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All Server Articles
  • Ubuntu
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Useful Linux Terminal Commands
    • All Ubuntu Articles
  • VM
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All VM Articles
  • WordPress
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
    • How to backup WordPress on a host that has CPanel
    • Moving WordPress to a new self managed server away from CPanel
    • Moving a CPanel domain with email to a self managed VPS and Gmail
    • All WordPress Articles
  • All

with

Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution

November 18, 2018 by Simon

This is how I replacing Google Analytics with Piwik/Matomo for a locally hosted privacy-focused open source analytics solution

Aside

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. PHP is my programming language of choice.

Now on with the post

Google Analytics

I will fully admit Google Analytics is good. I posted this a while ago on how you can set up Google Analytics on your site.

Google Analytics has some great charts and graphs. Simple to set up and easy to use.

Analitics Home

My site traffic is growing and I would prefer to hold my own analytics on user data. Matomo is an analytics solution that stays on my server and not in the hands of Google.

Blog Growth

Google Analytics can be Slow

Sometimes the Google Analytics server is slow (affecting the speed of my server). I blogged recently about speeding up a WordPress site here and Google Servers were not adding expiry headers on assets.

I did log a ticket with Google to fix this and the experience was terrible.

Support for Google Analytics is terrible

Gogole Analytics support of terrible

GT Metrix scores show poor delivery of tracking assets.

Google Slow Assets

Privacy

After the Cambridge Analytica fiasco (that made me decide to delete facebook) sending analytics to Google is not a good idea.

  • Google Removes ‘Don’t Be Evil’ Clause From Its Code Of Conduct
  • FUTURE SOCIETY Three Signs Google Is Turning to the Dark Side
  • Top 10 Ways Google Does Evil

I am not saying Google is evil but I want my site’s visitors tracking data to remain local.

Website Speed Benchmark before installing Matomo

I can load my site in 1.3 seconds at best, 1.5 seconds on average and 2.0 seconds at worst. My site is loading 11 assets.

GTmetrix 1.3 second page load time

Page Speed Scores

GTMerix page speed load times

Y Slow Scores, Gogol Assets are reporting no expiry headers (slowing down scores)

GTMetrix yslow load times

Google Analytics tracking assets are slow.

Gmetrix waterfall list

Optimizations to be made

Browser caching is not possible with Google Analytics.

Gogole lacking browser caching

Missing Expiry Headers (I can see a Google Tag Manager server is slowing down my servers benchmark score)

Google lacking Expiry Headers

Why Mamoto (instead of Google Analytics)

I came across

Someone pointed out that @haveibeenpwned got a bunch of traction on Reddit today. With pretty much everything now either cached by @Cloudflare or served by @AzureFunctions, the first I know of a 28x traffic increase is no longer when something scales it’s when someone tells me 😎 pic.twitter.com/ifj7nQg3n4

— Troy Hunt (@troyhunt) November 5, 2018

Mamoto was mentioned

It’s an Open Source, self hostable, privacy friendly alternative to Google Analytics:https://t.co/NiK7A7uQAE

— Lukas Winkler (@lw1_at) November 5, 2018

I visited https://matomo.org/

Mamoto webpage

Snip

> Take care of running Matomo yourself by installing it on your own server. There is no cost for Matomo itself but you need a server and update Matomo & your server regularly to keep it fast and secure. Need help? The Matomo team provides free help resources and paid support.

Mamoto Setup Instruction Guide

Source Code

Source code is available.

> Matomo is the leading open alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites, apps & the IoT and visualise this data and extract insights. Privacy is built-in. We love Pull Requests! https://matomo.org/

https://github.com/matomo-org/matomo

Installation Guide

I read the installation guide here https://matomo.org/docs/installation/

You can view the changelog here https://matomo.org/changelog/

Downloading Mamoto

I logged into my server via SSH and downloaded the 18MB download to the desired folder

cd /www-root/matomo-folder/
wget https://builds.matomo.org/matomo.zip

I unzipped the zip file

unzip matomo.zip

I loaded the URL where Matoto was installed (e.g “https://fearby.com/folder/subfolder/matomo/”)

I received this well-crafted error.

Matomo File Permission Error

Raw Output

An error occurred
Matomo couldn't write to some directories (running as user 'www-usr').

Advertisement:





Try to Execute the following commands on your server, to allow Write access on these directories:

chown -R www-usr:www-usr /www-root/folder/subfolder/matomo
chmod -R 0755 /www-root/folder/subfolder/matomo/tmp
chmod -R 0755 /www-root/folder/subfolder/matomo/tmp/assets/
chmod -R 0755 /www-root/folder/subfolder/matomo/tmp/cache/
chmod -R 0755 /www-root/folder/subfolder/matomo/tmp/logs/
chmod -R 0755 /www-root/folder/subfolder/matomo/tmp/tcpdf/
chmod -R 0755 /www-root/folder/subfolder/matomo/tmp/templates_c/

If this doesn't work, you can try to create the directories with your FTP software, and set the CHMOD to 0755 (or 0777 if 0755 is not enough). To do so with your FTP software, right click on the directories then click permissions.

After applying the modifications, you can refresh the page.

I refreshed the page after running the commands above on my site (via SSH)

Matomo Setup Step 1

A system check was performed. I installed when PHP 7.2.11 was the latest, PHP 7.2.12 or higher might be available. Follow my guide to update PHP on Ubuntu.

System Check

I had one Issue with Freetype not being installed.

Install Freetype

I solved this error by installing FreeType

sudo apt-get install freetype*

Output

Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'freetype-tools' for glob 'freetype*'
Note, selecting 'freetype2-demos' for glob 'freetype*'
The following NEW packages will be installed:
  freetype2-demos
0 upgraded, 1 newly installed, 0 to remove and 66 not upgraded.
Need to get 123 kB of archives.
After this operation, 728 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 freetype2-demos amd64 2.8.1-2ubuntu2 [123 kB]
Fetched 123 kB in 0s (965 kB/s)
Selecting previously unselected package freetype2-demos.
(Reading database ... 122574 files and directories currently installed.)
Preparing to unpack .../freetype2-demos_2.8.1-2ubuntu2_amd64.deb ...
Unpacking freetype2-demos (2.8.1-2ubuntu2) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up freetype2-demos (2.8.1-2ubuntu2) ...

Then I installed “php-gd”

sudo apt-get install php-gd

Output:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libapache2-mod-php7.2 php7.2-cli php7.2-common php7.2-curl php7.2-dev php7.2-fpm php7.2-gd php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml php7.2-zip
Recommended packages:
apache2
The following NEW packages will be installed:
php-gd php7.2-gd
The following packages will be upgraded:
libapache2-mod-php7.2 php7.2-cli php7.2-common php7.2-curl php7.2-dev php7.2-fpm php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml php7.2-zip
13 upgraded, 2 newly installed, 0 to remove and 53 not upgraded.
Need to get 33.2 kB/6621 kB of archives.
After this operation, 150 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://ppa.launchpad.net/ondrej/php/ubuntu bionic/main amd64 php7.2-gd amd64 7.2.11-4+ubuntu18.04.1+deb.sury.org+1 [27.1 kB]
Get:2 http://ppa.launchpad.net/ondrej/php/ubuntu bionic/main amd64 php-gd all 2:7.2+68+ubuntu18.04.1+deb.sury.org+1 [6036 B]
Fetched 33.2 kB in 0s (75.9 kB/s)
Reading changelogs... Done
(Reading database ... 122597 files and directories currently installed.)
Preparing to unpack .../00-php7.2-mysql_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-mysql (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../01-php7.2-opcache_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-opcache (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../02-php7.2-json_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-json (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../03-php7.2-readline_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-readline (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../04-php7.2-mbstring_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-mbstring (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../05-php7.2-curl_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-curl (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../06-php7.2-zip_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-zip (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../07-php7.2-fpm_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-fpm (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../08-php7.2-xml_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-xml (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../09-php7.2-dev_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-dev (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../10-libapache2-mod-php7.2_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking libapache2-mod-php7.2 (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../11-php7.2-cli_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-cli (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Preparing to unpack .../12-php7.2-common_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-common (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) over (7.2.11-2+ubuntu18.04.1+deb.sury.org+1) ...
Selecting previously unselected package php7.2-gd.
Preparing to unpack .../13-php7.2-gd_7.2.11-4+ubuntu18.04.1+deb.sury.org+1_amd64.deb ...
Unpacking php7.2-gd (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Selecting previously unselected package php-gd.
Preparing to unpack .../14-php-gd_2%3a7.2+68+ubuntu18.04.1+deb.sury.org+1_all.deb ...
Unpacking php-gd (2:7.2+68+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-common (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Processing triggers for ureadahead (0.100.0-20) ...
Setting up php7.2-curl (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-mbstring (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-readline (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Processing triggers for systemd (237-3ubuntu10.4) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up php7.2-json (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-opcache (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-mysql (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-gd (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...

Creating config file /etc/php/7.2/mods-available/gd.ini with new version
Setting up php7.2-xml (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-zip (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-cli (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php-gd (2:7.2+68+ubuntu18.04.1+deb.sury.org+1) ...
Setting up libapache2-mod-php7.2 (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Warning: Could not load Apache 2.4 maintainer script helper.
Setting up php7.2-dev (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...
Setting up php7.2-fpm (7.2.11-4+ubuntu18.04.1+deb.sury.org+1) ...

I refreshed the Matomo setup wizard page, Freetype is now installed 🙂

FreeType is installed

Database Settings

For the life of me, I could not get Matomo to talk to a database on another server so I set it up on my localhost.

I used this guide to help in mysql CLI to create the database and users.

Enter Matomo Database settings

Commands in mysql to create a database and user and assign the user to the database. If you are not comfortable with MySql CLI you can use Adminder GUI.

CREATE DATABASE tbdatabasename;
GRANT ALL PRIVILEGES ON tbdatabasename.* TO 'databaseuser'@'localhost' IDENTIFIED BY '#####################################';
GRANT SELECT ON tbdatabasename.* TO 'databaseuser'@'localhost';

I used this PHP code to test connecting to the dedicated server before using the localhost

<?php
$servername = "localhost";
$username = "databaseuser";
$password = "#################";
$dbname = "tbdatabasename";

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);

// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
} else {
        echo "Connection Success";
}

$conn->close();
?>

Database created ok

Database OK

I created a Matomo user then I grabbed the javascript tracking ID code so I could paste this into WordPress.

Matomo Tracking ID

I opened my WordPress theme settings and deleted the Google tracking tags and added the Matomo tracking code.

Delete Google Tracking Tags

I added the Matomo tracking javascript in the head section.

The dashboard is up and collecting data.

Matomo Dashboard

Some reports are missing data so I will come back later.

After 1 week I could see data

Matomo is not collecting daya

Securing Mamoto

I read this guide here to secure Matomo

Opt Out Tracking

I enabled Opt Out Tracking in the Mamoto settings and added the generated opt-out code to my front page and at the bottom or all existing articles.

I had to allow iframe tags on my site by adding this header in NGINX (previously I blocked iframes)

add_header X-Frame-Options sameorigin

Add Opt Out Tracking Code to WordPress.

Matomo Opt Out Added to WordPRess widgets

I updated my privacy page and my GDPR notification bar. Now visitors will see a opt-out of tracking on the front page and all article pages.

Opt out of tracking enabled

SMTP Settings

I added my GSuite mail server settings to enable sending of reports via email. I loaded my old guide here to get the GSuite SMTP settings.

GSuite SMTP Settings Added

I enabled force https on the Mamoto application (edited: config/config.ini.php file)

[General]
...
force_ssl = 1

Matomo Plugins (Marketplace)

I opened the System then Plugins section of Matomo to open the Marketplace

Plugins

I installed these plugins

  • Force SSL
  • HidePasswordReset
  • Google Authenticator
  • Device Pixel Ratio
  • Bandwidth
  • Js Tracker Force Async
  • Treemap Visualization
  • Security Info
  • Custom Alerts
  • IP Reports
  • Live Tab
  • etc

Updating PHP

Matomo Admin (Panel – Security/Diagnostics) section will report if your PHP gets out of date.

Matomo warning of PHP being out of date

Hardening Advice

I enabled 2fA Authorisation at logins (Google Analytics Plugin).

Matomo 2fa Login screenshot

Read my guide here on hardware 2FA YubiCo YubiKeys here.

php.ini hardening changes

Matomo also recommended some php.ini file changes.

> open_basedir – open_basedir is disabled. When this is enabled, only files that are in the given directory/directories and their subdirectories can be read by PHP scripts. You should consider turning this on. Keep in mind that other web applications not written in PHP will not be restricted by this setting.

> upload_tmp_dir – upload_tmp_dir is disabled, or is set to a common world-writable directory. This typically allows other users on this server to access temporary copies of files uploaded via your PHP scripts. You should set upload_tmp_dir to a non-world-readable directory

This may break your WordPress so enable at your own risk. I might move Mamoto to a dedicated “analytics” subdomain then enable these options.

Troubleshooting

I had to run this command when installing Device Pixel Ratio, Device Network Information, Bandwidth plugins

php /www-root/path/matomo/console core:update

Output:

    *** Update ***

    Database Upgrade Required

    Your Matomo database is out-of-date, and must be upgraded before you can continue.

    The following dimensions will be updated: log_visit.device_pixel_ratio.

    *** Note: this is a Dry Run ***

    ALTER TABLE `matomo_log_visit` ADD COLUMN `device_pixel_ratio` DECIMAL(5,2) DEFAULT NULL;

    *** End of Dry Run ***

A database upgrade is required. Execute update? (y/N) y

Starting the database upgrade process now. This may take a while, so please be patient.

    *** Update ***

    Database Upgrade Required

    Your Matomo database is out-of-date, and must be upgraded before you can continue.

    The following dimensions will be updated: log_visit.device_pixel_ratio.

    The database upgrade process may take a while, so please be patient.

  Executing ALTER TABLE `matomo_log_visit` ADD COLUMN `device_pixel_ratio` DECIMAL(5,2) DEFAULT NULL;... Done. [1 / 1]

Matomo has been successfully updated!

GTMetrix (After)

GT Metrix reports that my site is not slower (still 1.5 seconds)

GTMetrix After Pagespeed

I can see that some JavaScript is not being picked up by CDN.

GTMetrix After YSlow

Also 2 More files loading (when compared to Google Analytics)

2 More Files

Time to add the Mamoto files to my CDN.

Adding Matomo Resources to a CDN

I read this Matomo forum post.

I copied these 2 assets to my WordPress wp-content folder (my WordPress CDN ewww.io will then upload them to the CDN).

cd /www-root/wp-content/
cp /www-root/utils/matomo/piwik.js ./piwik.js
cp /www-root/utils/matomo/plugins/CoreAdminHome/javascripts/optOut.js ./optOut.js
chown www-data:www-data *.js

I have cache everything enabled in ewww.io and this will copy the javascript assets ot my CDN.  I will need to manually update these js files each time a Matomo update is installed.

I change my Matomo tracker code to include the new CDN location

<!-- Matomo -->
<script type="text/javascript">
  var _paq = _paq || [];
  /* tracker methods like "setCustomDimension" should be called before "trackPageView" */
  _paq.push(['trackPageView']);
  _paq.push(['enableLinkTracking']);
  (function() {
    var u="//fearby.com/utils/matomo/";
    _paq.push(['setTrackerUrl', u+'piwik.php']);
    _paq.push(['setSiteId', '1']);
    var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
    g.type='text/javascript'; g.async=true; g.defer=true; g.src='https://fearby-com.exactdn.com/wp-content/piwik.js'; s.parentNode.insertBefore(g,s);
  })();
</script>
<!-- End Matomo Code -->

I could not find out how to change the location of my (now CDN cached https://fearby-com.exactdn.com/wp-content/optOut.js) so I temporarily disabled the opt-out form on my front page.

todo: Find out how to change the CDN location of optOut.js and re-enabled the form.

All assets are loading from CDN.

GT Metrix shows my site loads in 1.4 seconds

Analytics Reporting

Graphs are not as pretty as Google Analytics but they are working.

Matomo is not collecting daya

Mobile Reporting

Mobile reporting is good too.

Screenshot of the Matomo Mobile app

Updating Matomo Plugins

Don’t forget to update your plugins from the Matomo dashboard.

Updating Matomo (Core)

Matomo has an official guide on how to update Matomo here.

I do not have FTP so I will perform the manual three step update.

But before I do that I will manually backup my web server and database server just in case.

I backed up my Matomo config (I SSH”ed to the server)

$ cd /www-root/matomo-root/

$ cp ./config.ini.php ./config.ini.3.x.x.php

I navigated to the folder above my Matomo folder

$ cd ..

$ cd ..

I downloaded Matomo

$ wget https://builds.matomo.org/matomo.zip

I unzipped the zip file

$ unzip -o matomo.zip

I removed the matomo.zip file

$ rm matomo.zip

I loaded the Matomo Login page again and was prompted to update the database.

Matomo Database Update Required

Matomo reported it was updated Successfully.

Matomo was updated message

Oops, and error in config error appeared when I tried to log in.

Matomo Error in config

Oh, Do I need to replace the config file with my backed up config file?

(edit: Yes Matomo say to do this, my bad)

Ten seconds later I accidentally deleted all my config files (I had zero backups), the next 2 minutes were spent shutting down my servers (web and db) and restoring them from backup. Thank goodness UpCloud are awesome hosts.

I now had to restore my servers and repeat the steps but this time restore my config file before logging back in.

I did this but had the same error

> An error occurred
> Authentication object cannot be found in the container. Maybe the Login plugin is not activated?
> You can activate the plugin by adding:
> Plugins[] = Login under the [Plugins] section in your config/config.ini.php

I checked my replaced config.ini.php and it did have

> [PluginsInstalled]
> PluginsInstalled[] = “Login”

I googled and found this page that said reset your password (this was not an option as Matomo was not loading)

I logged into mysql with my Matomo user

> mysql -u matomodbusername -p
> Enter password:
> Welcome to the MySQL monitor. Commands end with ; or \g.
> Server version: 5.7.xxxx

> Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

> Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

> Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

> mysql> show databases;
+——————–+
| Database |
+——————–+
| information_schema |
| matomodb |
+——————–+
> 2 rows in set (0.00 sec)

The account and database seem ok.

I tried “FLUSH PRIVILEGES;” with no luck

I tried to sop mysql but it was locked

It was late so I rebooted my server (it did not come back up after a few minutes, I forced a reboot)

I still had an “Authentication object cannot be found in the container.” error when trying to login to Matomo???

I re-checked the “config.ini.php” file after reding threads at the Matomo Forums

$ sudo nano /www-root/matomo-root/config.ini.php

“Plugins[] = “Login”” was not in the “[Plugins]” area of the file???  I added it, saved the change and was able to reload the Matomo GUI.

I checked some key reports.

Visitors over time:

Visitors over time report

Visitor Location Map

Visitor Location Map

Visitor Overview

Visitor Overview

Out links Clicked

Out links Clicked

Nice

I subscribed to the Matomo newsletter here to keep up to date with Matomo update releases: https://matomo.org/newsletter/

Good luck and I hope this guide helps someone

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.2 Hardening info

v1.1 Updating Matomo

v1.0 Initial post

Filed Under: Analytics, Cloud, Free, Privacy Tagged With: a, analytics, focused, for, google, hosted, locally, Matomo, Open, Piwik, privacy, Replacing, solution, source, with

Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker

November 7, 2018 by Simon

This is how I checked the compatibility of my WordPress theme and plugin (code) with PHP Compatibility Checker

Aside

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. PHP is my programming language of choice.

Now on with the post

Snip from: https://wordpress.org/plugins/php-compatibility-checker/

What is PHP Compatibility Checker

> The WP Engine PHP Compatibility Checker can be used by any WordPress website on any web host to check PHP version compatibility.

> This plugin will lint theme and plugin code inside your WordPress file system and give you back a report of compatibility issues for you to fix. Compatibility issues are categorized into errors and warnings and will list the file and line number of the offending code, as well as the info about why that line of code is incompatible with the chosen version of PHP. The plugin will also suggest updates to themes and plugins, as a new version may offer compatible code.

> This plugin does not execute your theme and plugin code, as such this plugin cannot detect runtime compatibility issues.
Please note that linting code is not perfect. This plugin cannot detect unused code-paths that might be used for backwards compatibility, and thus might show false positives. We maintain a whitelist of plugins that can cause false positives. We are continuously working to ensure the checker provides the most accurate results possible.
This plugin relies on WP-Cron to scan files in the background. The scan will get stuck if the site’s WP-Cron isn’t running correctly. Please see the FAQ for more information.

Install PHP Compatibility Checker

PHP Compatibility Checker

I instaled by SSH’ing to my server and opening my WP Plugins folder

cd /www-root/wp-content/plugins/

I grabbed the latest download URL from here (hover over the download button), at the time of writing this was the latest version: https://downloads.wordpress.org/plugin/php-compatibility-checker.1.4.6.zip

I downloaded the plugin on my server (then unzipped it and deleted the zip)

wget https://downloads.wordpress.org/plugin/php-compatibility-checker.1.4.6.zip
unzip php-compatibility-checker.1.4.6.zip
rm php-compatibility-checker.1.4.6.zip

Enable PHP Compatibility Checker Plugin

I enabled the plugin

Enable the Plugin

I clicked on the following message

> You have just activated the PHP Compatibility Checker. Start scanning your plugins and themes for compatibility with the latest PHP versions now!

Start Scan

I already have PHP 7.2 installed so let’s scan my site. PHP 7.3 will be available in December and it is already being tested in beta.

Scan PHP 7.2

PHP Versions

PHP Versions

Site Scanning

PHP Compatibility Checker site scanning is very business-like

Site Scan Progress

PHP Compatability Checker Scan Results

2 of 22 plugins I use were not PHP 7.2 compatible (WordFence and WP Meta SEO)?

PHP Compatibility Report

I read on twitter that Wordfence may be a false positive.

Clicking toggle details reveal why the scan failed. A Two Factor Auth plugin was all OK.

Scan Results

Your results will hopefully be…

> PHP 7.2 compatible

Of if errors exist it should explain why it did not pass.

FILE: /www-root/wp-content/plugins/wp-meta-seo/jutranslation/jutranslation.php
> —————————————————————————————-
> FOUND 1 ERROR AFFECTING 1 LINE
> —————————————————————————————-
> 251 | ERROR | The function is_countable() is not present in PHP version 7.2 or earlier
> —————————————————————————————-

I can’t wait for PHP 7.3 scanning.  I will update this post in December 2018 after PHP 7.3 is released.

Good luck and I hope this guide helps someone

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.0 Initial post

Filed Under: Compatibility, PHP Tagged With: and, check, Checker, code, compatibility, of, php, plugin, the, theme, with, wordpress, your

Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App

October 28, 2018 by Simon

Here is a quick guide to show you how to add two-factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA authenticator app

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line.

Why Secure WordPress

WordPress CMS is a widely targeted CMS for hackers. View the official WordPress stats on WordPress Version/PHP and MySQL Version. View WordPress vulnerabilities here.

Read the Sucuri 2017 report on reported WordPress Hacks here (spoiler 34,371 infected websites in 2017).

Plugins exist to secure and scan WordPress. Read my blog post here on the now-retired Gravityaity Scan plugin and the awesome WordFence security plugin.

You (and hackers) can scan your site with https://wpscans.com/ or other open-source tools like wp-scan from OWASP ZAP. If you manage a WordPress site I’d recommend you install Kali Linux to scan your site.

Running a wp scan in Kali Linux is easy.

wpscan --url https://fearby.com --debug-output 2> ~/Desktop/wpscan.txt

The output from the Kali Linux wpscan tool

WPscan tool in KaiLinux

What are Hardware YubiCo YubiKeys

Read my guide here to see what YubiCo YubiKeys are and how to use them.

Yubico YubiKeys

Get the Two-Factor Plugin for WordPress Plugin

Plugin: https://en-au.wordpress.org/plugins/two-factor/

Two-Factor

Plugin Page at WordPress.org

Two Factor Auth Plugin

The source code for this plugin is available (nice): https://github.com/georgestephanis/two-factor. This plugin was updated 2 weeks ago (nice).

Downloading the Plugin

FYI: I do not allow downloading or updating of plugins in WordPress (via FTP), I prefer SSH manual downloading. FTP plugin installation and updating are not allowed on my site.

I got the latest download URL (e.g. https://downloads.wordpress.org/plugin/two-factor.zip) by copying the URL from the download button above.

I connected to my server via SSH and navigated to my WordPress plugin folder

cd /your-www-root/wp-content/plugins

I download the plugin.

[email protected]:/your-www-root/wp-content/plugins# wget https://downloads.wordpress.org/plugin/two-factor.zip
--2018-10-28 14:44:27--  https://downloads.wordpress.org/plugin/two-factor.zip
Resolving downloads.wordpress.org (downloads.wordpress.org)... 198.143.164.250
Connecting to downloads.wordpress.org (downloads.wordpress.org)|198.143.164.250|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 47882 (47K) [application/octet-stream]
Saving to: 'two-factor.zip'

two-factor.zip                             100%[=======================================================================================>]  46.76K  --.-KB/s    in 0.001s

2018-10-28 14:44:27 (37.1 MB/s) - 'two-factor.zip' saved [47882/47882]

I extracted the plugin zip file

[email protected]:/your-www-root/wp-content/plugins# unzip two-factor.zip
Archive:  two-factor.zip
   creating: two-factor/
   creating: two-factor/assets/
  inflating: two-factor/assets/banner-1544x500.png
  inflating: two-factor/assets/banner-772x250.png
  inflating: two-factor/assets/icon-128x128.png
  inflating: two-factor/assets/icon-256x256.png
  inflating: two-factor/class.two-factor-core.php
   creating: two-factor/includes/
  inflating: two-factor/includes/function.login-header.php
   creating: two-factor/includes/Google/
  inflating: two-factor/includes/Google/u2f-api.js
   creating: two-factor/includes/Yubico/
  inflating: two-factor/includes/Yubico/U2F.php
   creating: two-factor/providers/
  inflating: two-factor/providers/class.two-factor-backup-codes.php
  inflating: two-factor/providers/class.two-factor-dummy.php
  inflating: two-factor/providers/class.two-factor-email.php
  inflating: two-factor/providers/class.two-factor-fido-u2f-admin-list-table.php
  inflating: two-factor/providers/class.two-factor-fido-u2f-admin.php
  inflating: two-factor/providers/class.two-factor-fido-u2f.php
  inflating: two-factor/providers/class.two-factor-provider.php
  inflating: two-factor/providers/class.two-factor-totp.php
   creating: two-factor/providers/css/
  inflating: two-factor/providers/css/fido-u2f-admin.css
   creating: two-factor/providers/js/
  inflating: two-factor/providers/js/fido-u2f-admin-inline-edit.js
  inflating: two-factor/providers/js/fido-u2f-admin.js
  inflating: two-factor/providers/js/fido-u2f-login.js
  inflating: two-factor/readme.md
  inflating: two-factor/readme.txt
  inflating: two-factor/two-factor.php
  inflating: two-factor/user-edit.css

Enable the Plugin

Don’t forget to update the plugin in WordPress.

Enable the Plugin in WordPress

Once the plugin is enabled I can setup Two-factor authentication

Edit your Users

To setup two-factor authentication open your WordPress users screen (/wp-admin/users.php).

WordPress Users List /wp-admin/users.php

Notice the Two-Factor column

Edit your desired user to enable two-factor login options

Scroll down to Two Factor Options header, you will see a QR code that you can scan with your two-factor authentication app (e.g Google Authenticator or YubiCo Authenticator).

Enable 2FA via plugin

Always generate and save backup codes in case you lose your YubiKeys or authenticator app.

You can enable authentication methods as required.

Add the code to your Authenticator app. I will add mine to my Yubico Authenticator app that requires the insertion of a physical YubiKey. I can read my YubiKey via NFC and use my mobile phone to generate one time passwords too. Read here to learn about YubiKey 2FA (touch) devices. I have secured my Ubuntu/Debian and macOSX with these keys,

TIP: Don’t forget to save the user after editing.

Add the YubiKey 2FA (touch) to WordPress logins.

While editing a user click Register New Key under Security Keys

Add the YubiKey 2FA to WordPress

Add your primary and backup YubiKey as required (I added both of mine).

Screenshot showing two YubiKeys added to WordPress.

Enable all desired 2FA options

  • Email (OFF)
  • Time based One-Time Password (Authenticator App) (ON)
  • FIDO Universal 2nd Factor (U2F) – YubiKey Insertion and touch (ON)
  • Backup Codes (ON)

Set all desired 2FA login methods

TIP: Don’t forget to save the user after editing.

Users Table

Aim to set up every user who has access to your WordPress to use 2FA.

Mobile 2FA login

I tested logos via mobile and I was prompted to tab my YubiKey to my phone. Nice.

What happens at login?

When One Time Password is enabled as the primary authentication method I am prompted for a one-time password after entering my username and password. I then need to insert my YubiKey (or tap the YubiKey to my phone (via NFC)) to generate a one time password.

Screenshot of 2FA login prompt

When FIDO is enabled I need to insert my YubiKey and press the button.

Enter Security Key

Conclusion

I can now secure my WordPress site with 2FA protections without expensive security plugins.

I hope this guide helps someone.

More

Read more here

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.1 Added Mobile login details

v1.0 Initial post

Filed Under: 2FA, 2nd Factor, Auth, Authorization, Blog, MFA, NFC, owasp, Security, SSH, Vulnerability, Yubico, YubiKey Tagged With: 2FA, add, and, app, auth, authenticator, factor, hardware, login, or, Protection, to, two, with, wordpress, Yubico, YubiKeys

Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc

October 3, 2018 by Simon

This is a draft post showing how you can monitor the performance of a server (or servers) with NixStats and receive alerts by SMS, Push, Email, Telegram etc

fyi: This is not a paid post, this is just me using the NixStats software to monitor my servers and send alerts.

Finding a good host

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Domain names for just 88 cents!

Monitoring Servers

The post below will show you how you can monitor servers online with https://nixstats.com/ and send alerts when resources reach limits or servers fail.

https://nixstats.com/

I signed up and started a Nixstats (14 day free trial).

Start Nixstats Trial

After I created an account I was emailed by Nixtsats with agent install instructions for Linux (1 line). I was also advised to add contacts and to set up alerts.

I logged into the Nixstats settings and set up…

  • My Timezone
  • Default reporting period
  • First name and Surname
  • Reporting emails
  • etc

Nixstats Subscription Upgrade

Subscription options

  • Free (5 monitors, 1 server, 24-hour data retention etc)
  • Founder (25 monitors, 10 servers, 30-day data retention etc)
  • Business (100 monitors, 15 servers, 60-day data retention etc)

Subscription Options

I enabled the limited founder subscription so I can monitor 10x servers (this deal is too good to miss). I tried creating a status page myself last year and it is terribly hard.

Subscriptiosn page

I am now out of the free trial period 🙂 Let’s start monitoring many servers.

Subscription Activated

I enabled two factor Auth to Nixstats logins

Nixstats Two Factor Auth

I created a Nixstats API key for future use (watch this space)

Create API Key

I installed the Nixstats agent (the dashboard gave a 1 line command you can run as root to install the agent (on Linux)).

Instal Nixstats Agent

FYI: Command (######################## is a number linked to your account)

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################

Output

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################
--2018-10-02 09:53:56--  https://www.nixstats.com/nixstatsagent.sh
Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8013, 2400:cb00:2048:1::6819:8113, 104.25.128.19, ...
Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8013|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 38708 (38K) [application/octet-stream]
Saving to: 'nixstatsagent.sh'

nixstatsagent.sh 100%[====================================================>] 37.80K –.-KB/s in 0.1s

2018-10-02 09:53:56 (338 KB/s) – ‘nixstatsagent.sh’ saved [38708/38708]

Found Ubuntu …
Installing …
Installing Python2-PIP …
Installing nixstatsagent …
Generation a server id …
Got server_id: ######################
Creating and starting service
Created symlink /etc/systemd/system/multi-user.target.wants/nixstatsagent.service -> /etc/systemd/system/nixstatsagent.service.
Created the nixstatsagent service

Server Dashboard
below is a summary of all connected servers ( https://nixstats.com/dashboard/servers ).

Server Sumamry

Monitor Setup

I set up a number of monitors to monitor ping replies and https traffic

Monitors

Advanced Monitoring

I can also set the monitor credentials, timeouts, retries, auth methods, max redirects and frequency. If you server blocks login or resource GET attempts you may need to whitelist IP’s. IP’s of monitoring servers are located here https://nixstats.com/whitelist.php

Monitoring advanced options

Monitor Summary

The default dashboard is very informative. Feel free to create your own dashboards that focus on your own infrastructure or apps.

Monitor Summary

Individual Server Reports

You can click on a server and monitor it in detail.

Nixstats Graphs

Server Memory Graphs

Long-term memory graphs.

Memory Graph

Install Optional Nixstats Plugins

Nixstats offers many plugins to monitor software that is installed on your server (e.g NGINX, MySQL, PHP etc).

1) NGINX Monitoring (Plugin)

To enable NGINX monitoring I read https://help.nixstats.com/en/article/monitoring-nginx-50nu7f/

I edited my NGINX sites-enabled config.

sudo nano /etc/nginx/sites-enabled/default

I added the following

server {
    listen 127.0.0.1:8080;
    server_name localhost;
    location /status_page {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }
}

I tested, reloaded and restarted NGINX

nginx -t
nginx -s reload
/etc/init.d/nginx restart

The status page will only be available on the local machine, I tested the page on the local machine

wget -qO- http://127.0.0.1:8080/status_nginx
Active connections: 3
server accepts handled requests
 15 15 31
Reading: 0 Writing: 1 Waiting: 2

It’s Working.

I edit /etc/nixstats.ini

sudo nano /etc/nixstats.ini

I remove comments before these lines to enable the plugin

[nginx]
enabled = yes
status_page_url = http://127.0.0.1:8080/status_nginx

I ran the following command to see if NGINX monitoring is possible

nixstatsagent --test nginx

Output

nginx:
{
    "accepts": 39,
    "accepts_per_second": 0.0,
    "active_connections": 6,
    "handled": 39,
    "handled_per_second": 0.0,
    "reading": 0,
    "requests": 119,
    "requests_per_second": 0.0,
    "waiting": 5,
    "writing": 1
}

It’s Working

I restarted the nixstatsagent

service nixstatsagent restart

I can now view NGINX properties like active_connections in my dashboard. 🙂

2) Enable PHP-FPM Monitoring (Plugin)

Looks like a PHP-FPM monitoring was recently added lets set that up too. Read my guide on setting up PHP child workers here.

We’ve added a premade dashboard for PHP-FPM. If you’re not yet monitoring PHP-FPM take a look at the integration guide https://t.co/X4ywRHw9hX pic.twitter.com/aag1fTsr3R

— Nixstats (@nixstats) September 6, 2018

To enable PHP-FPM monitoring I read https://help.nixstats.com/en/article/monitoring-php-fpm-1tlyur6/

I edited my PHP-FPM ini file

sudo nano /etc/php/7.2/fpm/pool.d/www.conf

I added the following line

pm.status_path = /status_phpfpm

Restart PHP

sudo service php7.2-fpm restart

I added the following to /etc/nginx/sites-enabled/default localhost server block added above Note I use php 7.2 below (read more here).

server {
    listen 127.0.0.1:8080;
    server_name localhost;

location /status_phpfpm {
access_log off;
allow 127.0.0.1;
deny all;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass unix:/run/php/php7.2-fpm.sock;
log_not_found off;
}
}

I tested, reloaded and restarted NGINX

nginx -t
nginx -s reload
/etc/init.d/nginx restart

I restart PHP-FPM

sudo systemctl restart php7.2-fpm

Enabled the plugin in /etc/nixstats.ini

[phpfpm]
enabled = yes
status_page_url = http://127.0.0.1:8080/status_phpfpm?json

I tested the status page

wget -qO- http://127.0.0.1:8080/status_phpfpm?json

Output:

{
	"pool":"www",
	"process manager":"static",
	"start time":1538654543,
	"start since":178,
	"accepted conn":28,
	"listen queue":0,
	"max listen queue":0,
	"listen queue len":0,
	"idle processes":49,
	"active processes":1,
	"total processes":50,
	"max active processes":2,
	"max children reached":0,
	"slow requests":0
}

I tested the agent

 nixstatsagent --test phpfpm

Output:

phpfpm:
{
    "accepted_conn": 51,
    "accepted_conn_per_second": 0.0,
    "active_processes": 1,
    "idle_processes": 49,
    "listen_queue": 0,
    "listen_queue_len": 0,
    "max_active_processes": 2,
    "max_children_reached": 0,
    "max_listen_queue": 0,
    "pool": "www",
    "process_manager": "static",
    "slow_requests": 0,
    "start_since": 318,
    "start_time": 1538654543,
    "total_processes": 50
}

I can now query PHP-FPM status values in Nixstats 🙂

Query PHP-FPM

Enable MySQL Monitoring (Plugin)

To enable MySQL monitoring I read https://help.nixstats.com/en/article/monitoring-mysql-1frskd8/

I edited the nixstats.ini

sudo nano /etc/nixstats.ini

I enabled the mysql section in nixstats.ini and added my mysql credentials

[mysql]
enabled=yes
username=mysqluser
password=#######################
host=127.0.0.1
database=mysql
port=3306
socket=null

I ran this command to test MySQL querying

nixstatsagent --test mysql
mysql:
Load error: No module named MySQLdb

I had an error.

A quick Google revealed I had to install a mysql python module.

sudo apt-get install python-mysqldb

Allow localhost to connect to MySQL

Edit /etc/mysql.cnf and allow all localhost and external connections (I could not bind to localhost and an external IP at the same time)

bind-address    = 0.0.0.0

TIP: Ensure you have firewalled access to your MySQL server, never open it up without protection.

Let’s try again

nixstatsagent --test mysql

Output

mysql:
{
    "aborted_clients": 0,
    "aborted_connects": 0,
    "binlog_cache_disk_use": 0,
    "binlog_cache_use": 0,
    "bytes_received": 0,
    "bytes_sent": 0,
    "com_delete": 0,
    "com_delete_multi": 0,
    "com_insert": 0,
    "com_insert_select": 0,
    "com_load": 0,
    "com_replace": 0,
    "com_replace_select": 0,
    "com_select": 0,
    "com_update": 0,
    "com_update_multi": 0,
    "connections": 0,
    "created_tmp_disk_tables": 0,
    "created_tmp_files": 0,
    "created_tmp_tables": 0,
    "key_read_requests": 0,
    "key_reads": 0,
    "key_write_requests": 0,
    "key_writes": 0,
    "max_used_connections": 3.0,
    "open_files": 14.0,
    "open_tables": 316.0,
    "opened_tables": 0,
    "qcache_free_blocks": 1.0,
    "qcache_free_memory": 16760152.0,
    "qcache_hits": 0,
    "qcache_inserts": 0,
    "qcache_lowmem_prunes": 0,
    "qcache_not_cached": 0,
    "qcache_queries_in_cache": 0,
    "qcache_total_blocks": 1.0,
    "questions": 0,
    "select_full_join": 0,
    "select_full_range_join": 0,
    "select_range": 0,
    "select_range_check": 0,
    "select_scan": 0,
    "slave_open_temp_tables": 0.0,
    "slow_launch_threads": 0,
    "slow_queries": 0,
    "sort_range": 0,
    "sort_rows": 0,
    "sort_scan": 0,
    "table_locks_immediate": 0,
    "table_locks_waited": 0,
    "threads_cached": 2.0,
    "threads_connected": 1.0,
    "threads_created": 0,
    "threads_running": 1.0,
    "uptime": 35837.0
}

Nice,

I restart MySQL

sudo systemctl restart mysql

I restart my Nixstats service

service nixstatsagent restart

Now let’s monitor MySQL in Nixstats

I can now view MySQL metrix

MySQL MEtrix

Status Page

Nixstats allows you to create a status page ( https://nixstats.com/pages/overview ) where you can add any servers or monitors to that page. This stats page is truly awesome, it builds a live status page based on data coming from your installed agents.

You can even set up a custom subdomain that points to a Nixstats hosted status page too (e.g https://status.yourdomain.com).

FYI: An SSL certificate on your staus page may take a few hours to set up. Don’t panic if it is not instantatly available.

Custom Status Page

Nice.

This saves doing it yourself. The status page will look like it running on your server.

Status Page

You can create a status page that automatically aggregates collected data from monitors and displays them in a nice layout.

Status Page

This is great, I used to do my own status pages but not anymore.

Alerts

I added a contact so I could receive alerts. I could then add my mobile, email and PushOver key (to receive push notifications) and Telegram Bot API token.

Contact

Test Alerts

I sent a test alert to each service against the contact.

Test Alerts

I activated a Pushover licence on my Android device for about $7.49 AUD (one off) to ensure I keep getting Push Notifications.

Bought licence for PushOver

Nixstats have links that show you how you can create a Telegram Bot and Pushover.net account.

Pushover will cost about $5 USD one off per device (see faq).

I created the following alerts

Alert: Disk Usage higher than 90%

Alert Disk Usage higher than 90%

Alert: Load greater than 90 per cent for 1 minute

Alert load greater than 90 percent for 1 minute

Alert: Less than 5 percent memory free.

Alert less than 5 percent memory free

Summary of alerts.

Alert Sumamry

I also added a CPU reached 95% one for 5 mins alert too (but it’s not pictured above)

I forgot to specify alert recipients and methods for each alert so I edited each alert and added the contact.

Selected Alert Recipients and methods

Now it’s time to test the alerts.

I shut down a server to test alerts

shutdown -h now

Alerts to my defined Email, SMS, Push and Telegram are working a treat 🙂

Alerts Working

After I rebooted the server I also received alerts about the server being back up.

The status page showed the server that was offline too.

Server Offline

Nice

Troubleshooting

I had an issue instaling the agent on Debian

I ran the following command

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh #######################
--2018-10-02 00:41:38--  https://www.nixstats.com/nixstatsagent.sh
Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8113, 2400:cb00:2048:1::6819:8013, 104.25.129.19, ...
Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8113|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'nixstatsagent.sh' not modified on server. Omitting download.

nixstatsagent.sh: line 508: [: Installer exited with error code 0. See nixstatsagent.log for details.: integer expression expected

An error occurred, please check the install log file (nixstatsagent.log)!

Contents of nixstatsagent.log

cat nixstatsagent.log
Ign:1 http://deb.debian.org/debian stretch InRelease
Hit:2 http://deb.debian.org/debian-security stretch/updates InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Ign:5 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic InRelease
Ign:7 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic Release
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Hit:9 https://packages.sury.org/php stretch InRelease
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Err:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
  404  Not Found
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Reading package lists...
W: The repository 'http://ppa.launchpad.net/ondrej/php/ubuntu cosmic Release' does not have a Release file.
E: Failed to fetch http://ppa.launchpad.net/ondrej/php/ubuntu/dists/cosmic/main/binary-amd64/Packages  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
nixstatsagent.sh: line 118: apt-get upgrade returned error code 100. Please see nixstatsagent.log for details.: command not found

I asked the Nixstats chat help and I was advised I had a dead repository (I removed this (editing the dead repo in the appropriate file in /etc/apt/) and all was ok)

I had trouble testing my Telegram alerts but it was my fault as I forgot to follow the bot account I created. Telegram does not allow message from a user (bot) unless you follow them.

A chat with the Nixstats staff sorted me out. Thanks, Nixstats chat team.

Nixstats chat

I had an issue with a missing python mysql package

Load error: No module named MySQLdb

I solved it by instaling python-mysqldb

sudo apt-get install python-mysqldb

Nixstats Help

Nixstats have a help subdomain: https://help.nixstats.com/en/

Nixstats Help

Error Logs Plugin

I did ask Nixstats on Twitter and they said they are working on a logging plugin, I can’t wait for that.

We’re launching a closed beta for Logging at Nixstats. Contact us to get setup! You can search and tail log files across all your servers! pic.twitter.com/FIeip2SOUw

— Nixstats (@nixstats) October 4, 2018

I now have access to beta log features and can see log tabs in Nixstats

I had or check the version of my rsyslogd

rsyslogd -v
rsyslogd 8.32.0, compiled with:
        PLATFORM:                               x86_64-pc-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        systemd support:                        Yes
        Number of Bits in RainerScript integers: 64

I edited: /etc/rsyslog.d/31-nixstats.conf

I pasted

##########################################################
### Rsyslog Template for Nixstats ###
##########################################################

$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

template(name=”NixFormat” type=”string”
string=”<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [[email protected] tag=\”rsyslog\”] %msg%\n”
)

action(type=”omfwd” protocol=”tcp” target=”log.nixstats.com” port=”514″ template=”NixFormat”)
#################END CONFIG FILE#########################

I restarted the rsyslog service

sudo service rsyslog restart

Live Log Output

I can see a live log from (unknown) logs.

I can see the firewall blocking access to certain ports.

Live Log

Search logs

Search

Blacklist Checking (Beta)

Nixstats tweeted “We just launched a new blacklist check feature. Monitor your IP and hostname reputation. Free during the Beta!”

I enabled it.

I received a Blacklist notification

I requested removal at junkmailfilter.com

Thanks, Nixstats

Conclusion

This is one of the best software packages I have seen in a while. I have developed status

pages in the past (no more). Great work Nixstats.

Please check out Nixstats today, they are awesome. Signup for a free account and consider the limited time founder plan (it’s a bargain).

Nixstats Live chat support is awesome

Server Plug

If you need a server, consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.3 Blacklist beta

V1.2 Logs beta

v1.1 Logging Tweet

v1.0 Initial Post

Filed Under: Alert, Analytics, Cloud, Domain, Monitor Tagged With: alerts, and, by, email, etc, monitor, NixStats, Performance, push, receive, server, SMS, Telegram, with

How to install Windows 10 Pro alongside an OSX partition with Apple Boot Camp

April 20, 2018 by Simon

This guide will show you how to install Windows 10 Pro alongside an OSX partition with Apple Boot Camp.

I have a number of guides on moving away from CPanel, Setting up VM’s on UpCloud, AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. I use OSX to develop and occasionally need to use Windows to achieve some development tasks, this is how you can install Windows on an Apple PC.

Apple Boot Camp

Apple Bootcamp software can be found here: https://support.apple.com/en-au/HT201468

Bootcamp

Bootcamp Steps.

BootStrap Steps

You can download a Windows 10 ISO here: https://www.microsoft.com/en-au/software-download/windows10

Download ISO

You will need to plug in a 16GB or larger USB key to use as an install medium.

Download Boot Camp

Download AppBoot Camp here: https://support.apple.com/en-au/boot-camp

Start Apple Boot Camp, click Continue, Tick all options, select the Windows 10 ISO file (beware Bootcamp will select the first ISO file it finds, ensure your USB is selected, Click Continue (tip: You may need to er format and prepare the USB key 10 times, Apple Bootcamp is not the most reliable program), It may take 6 hours to copy files pre Window Setup).

Bootstrap Steps

Choose your Windows partition size (at least 200GB is ideas)

Partition

When the Bootcamp Wizard is complete you can reboot into Windows (automatic at the end of the wizard)

Windows Booting

Enter the desired details in each step (e.g Local, Windows Key, Windows Version, Partition and Country etc)

Windows Setup Options

Loads more steps like Keyboard Layout, Domain, Microsoft Account, Password and Cortana and Privacy etc.

Windows Setup Options

Windows is now installed, Apple Bootcamp will set up appropriate drivers for your Mac,

Instaling

While this is happening I will install Google Chrome Canary.

Chrome

Bootcamp has now finished setting up drivers.

Bootstrap Done

Windows 10 Start menu (I prefer OSX’s simplicity)

Windows Strart Menu

You can set up your prefered Startup disk in the Apple System Preferences or press the Option key on startup and choose a partition to boot.

Bootup Options

Update: Open a Windows 10 Boot Camp Installation on OSX in Parallels (like a VM)

Read this guide will show you how you can open a Windows 10 Boot Camp Installation on OSX in Parallels (like a VM).

Windows

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.1 Added Open Bootcams as VM

v1.0 Initial post

Filed Under: Bootcamp, Windows Tagged With: alongside, an, Apple, Boot Camp, How, install, OSX, partition, pro, to, Windows 10, with

Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare

April 14, 2018 by Simon

This post will show you how you can setup Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM) Signatures and Domain-based Message Authentication, Reporting, and Conformance (DMARC) on your GMail (G Suite) email to limit spam and increase security.

I have a number of guides on moving away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. I use Google G Suite to send and receive emails that are linked to my domain (even via the command line) using multiple domains (with aliases).

For the best performing VM host (UpCloud) read my guide on the awesome UpCloud VM hosts (get $25 free credit by signing up here).

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Buy how can you extend your email security and limit spam?

Enter..

Sender Policy Framework (SPF) for Authorizing Use of Domains in Email

Background: SPF summary from the RFC document from the Internet Engineering Task Force (IETF).

“Email on the Internet can be forged in a number of ways. In particular, existing protocols place no restriction on what a sending host can use as the “MAIL FROM” of a message or the domain given on the SMTP HELO/EHLO commands. This document describes version 1 of the Sender Policy Framework (SPF) protocol, whereby ADministrative Management Domains (ADMDs) can explicitly authorize the hosts that are allowed to use their domain names, and a receiving host can check such authorization.”

Google has a guide in setting up SPF records for your G Suite account.

Scan your site for SPF, DKIM and DMARC configuration(s).

Gmail has a test site where you can check your site SPF, DKIM and DMARC etc: https://toolbox.googleapps.com/apps/checkmx/

Secure GSuite

How to set up an SPF Record

I followed this guide to set up an SPF record on my G Suite account. I use Cloudflare for my DNS provider so I’ll make my DNS changes there.

Add SPF Record

Update: Google instructions were wrong, use a TXT record and not a SPF record.

Read more on SPF at Wikipedia here.

DNS will take a while to replicate so do wait a few hours before checking again with the checkmx tool.

Now let’s set up DomainKeys Identified Mail (DKIM) Signatures

Read more on DKIM at Wikipedia here.

Background: The DKIM RFC form the Internet Engineering Task Force (IETF) states…

“DomainKeys Identified Mail (DKIM) permits a person, role, or organization that owns the signing domain to claim some responsibility for a message by associating the domain with the message. This can be an author’s organization, an operational relay, or one of their agents. DKIM separates the question of the identity of the Signer of the message from the purported author of the message. Assertion of responsibility is validated through a cryptographic signature and by querying the Signer’s domain directly to retrieve the appropriate public key. Message transit from author to recipient is through relays that typically make no substantive change to the message content and thus preserve the DKIM signature.”

Google has a DKIM FAQ: https://support.google.com/a/answer/174124

Login to your G Suite account and load this FAQ.

The FAQ page states..

“You can help prevent spoofing by adding a digital signature to outgoing message headers using the DKIM standard. This involves using a private domain key to encrypt your domain’s outgoing mail headers, and adding a public version of the key to the domain’s DNS records. Recipient servers can then retrieve the public key to decrypt incoming headers and verify that the message really comes from your domain and hasn’t been changed along the way.”

Click Generate the Domain Key

Generate Domain Key

Follow the steps and generate a key

Generate Key

Generate a new record

Generate key

Add the DKIM key to your DNS record

Add DNS record

DNS will take a while to replicate so do wait a few hours before checking again with the checkmx tool.

Domain-based Message Authentication, Reporting, and Conformance (DMARC)

Read more on DMARC at Wikipedia here. Read the official page here https://dmarc.org/.

Background: The DMARC RFC form the Internet Engineering Task Force (IETF) states…

DMARC Flow

DMARC Flow

“Domain-based Message Authentication, Reporting, and Conformance (DMARC) is a scalable mechanism by which a mail-originating organization can express domain-level policies and preferences for message validation, disposition, and reporting, that a mail-receiving organization can use to improve mail handling.

Originators of Internet Mail need to be able to associate reliable and authenticated domain identifiers with messages, communicate policies about messages that use those identifiers, and report about mail using those identifiers. These abilities have several benefits: Receivers can provide feedback to Domain Owners about the use of their domains; this feedback can provide valuable insight about the management of internal operations and the presence of external domain name abuse.

DMARC does not produce or encourage elevated delivery privilege of authenticated email. DMARC is a mechanism for policy distribution that enables increasingly strict handling of messages that fail authentication checks, ranging from no action, through altered delivery, up to message rejection.”

Google G Suite has a guide to setting up a DMARC record here

Snip from the Google guide here..

“Spammers can sometimes forge the “From” address on email messages so the spam appears to come from a user in your domain. To help prevent this sort of abuse, Google is participating in DMARC.org, which gives domain owners more control over what Gmail does with spam email messages from their domain.

G Suite follows the DMARC.org standard and allows you to decide how Gmail treats unauthenticated emails coming from your domain. Domain owners can publish a policy telling Gmail and other participating email providers how to handle unauthenticated messages sent from their domain. By defining a policy, you can help combat phishing to protect users and your reputation.“

Login to your G Suite account and load this FAQ

Click Add A DMARC Record

Add DMARC

You will then need to set up a DKIM Domain Key (if you have not done so yet)

When you are done you need to choose your DMARC rules, I would suggest you go to https://mxtoolbox.com/DMARCRecordGenerator.aspx to generate a record

I generated these rules

Dmarc Rules

Warning: Setting a DMARC policy that is too strict may block mail from being delivered. Tighten rules over time.

Login to your DNS provider and add your TXT record.

DMARC Record

You should now have an SPF, DKIM and DMARC record in DNS.

DNS

Update: The SPD record above should be a TXT (Google led me astray)

DNS will take a while to replicate so do wait a few hours before checking again with the checkmx tool.

Now go to bed and wait for DNS to replicate.

Troubleshooting SPF

My TXT record would not validate with https://toolbox.googleapps.com/apps/checkmx/check

Google Toolbox

The MX Toolbox SPF checker reports that SPF records are deprecated and to use TXT records instead.

SPF TXT Record

Fix (remove the SPF record and add a TXT record with the same contents). Don’t forget to delete the old SPF record.

TXT Record

Results

SPF Setup

Reports

SPF/DKIM reports will let me know when unauthorized people send email from my domain.

This is a spf/dkim authentication-failure report for an email message received from IP 125.105.176.155 on Sat, 14 Apr 2018 13:14:09 +0800.
Below is some detail information about this message:
 1. SPF-authenticated Identifiers: none;
 2. DKIM-authenticated Identifiers: none;
 3. DMARC Mechanism Check Result: Identifier non-aligned, DMARC mechanism check failures;

For more information please check Aggregate Reports or mail to [email protected]



Feedback-Type: auth-failure
User-Agent: NtesDmarcReporter/1.0
Version: 1
Original-Mail-From: <[email protected]>
Arrival-Date: Sat, 14 Apr 2018 13:14:09 +0800
Source-IP: 125.105.176.155
Reported-Domain: fearby.com
Original-Envelope-Id: VcCowECJ7EIejtFanCHFLg--.51187S2
Authentication-Results: 163.com; dkim=none; spf=softfail [email protected]
Delivery-Result: delivered
Identity-Alignment: none



Received: from mitai (unknown [208.136.26.72])
	by fearby.com with SMTP id LyDKBHx6xsr7XZkf.1
	for <[email protected]>; Sat, 14 Apr 2018 13:14:03 +0800
Message-ID: <[email protected]>
From: =?utf-8?B?5rip5a6D?= <[email protected]>
To: <[email protected]>
Subject: =?utf-8?B?UmXvvJrlm57lpI3vvJrovazlj5HvvJrml7bpl7Q05pyIMjAtMjHml6XkuIo=?=
	=?utf-8?B?5rW3IOWcsOeCuSDnvo7oh6PljJblpoblk4Hlhazlj7jln7norq3ln7rlnLA=?=
Date: Sat, 14 Apr 2018 13:13:56 +0800
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_NextPart_000_08FE_016CD6FE.1A359D20"
X-mailer: Bagf 2

Also, DMARC will alert me to unauthorized activity

<?xml version="1.0" encoding="UTF-8" ?>
<feedback>
  <report_metadata>
    <org_name>google.com</org_name>
    <email>[email protected]</email>
    <extra_contact_info>https://support.google.com/a/answer/2466580</extra_contact_info>
    <report_id>4329490063964523747</report_id>
    <date_range>
      <begin>1523750400</begin>
      <end>1523836799</end>
    </date_range>
  </report_metadata>
  <policy_published>
    <domain>fearby.com</domain>
    <adkim>r</adkim>
    <aspf>r</aspf>
    <p>quarantine</p>
    <sp>none</sp>
    <pct>5</pct>
  </policy_published>
  <record>
    <row>
      <source_ip>2001:19f0:5801:5fa:5400:ff:fe80:ec7a</source_ip>
      <count>2</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>fail</dkim>
        <spf>fail</spf>
        <reason>
          <type>sampled_out</type>
          <comment></comment>
        </reason>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>fearby.com</header_from>
    </identifiers>
    <auth_results>
      <spf>
        <domain>unknown</domain>
        <result>none</result>
      </spf>
    </auth_results>
  </record>
</feedback>

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.4 Reports

v1.3 DMARC Flow image

V1.2 Updated wording

V1.1 Fixed typos (they were free)

v1.0 Initial post

Filed Under: Cloudflare, DKIM, DMARC, DNS, Domain, Email, GSuite, Security, SPF Tagged With: and, by, Cloudflare, DKIM, DMARC, email, G Suite, google, securing, Setting, SPF, up, with

Backing up OSX or an Ubuntu server with Backblaze B2 Cloud Storage from the Command Line

March 14, 2018 by Simon

This computer will show you can back up computer or server with Backblaze B2 Cloud Storage from the Command Line n OSX and Ubuntu.

This post is still being written. I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Also, I have blogged about how you can add block storage to a Vultr server, backup and restore snapshots , syncing files with rsync along with using GitHub and Bitbucket but what do you do if you need to backup large amounts of data?

Backblaze has a Cloud storage solution that costs as low as $0.005c a GB (a month), The first 10G is free. Backblaze say “From bytes to petabytes Backblaze B2 is the lowest cost high-performance cloud storage in the world. ”

Back Blaze have open sourced internal drive enclosure designs and drive failure stats and it’s time I gave them a try.

Goto https://www.backblaze.com

Backblaze

Create or sign in.

Backblaze Login

After you login got the dashboard.

B2 Cloud

Click Backblaze B2 Cloud Storage

Activate B2 Cloud

Signup

Create a Bucket

Create Bucket

Name the bucket (long names with a GUID are good).

Name the Bucket

You can rename the bucket here and change public/private and or upload/download files manually.

Manage Bucket

The first thing I did was limit the versions of files under the lifecycle settings for the bucket.

Version Settings

Now I created a series of subfolders to store files from different servers (I could have used many buckets but one bucket will do).

Folders

I can upload files via the Backblaze bucket GUI if I needed to.

Upload and Download

Back Blaze has a command line tool for uploading: https://www.backblaze.com/b2/docs/quick_command_line.html

Install Steps

Backblaze state “The B2 command-line tool is available from the Python Package Index (PyPI) using the standard pip installation tool. Your first step is to make sure that you have either Python 2 (2.6 or later) or Python 3 (3.2 or later) installed.”

I have Python 2.7 installed

python --version
Python 2.7.10

Install PIP

sudo easy_install pip
Password:
Searching for pip
Best match: pip 1.5.6
Processing pip-1.5.6-py2.7.egg
pip 1.5.6 is already the active version in easy-install.pth
Installing pip script to /usr/local/bin
Installing pip2.7 script to /usr/local/bin
Installing pip2 script to /usr/local/bin

Using /Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

I ran into issues updating b2 CLI

sudo pip install --upgrade b2
Requirement already up-to-date: b2 in /Library/Python/2.7/site-packages
Requirement already up-to-date: arrow>=0.8.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: logfury>=0.1.2 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: requests>=2.9.1 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: six>=1.10 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: tqdm>=4.5.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: futures>=3.0.5 in /Library/Python/2.7/site-packages (from b2)
Downloading/unpacking python-dateutil from https://pypi.python.org/packages/bc/c5/3449988d33baca4e9619f49a14e28026399b0a8c32817e28b503923a04ab/python_dateutil-2.7.0-py2.py3-none-any.whl#md5=5a86a548fe776cc079bf4a835473e3f8 (from arrow>=0.8.0->b2)
  Downloading python_dateutil-2.7.0-py2.py3-none-any.whl (207kB): 207kB downloaded
Installing collected packages: python-dateutil
  Found existing installation: python-dateutil 1.5
    Uninstalling python-dateutil:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

I tried installing via the alternative method (with no luck)

git clone https://github.com/Backblaze/B2_Command_Line_Tool.git
Cloning into 'B2_Command_Line_Tool'...
remote: Counting objects: 5084, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 5084 (delta 1), reused 1 (delta 0), pack-reused 5076
Receiving objects: 100% (5084/5084), 1.25 MiB | 689.00 KiB/s, done.
Resolving deltas: 100% (3622/3622), done.
cd B2_Command_Line_Tool/

I tried running the setup script (with no luck)

sudo python setup.py install
setuptools 20.2 or later is required. To fix, try running: pip install "setuptools>=20.2"

Upgrading setup tools also failed

sudo pip install "setuptools>=20.2"
Downloading/unpacking setuptools>=20.2
  Downloading setuptools-38.5.2-py2.py3-none-any.whl (490kB): 490kB downloaded
Installing collected packages: setuptools
  Found existing installation: setuptools 18.5
    Uninstalling setuptools:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

Backing up a Mac via command line with B2

More to come when I can get B2 CLI Installed.

Backing up an Ubuntu machine via command line with B2

More to come when I can get B2 CLI Installed.

Update

My ticket with Backblaze was automatically closed with this note “If the issue is persisting, it may be easiest to map the installation to the user folder, rather than the system level.”

No ideas how but something to research.

Ask a question or the recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

V1.1 ticket closed

v1.0 Initial post

Filed Under: Backup Tagged With: an, B2, backblaze, Backing, cloud, command, from, line, or, OSX, server, storage, the, ubuntu, up, with

PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API

March 1, 2018 by Simon

Developed by Simon Fearby https://www.fearby.com to allow PHP developers to integrate haveibeenpwned exposed password checks into their websites sign up’s (or logins). Get the latest version of this code from https://github.com/SimonFearby/phphaveibeenpwned/.

Update 2018: For the best performing VM host (UpCloud) read my guide on the awesome UpCloud VM hosts (get $25 free credit by signing up here).

This demonstrates a PHP framework less way (using HTML 5, Javascript and PHP) to validate a password by hashing (SHA1) the password (before the HTML form is submitted). A part of the password hash is checked at https://api.pwnedpasswords.com/range/{[}xxxxx} API (before a decision to save the form data is made). A Password exposed result returns the user to the sign-up form and no match completes the submission process.

SHA is performed on the password entered in the HTML form in Javascript (Your password never leaves the browser) and the PHP submit receiver performs a partial hash check at api.pwnedpasswords.com. Only a fraction of your password hash is sent to api.pwnedpasswords.com and only a partial hash of your password is returned with other partial matches (Making it hard (for anyone listening) to know what password you used).

This demo does not enforce SSL, sanitize, validate any form data or save the password to a database etc. The aim of this page is to demonstrate integration with api.pwnedpasswords.com. This demo displays a password strength meter. signup_submit.php allows you to enable debugging to see what is going on (detected errors are sent back to the submit.php and alerts shown.

Basics

signup.php – Main PHP file with a form with basic Javascript validation.

signup_submit.php – The form submit sends the form data here and calls the pwnedpasswords API

signup_ok.php – Is loaded if the password is not exposed

The initial HTML code was generated with the Platforma GUI web generator.  I added to the Javascript and relevant code. the HTML input field types are set to “text” (not “password”) so you can see the passwords.  A password SHA1 hash is generated on form submission and the user’s password never leaves the browser.

haveibeenpwned-001

A SHA1 hash of the password is updated and displayed (I am using the jsSHA library).

<script type="text/javascript" src="./js/sha/sha1.js"></script>

haveibeenpwned-002

After a basic HTML Javascript form validation is performed the uses password is replaced with a hash then the form is submitted (to signup_submit.php).

// Generate SHA1 Hash
var shaObj = new jsSHA("SHA-1", "TEXT");
shaObj.update(document.forms["submitform"]["password1"].value);
var passwordhash = shaObj.getHash("HEX");
document.getElementById("sha135").value = passwordhash;

signup_submit.php then takes the password hash, get the first 5 chars and fires up a curl connection to https://api.pwnedpasswords.com/range/$data when the data returns PHP checks the haveibeenpwned API body for matches of the matching password hashed and compared the known hash with the passwords has. Read more about how the API works here.

The PHP function that does the AI check is located here

function sendPostToPwnedPasswordsCom($data) {

    $curl = curl_init();		// Init Curl Object

    if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
    	echo "Data to Send: $data <br />";
    	echo "Sending Data to: https://api.pwnedpasswords.com/range/$data <br />";
    }

    // Set Curl Options: http://php.net/manual/en/function.curl-setopt.php
    curl_setopt($curl, CURLOPT_URL, "https://api.pwnedpasswords.com/range/$data");
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true); 
	curl_setopt($curl, CURLOPT_FRESH_CONNECT, true); 		// TRUE to force the use of a new connection instead of a cached one.
	curl_setopt($curl, CURLOPT_FORBID_REUSE, true); 		// TRUE to force the connection to explicitly close when it has finished processing, and not be pooled for reuse.
	curl_setopt($curl, CURLOPT_TIMEOUT, 10); 
    curl_setopt($curl, CURLOPT_MAXREDIRS, 10); 
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);		// TRUE to follow any "Location: " header that the server sends as part of the HTTP header (note this is recursive, 
    														// PHP will follow as many "Location: " headers that it is sent, unless CURLOPT_MAXREDIRS is set).

	// Make a request to the api.pwnedpasswords.com
    $http_request_result = curl_exec ($curl);
    $http_return_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);

	// api.pwnedpasswords.com Response codes
	/*
	Semantic HTTP response code are used to indicate the result of the search:

	Code	Description
	200	Ok — everything worked and there's a string array of pwned sites for the account
	400	Bad request — the account does not comply with an acceptable format (i.e. it's an empty string)
	403	Forbidden — no user agent has been specified in the request
	404	Not found — the account could not be found and has therefore not been pwned
	429	Too many requests — the rate limit has been exceeded
	*/


	// Change the return code to debug
	//$http_return_code = 429;
	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
      	echo "Return HTTP CODE: $http_return_code <br />";
    }
	
    // What was the http response code from api.pwnedpasswords.com
	if ($http_return_code == 200) {
		// OK (All other return codes direct the user back with an error)
		if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
	    	echo "Return HTTP Data site: " . strlen($http_request_result) . " bytes. <br />";
	    }

	} elseif ($http_return_code == 400) {
		// api.pwnedpasswords.com: API Bad Request
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Bad Request <br />";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIBadRequest&code=" . $http_return_code);
		die();

	} elseif ($http_return_code == 403) {
    	// api.pwnedpasswords.com: API Bad User Agent
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Bad User Agent <br />";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIBadUserAgent&code=" . $http_return_code);
		die();

	} elseif ($http_return_code == 404) {
		// api.pwnedpasswords.com: API User Not Found, not needed in his password hash check but we may as well catch now
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API User Not Found <br />";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIuserNotFound&code=" . $http_return_code);
		die();

    } elseif ($http_return_code == 429) {
    	// api.pwnedpasswords.com: API Too Many Requests
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Too Many Requests</ br>";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIuserTooManyRequests&code=" . $http_return_code);
		die();

    } else {
    	// api.pwnedpasswords.com: API Down

		if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Down!</ br>";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIDown&code=" . $http_return_code);
		die();
    } 

    // Tidy up the curl object and return the request api body
    curl_close ($curl); 
	return $http_request_result; 
}

You can enable debugging in signup_submit.php if you wish (but if an echo has presented debug data and a header redirect happens it will produce an error).

// define('ENABLE_DEBUG_OUTPUT', true);
define('ENABLE_DEBUG_OUTPUT', false);

echo statements are wrapped

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
    echo "API Bad Request <br />";
}

signup_submit.php will redirect the browser back to signup.php if an error is found

header("Location: signup.php?Error=PwnedpasswordsAPIuserTooManyRequests&code=" . $http_return_code);
die();

A success event (no password hash found) the user is sent to ignup_ok.php

header("Location: signup_ok.php");
die();

signup.php will check for the Error query string

Then display bootstrap alert errors for each error case

if ($error == "PwnedpasswordsAPIuserTooManyRequests") {
    echo '<div class="alert alert-danger" role="alert">';
    echo '<br /><a target="_blank" href="https://api.pwnedpasswords.com">https://api.pwnedpasswords.com</a> reported too many requests to the API from this IP, please wait a few seconds and try again.<br /> (E009)<br />';
    echo '<br />Please consider donating to Troy Hunt <a target="_blank" href="https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/">https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/</a> (<em>developer of <a target="_blank" href="https://haveibeenpwned.com">https://haveibeenpwned.com</a></em>).<br />';
    echo '</div>'; 
}
if ($error == "PwnedpasswordsAPIuserTooManyRequests") {
    echo '<div class="alert alert-danger" role="alert">';
    echo '<br /><a target="_blank" href="https://api.pwnedpasswords.com">https://api.pwnedpasswords.com</a> reported too many requests to the API from this IP, please wait a few seconds and try again.<br /> (E009)<br />';
    echo '<br />Please consider donating to Troy Hunt <a target="_blank" href="https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/">https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/</a> (<em>developer of <a target="_blank" href="https://haveibeenpwned.com">https://haveibeenpwned.com</a></em>).<br />';
    echo '</div>'; 
}

Sample Errors

Sample Password exposed to error.

haveibeenpwned-003

Sample API Offline alert

haveibeenpwned-005

If form field needs attention a JavaScript event is written to set the focus etc.

if ($error == "PasswordExposed") {
    echo '<script>';
    echo 'document.getElementById("password1").focus();';
    echo 'document.getElementById("password1").select();';
    echo '</script>';
}

If no password hash has been matched with pwnedpasswords the user is directed to signup_ok.php (not very exciting but that’s your jobs to integrate it with your system and harden).

haveibeenpwned-006

Sample debugging output

haveibeenpwned-007

Get the code: https://github.com/SimonFearby/phphaveibeenpwned/

More Reading

If you are using Ubuntu don’t forget to set up a free SSL cert, setting up an SSL cert on OSX is also a good idea. I have guides on setting up an Ubuntu server on AWS, Digital Ocean and Vultr. I love Vultr VM hosts and have blogged about setting up WordPress via the CLI, uploading files with SSH, restoring Vultr Snapshots etc.

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.0 Initial post

Filed Under: Pwned Tagged With: a, api, check, exposure, Hunts, implementation, level, password, php, pwnedpasswords, to, Troy, with

Scanning WordPress with Gravity scan for free to detect Supply Chain Attacks and WordPress malware

January 5, 2018 by Simon

A recent trend with some WordPress Plugins (and Google Chrome Extensions) is malicious parties will purchase existing plugins (extensions) and inject malicious code into new versions to infect sites and software, this is called “Supply Chain Attacks”. This is a personal unpaid review of Gravity Scan.

Update Feb 2018: Gravity Scan is shutting down 🙁

Recently WordFence wrote a blog post about Supply Chain Attacks found cases where older plugins are being purchased by malicious people in order to infect WordPress sites. WordPress CMS apparently runs 29% of the websites on the internet. Wordfence is a firewall and Gravity Scan is a vulnerability scanner, they complement each other.

I have blogged here about setting up WordPress via Command line and setting up an Ubuntu server for as low as 42.5 a month on Vultr.

What can you do to protect your WordPress sites from “Supply Chain Attacks”? First, install the WordFence plugin (I blogged about it here). Wordfence gives you a great set of security settings and reports to keep your site safe. The Wordfence dashboard page on your site is a good place to stay up to date.

WordFence is a Firewall, Gravity scan is a vulnerability and malware scanner. Read more here.

wordfence dashboard

Gravity Scan

Gravity scan is also made by the WordFence people to enable external audits and reports.

Gravity Scan Website

Sign up at https://www.gravityscan.com/ Verify your email and log in.

At login, you will be prompted to add a domain.

Add A Site

Scan

tip: You may want to whitelist Gravity scan servers. Read my guide about securing Ubuntu in the cloud.

sudo ufw allow from 68.64.48.0/27 to any port 443

A site scan will automatically be started.

Scan Started

Scan Results

Scan Results

Post Scan Actions

Speed Up future scans by downloading the Install Gravity Scan Accelerator (by clicking “Not instaled” under “Accelerator” in scan results) and follow the instructions to download, upload and verify the accelerator.

Install Gravityscan Accelerator

Read the Gravity Scan Accelerator Install Instructions here.

tip: I had to run the following command to make the

sudo chown www-data.www-data /www/gravityscan-agent-#############################################.php

Accelerator Installed

I also clicked “Trust Badge” link and added the script code to my site and verified it.

Trust Badle

I now have a scan badge in my site footer.

Future Scans

Future scans are all good to go.

Ready

New Scan Options

New Scan Options

It looks as if the accelerator gives more server-side verifications of checks of WordPress and PHP versions etc.

Go Pro

Gravity Scan also offers a non-free (paid) version where you can enable more options, enable scan schedules and set up SMS alerts and more for $4.95 a month per site.

Go Pro

To be honest I am happy with performing manual scans and I’d rather pay for a premium WordFence subscription first.

The Catch

Hang on Gravity scan requires a Pro membership to see High and Critical issues 🙁

Critical Issues

I decided not to go pro to reveal issues.

A few months later

I started receiving scan results with severity Critical (but I can’t see results until I start a trail (and enter payment details)).

Issues

Time to start a trial

I started a trial and full details were shown, the critical error was my fault

Gone Pro

Critical Issue

This was my fault, I left a previous version of WordPress in a subfolder from when I moved the site to a self-managed server. A quick few Linux commands later (removed) and this was fixed.

old files visible

High Issue

Publically accessible file (fixed with a chmod command)

File Visible

Current Scan Results

Current

Remote Scan Options

Daily Scans, Alter levels, Malware, Vulnerability and status checks. Definitely, install the Accelerator as it found my local backup of WordPress.

Manage Remote

Pros

  • Found a publically readable file
  • Found a past copy of my WordPress site (and all known issues with the old WordPress backup).
  • Can setup daily remote scans.

Cons

  • You have to go pro.
  • Can’t read my NGINX version (“Nginx version not detected, Gravityscan is unable to detect any associated vulnerabilities.“). I logged a ticket. Surely they can add a shell to”nginx -v” to the scan accelerator.
  • No word fence discount bundle?
  • Gravity scan and Word fence on twitter are slow to respond.

More to come.

More Reading

  • Run and Ubuntu Security scan with Lynis
  • WordFence security plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Setting up additional server storage on cloud servers (block storage on Vultr)

Hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

V.1.5 Gravity Scan shutting down

v1.4 Added remote scan options

v1.3 Pros and Cons and current results.

v1.2 Added more

v1.1 Fixed a few issues

v1.0 Initial Version

Filed Under: WP Security Tagged With: and, Attacks, Chain, detect, for, free, Gravity, malware, scan, Scanning, Supply, to, with, wordpress

How to upload and download files to a server with sftp (over the SSH protocol)

December 7, 2017 by Simon

I have blogged about http://c9.io before and how it makes managing a remote Ubuntu server easier. Recently AWS acquired C9 and integrated it into AWS. This has triggered me to find a more open/free way to connect to my servers. I like AWS but I can tell that C9 will someday block you from talking to non-AWS servers.

C9 Aquisition

C9 on AWS: https://aws.amazon.com/cloud9/?origin=c9io

C9 Aquisition: https://c9.io/announcement

I have blogged about connecting to an AWS EC2 instance with C9 (before AWS acquired C9), The quickest way to set up a scalable development IDE and web server when I was using Digital Ocean Servers, How to buy a new domain (dedicated server from digital ocean) and add an SSL certificate from NameCheap but when I moved to Vultr serves I decided to ditch C9 and save $9 a month.  I got used to setting up an SSH connection, using rmate to edit remote files locally with Sublime and SSH’ing into the box with vSSH on OSX but nothing replaced the C9 file management experience.

C9 was a good IDE (but recent price hikes and AWS purchase have made me cautious).

C9

Uploading files without C9 to Ubuntu

Uploading files from an OSX laptop to a remote Ubuntu server is possible with the SCP command

scp ~/Desktop/FilesToUpload.zip [email protected] /www/FilesToUpload.zip
< # Enter the remote server password

Server Actions (to upload the old way, with scp)

#Install Unzip (if you do not have it)
sudo apt-get install unzip

# Unzip the file
cd /www/
unzip /www/FilesToUpload.zip

There has to be a better way

I have used the Forklift 3 program before (my review here) and in recent google searches, it was suggested that Forklift best SFTP program on OSX. Yay. This unix.stackexchange.com thread mentions the difference between SFTP and SSH.

Below are the steps to use SFTP in Foftlift 3 on OSX to connect to a Vultr server (you could use Digital Ocean). Ensure you have a working SSH connection setup from your server to your local OSX.

Setting up an SFTP Connection in Forklift 3

Read my Forklift 3 guide here first.

Update your Forklift to the latest version.

Add a Favorite in Forklift 3

Add Favorite

Name the connection

Name the connection

In Forklift edit the connection

Edit

Specify SFTP and your servers working SSH username and password, and “/” folders

Settings

Save.

Double Click the entry to connect to your server.

Connected

To upload or download files with Forklift over SFTP, simply drag and drop.

Upload

Simple

Upload

You can now use Forklift to navigate folders, delete files, set remote file permissions all from a GUI. You can still use SSH CLI and or Sublime.

Quik Edit works too.

Quick Edit

Nice, this gives me a convenient GUI way to upload, download and edit files instead of via CSP/SSH/CLI.

Now is there an SFTP plugin for Sublime?

More to come soon.

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.0 Initial Draft

Filed Under: SFTP Tagged With: a, and, download, files, How, over the, protocol, server, sftp, ssh, to, upload, with

  • Go to page 1
  • Go to page 2
  • Go to Next Page »

Primary Sidebar

Poll

What would you like to see more posts about?
Results

Support this Blog

Create your own server today (support me by using these links

Create your own server on UpCloud here ($25 free credit).

Create your own server on Vultr here.

Create your own server on Digital Ocean here ($10 free credit).

Remember you can install the Runcloud server management dashboard here if you need DevOps help.

Advertisement:

Tags

2FA (9) Advice (17) Analytics (9) App (9) Apple (10) AWS (9) Backup (21) Business (8) CDN (8) Cloud (49) Cloudflare (8) Code (8) Development (26) Digital Ocean (13) DNS (11) Domain (27) Firewall (12) Git (7) Hosting (18) HTTPS (6) IoT (9) LetsEncrypt (7) Linux (20) Marketing (11) MySQL (24) NGINX (11) NodeJS (11) OS (10) PHP (13) Scalability (12) Scalable (14) Security (44) SEO (7) Server (26) Software (7) SSH (7) ssl (17) Tech Advice (9) Ubuntu (39) Uncategorized (23) UpCloud (12) VM (44) Vultr (24) Website (14) Wordpress (25)

Disclaimer

Terms And Conditions Of Use All content provided on this "www.fearby.com" blog is for informational purposes only. Views are his own and not his employers. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. Never make changes to a live site without backing it up first.

Advertisement:

Footer

Popular

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Add Google AdWords to your WordPress blog

Security

  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • Setting up DNSSEC on a Namecheap domain hosted on UpCloud using CloudFlare
  • Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx
  • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
  • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
  • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
  • Beyond SSL with Content Security Policy, Public Key Pinning etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Run an Ubuntu VM system audit with Lynis
  • Securing Ubuntu in the cloud
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Code

  • How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains
  • Useful Java FX Code I use in a project using IntelliJ IDEA and jdk1.8.0_161.jdk
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider
  • How to setup PHP FPM on demand child workers in PHP 7.x to increase website traffic
  • Installing Android Studio 3 and creating your first Kotlin Android App
  • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
  • How to use Sublime Text editor locally to edit code files on a remote server via SSH
  • Creating your first Java FX app and using the Gluon Scene Builder in the IntelliJ IDEA IDE
  • Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

Tech

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Is OSX Mojave on a 2014 MacBook Pro slower or faster than High Sierra
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • The case of the overheating Mac Book Pro and Occam’s Razor
  • Useful Linux Terminal Commands
  • Useful OSX Terminal Commands
  • Useful Linux Terminal Commands
  • What is the difference between 2D, 3D, 360 Video, AR, AR2D, AR3D, MR, VR and HR?
  • Application scalability on a budget (my journey)
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

Wordpress

  • Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution
  • Setting web push notifications in WordPress with OneSignal
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Wordfence Security Plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
  • Moving WordPress to a new self managed server away from CPanel
  • Moving WordPress to a new self managed server away from CPanel

General

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Using the WinSCP Client on Windows to transfer files to and from a Linux server over SFTP
  • Connecting to a server via SSH with Putty
  • Setting web push notifications in WordPress with OneSignal
  • Infographic: So you have an idea for an app
  • Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl
  • Building faster web apps with google tools and exceed user expectations
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in

Some ads on this site use cookies. You can opt-out if of local analytics tracking by scrolling to the bottom of the front page or any article and clicking "You are not opted out. Click here to opt out.". Accept Reject Read More
GDPR, Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT