and

DRAFT: Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software

October 4, 2018 by Simon

This post aims to show you how you can use a Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and other software and services.

Background

Although I am a developer I do like security related topics and I try and do as much as I can to secure my systems and applications. Reading the Multi-Factor Authentication Wikipedia page has all the details on Multi-Factor authentication.

I have been a big fan of 1Password to generate strong and unique passwords for separate accounts for a while now. Read my guide on upgrading from a standalone 1Password licence to a 1Password subscription. I love generating unique and complex passwords with 1Password.

But what happens if someone gets access to my 1password vault? Yubico has a catalogue of support services that I can use Yubikeys with to have, 1password is one supported service 🙂

I want to add Yubico protections with these services.

macOS Logins (DONE)
macOS Screensavers (DONE)
1Password (DONE)
Dropbox (DONE)
Twitter (DONE)
Google (DONE)
Google GSuite (DONE, WAITING TO VERIFY)
Google GMail (DONE)
Google Analytics and AdSense (DONE)
Github (DONE)
Thunderbird Email (DONE)
Bitbucket (TODO)
Debian servers in the cloud (SSH) (DONE)
Ubuntu servers in the cloud (SSH) (DONE)
Android mobile (NFC)? (TODO)

Etc

Final Warning

Do not attempt to activate Two Factor Authentication on a system unless you…

A) Have backups of your data
B) Have backup methods of getting into your account(s)

Murphy’s Law: “Anything that can go wrong will go wrong”

You never know when a Two Factor Authentication Key may die or an Authenticator app or a Mac/PC may stop working so always have a backup method just in case.

General

General Yubico YubiKey Setup guides https://www.yubico.com/setup/

Buying a Yubico YubiKey

International visitors can buy a YubiKey from the official store here. Australian readers can buy a key locally here. I grabbed 2x YubiKey YubiKey Neo 4 (with NFC) for $50 USD (about $75 AUD) each.

This blog post will aim to show how you can set up a primary key and backup key for use on macOS and other apps to add hardware-based two-factor authentication to logins.

Authenticator Apps

You can use Google Authenticator, Yubico Authenticator or freeOTP from https://freeotp.github.io

Plugging the YubiKey into macOS Mojave

First I read this guide: https://www.yubico.com/works-with-yubikey/catalog/macos/

1) I plugged in my Yubico Neo key into my USB slot.
2) I closed the Keyboard setup window that appeared (I guess the YubiKey is a kind of a keyboard to allow inserting of challenge-response character streams into apps and websites).

3) I followed the basic troubleshooting page and confirmed that the key was being detected (yes it was.)

4) I followed this guide to test U2F functionality and this guide to test OTP functionality. Web pages and Google Chome can talk to the plugged-in YubiKey(s).

I was prompted to register a UTF deice (and create an account)

I was prompted to (insert) and touch my Yubico key.

Google Chome asked for some permissions first.

FYI: Chrome 67 is recommended to securely allow the reading of UbiKey’s from web pages. Only allow sites you trust access to your USB devices and use a modern browser.

Success, Chrome could now see my YubiKey and my device was now verified.

Technical data is available to let you know what is going on in the background. I am not going to break down how this works but Yubico has in-depth whitepapers and documentation if you are interested.

Nice

Configuring OSX

I logged into my Mac with the account that I was going to secure.

I performed a complete time machine backup before proceeding. If you lock yourself out you will need to restore OSX from a Time Machine backup.

I Read the “Using Yubico Pluggable Authentication Module (PAM) with Challenge-Response” login guide: https://www.yubico.com/wp-content/uploads/2016/07/yubico_YubiKeyMacLoginGuide_en.pdf

I downloaded the Download the YubiKey Manager

I downloaded the yubikey-manager from here so I could configure the keys to use “HMAC-SHA1 Challenge-Response”.

Oops, I downloaded the wrong tool, good to know this one exists though.

I will update what this tool does in future (update firmware?)

I Downloaded the Yubikey Personalization Tool

I went back to the Yubico download page and downloaded the Personalization tool.

Many options are available here.

It’s time to configure a primary and backup (duplicate YubiKey) for use with macOS etc.

Enable Challenge Response

I opened the YubiKey Personalization Tool, Inserted my primary key, clicked the Settings tab, and in the Logging Settings group, selected Log configuration output and Yubico format.

I then clicked on the Challenge Response Tab, clicked the HMAC-SHA1 button, selected Configuration Slot 2, ticked “Program Multiple YubiKeys“, changed the “Parameter Generation Scheme = Same for all Keys“, Selected “Fixed 64 byte input” under “HMAC-SHA1 Parameters” and generated a new key (wrote it down).

Under “Configuration Protection” then I selected Enable Protection” I then visited here and generated a 6 digit string to convert to hex array (with spaces (e.g: “70 61 73 73 77 64”)).

Warning: If you set an access code and later forget it, you cannot make any programming changes to this YubiKey. You would need to buy another YubiKey.

I clicked on Write Configuration

If you chose Configuration Slot 1 you will receive a warning about not saving over Configuration Slot 1 due to Yubico VIP/Symantec, I personally do not trust Symantec or the https://vip.symantec.com/ service due to Symantec issuing non-compliant certificates for use on websites. Yubico allows you to swap configuration slots if want to keep the configuration data.

On the output of the first write, I was prompted to save a file. I saved this to “secretkey.csv” onto the Desktop.

When the write to my primary key was successful, I ejected it then inserted my backup key and wrote the same configuration data to it too (on Configuration Slot 2).

Testing the HMAX-SHA1 Challenge

I open the YubiKey Personalization Tool, then click the Tools tab and click Challenge Response. Choose Configuration Slot 2, I selected HMAC-SHA1. I typed a sample input challenge (e.g “hello world”) and clicked Perform.

I noticed the Yubico key touch panel was flashing. I pressed the button, then a response appeared below the input textbox. I copied this response text then insert your second key and perform the same test so I could compare the responses (they should be the same). They were.

If the responses don’t match rewrite the configuration to your primary and secondary keys and ensure the same key and secret was used for both keys.

FYI: I rewrote configuration a few times until I got it right.

Installing the Pluggable Authentication Module (PAM) on macOS

I re-read the Mac login guide here as I don’t want to lock myself out of my Mac.

I opened the Yubico Software Download page here and clicked Computer Login Tools and downloaded the PAM for Mac.

I installed the PAM package and verified the package installation with this command.

ls -al /usr/local/lib/security

1	ls -al /usr/local/lib/security

Output:

Text Output:

> drwxr-xr-x 3 root wheel 96 9 Oct 10:29 .

> drwxrwxr-x 74 simon admin 2368 9 Oct 10:29 ..

> -rwxr-xr-x 1 root wheel 143172 20 Apr 21:13 pam_yubico.so

Backup macOS

Again I ensured my Mac was backed up with Time Machine.

I logged in to my Mac with the account I wanted to be protected with the Yubico YubiKeys.

I ran the following command in terminal

mkdir –m0700 –p ~/.yubico

1	mkdir –m0700 –p ~/.yubico

I double checked that my Yubico key(s) were set up for challenge response (above).

I inserted my Uubico key and ran this command

ykpamcfg -2

1	ykpamcfg -2

Feel free to read the “ykpamcfg” manual here. The yubico-pam source code is located here.

Output:

The contents of “/Users/simon/.yubico/challenge-#######” looked like (I replaced 232 random chars with #’s below). The filename ended with my keys serial number.

v2:########################################################################################################################################################################################################################################:10000:2

Next, I was supposed to copy the challenge output from ykpamcfg to /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER] with this command..

sudo cp /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER] /Users/[USERNAME]/.yubico

1	sudo cp /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER] /Users/[USERNAME]/.yubico

But I had this error.

No such file or directory

1	No such file or directory

Weird as the source file existed?? macOS issues?

I Opened /Users/[USERNAME]/.yubico/challenge-[YUBIKEY SERIAL NUMBER] in the nano editor (sudo elevated process) and saved the file to /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER].

I reopened my terminal and verified the contents of /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER]. The file is now there.

Permissions on the file is “-rw——-“. Good.

I inserted my second backuP key and re-ran “ykpamcfg -2” and copied the file to “/Users/simon/.yubico”

I verified the file contents

sudo cd /var/root/.yubico/
ls -al

1 2	sudo cd /var/root/.yubico/ ls -al

Output

Text Output:

> drwxr-xr-x 4 root wheel 128 9 Oct 09:50 .
> drwxr-x— 12 root wheel 384 9 Oct 09:39 ..
> -rw-r–r– 1 root wheel 244 9 Oct 09:50 challenge-#######
> -rw-r–r– 1 root wheel 244 9 Oct 09:42 challenge-#######

Snip from: https://www.yubico.com/wp-content/uploads/2016/07/yubico_YubiKeyMacLoginGuide_en.pdf

“Program at least two YubiKeys when implementing a requirement for authentication with a YubiKey on your Mac. If you configure only one YubiKey and something happens to the YubiKey, you must restore the Mac from a Time Machine backup that you created before editing the authorization file before you can log back in to your account. ”

Reading the guide regarding multiple accounts (setting up a Key for each login). I have 5 logins on my Mac but when this works I will disable the other accounts from logging in.

Enable the use of the Yubico key when the screensaver is deactivated

I opened a terminal and edited “/etc/pam.d/screensaver ” (I use the easier nano editor)

sudo nano /etc/pam.d/screensaver

1	sudo nano /etc/pam.d/screensaver

I added this line

auth       required       /usr/local/lib/security/pam_yubico.so mode=challenge-response

1	auth required /usr/local/lib/security/pam_yubico.so mode=challenge-response

auth[7 spaces]required[7 spaces]/usr/local/lib/security/pam_yubico.so mode=challenge-response

I saved the file ( [CTRL+O], [CTRL+X] ) and exited nano.

I tested my screensaver and no extra protection was provided (the screensaver just exited).

I rebooted, still no change?

I reinstalled the PAM module.

Silly me, I needed to enable the password on the screensaver to then activate the /etc/pam.d/screensaver entries.

I enabled the screensaver passwords

I am now prompted to enter my password and inset and tap my Yubico Key on screensaver exit (on both keys). Awesome.

Next, I need to enable this at macOS login.

Enable the use of the Yubico key at macOS Login

I edited /etc/pam.d/authorization file with nano in the terminal

sudo nano /etc/pam.d/authorization

1	sudo nano /etc/pam.d/authorization

I added the same line as was added to the file /etc/pam.d/screensaver

auth       required       /usr/local/lib/security/pam_yubico.so mode=challenge-response

1	auth required /usr/local/lib/security/pam_yubico.so mode=challenge-response

auth[7 spaces]required[7 spaces]/usr/local/lib/security/pam_yubico.so mode=challenge-response

I saved the file ( [CTRL+O], [CTRL+X] ) and exited nano.

Now let’s log out and test this.

It’s working.

Excellent

Enable login to the 4 other macOS Logins?
todo…

Add Two Factor Authentication to 1Password

Here is a guide on using the Yubico YubiKey with 1Password. This directed me to https://support.1password.com/yubikey/

I downloaded the Yubico Authenticator app on macOS and installed it.

After I inserted my primary Key I received a “No Credentials Found”message.

I logged into https://my.1password.com/signin and clicked My Profile.

I clicked More Actions then Turn On Two-Factor Authentication

I added the generated QR code details to the Android Authenticator and macOS Yubico Authenticator app. At first, I could not scan the QR code in macOS (was Mojave blocking this?), I manually entered the details (after confirming them from the Android app QR code scan).

Details:

Issuer: 1Password
Account Name: my.1password.com
Secret Key: ###################
Time: 30
Algorithm: SHA-1
Period: 30
Digits: 6

Now, 1Password web and the desktop app are asking for the 2-factor code (generated in the Yubico Authenticator app after I insert my YubioKey).

Nice

I logged off and I was not prompted for my Two Factor code?

Snip from: https://support.1password.com/two-factor-authentication/

“Your 1Password account is now protected by two-factor authentication. From now on, you’ll need to enter a six-digit authentication code from your authenticator app when you sign in to 1Password on a new device.”

I logged in to 1Password from Google Chrome on Android and indeed I was prompted for a two-factor auth code form the Yubico Authenticator app (with a KubiKey inserted).

Add Two Factor Authentication to Dropbox
I read https://www.yubico.com/works-with-yubikey/catalog/dropbox-personal/. Dropbox also has setup instructions here.

I logged into Dropbox and went to Settings then Security then clicked Add next to Security Keys

I started the Wizard, entered my Dropbox password, then inserted my YubiKey.

Name the Key

I added my Primary and Backup Key(s)

I logged out and back in and no Security Key prompt?

I am using Chrome and had cleared past browsers from the Dropbox list of web browsers at https://www.dropbox.com/account/security

I discovered that I need to set the primary authentication method to Use Mobile App (My Bad, it would be nice if Dropbox set this as default after I added the keys).

I added the Dropbox QR code to the Yuboico Authenticator app

I was asked to enter a 6 digit code from my Yubico Authenticator app to verify the working link. I inserted my YubiKey into my machine to show the code.

Now Dropbox is configured 🙂

Success

I now have to insert my primary key when logging into Dropbox

I need to find a way to copy my Authenticator credentials to my Backup Key from my Primary key

Add Two Factor Authentication to Twitter

I read https://www.yubico.com/works-with-yubikey/catalog/twitter/ (Setup Instructions)

1) Login to Twitter

2) Open your Settings and Look For Security

3) Click Start

4) Enter Your Password

5) Accept and enter any SMS codes if you set up SMS Two Factor codes via SMS

6) Click “Review your login verification methods”

7) Click “Setup Key”

8) Insert Your YubiKey and follow the prompts to activate it.

9) Now the key will be requoted to log in to Twitter

Testing Two Factor Login to Twitter

I logged out of and back into Twitter but the SMS Two Factor Authentication method was still active?

I tried to disable the SMS method in Twitter but two factor was disabled altogether and the registered key was deleted. I re-added my key 🙁

I solved this by choosing “Choose a different verification method” when logging in then choosing “Use your security key“, Twitter then accessed my YubiKey and further login attempts used the key instead of SMS 🙂 I could use an Authenticator code but they YubiKey touch method is quicker.

Done

It would be nice if Twitter allowed multiple keys to be used to log in?

Add Two Factor Authentication to Google, Google cloud, Gsuite etc

I read https://www.yubico.com/works-with-yubikey/catalog/google-accounts (Instructions https://myaccount.google.com/).

Adding two Factor authentication details to Google was not easily accessible at Google so I Googled (lol) this https://support.yubico.com/support/solutions/articles/15000006418-using-your-yubikey-with-google

I loaded: https://myaccount.google.com/signinoptions/two-step-verification/enroll-welcome

I clicked Get Start

I clicked Choose Another Option (not SMS Two factor)

Clicked Security Key

As prompted I inserted my key and allowed access to it.

I named the Key

I repeated the steps and added my 2nd key.

Done

I logged out my https://myaccount.google.com and logged back in and I was prompted to insert my YubiKey

Nice

I did try and login to my google GSuite account at https://admin.google.com but it did not prompt me to insert a key. I will do this next.

Add Two Factor Authentication to GSuite

I logged into the GSuite admin interface at https://admin.google.com/ I generated some backup codes in case I need them in the future.

I checked my main admin user account and I could see the 2 google security keys synced through from Google.

I then searched GSuite for “Two Factor” and loaded the “Enforcement” Page

I enabled “Turn On Enforcement Now”

I enabled “Only Security Keys”

I logged out and back into https://gsuite.google.com/ TWICE and no security key prompt.

Silly me: I forgot to click save at the bottom of the screen and it appears there is a 24-hour delay?

Add Two Factor Authentication to GMail

This is already done (above), GSuite email takes up to 24 hours to become active, GMail is instant.

Add Two Factor Authentication to Google Analytics

I can’t see an option to turn Two Factor Auth on in Google Analytics 🙂

I did send feedback to the Google Analytics team.

Add Two Factor Authentication to Google Adsense

I can’t see an option to turn Two Factor Auth on in Google Adsense either 🙂

I did send feedback to the Google AdSense team.

Add Two Factor Authentication to Github

I logged into Github, opened my Settings and clicked Security then Enable two-factor authentication

Click Setup using an app save the recovery codes.

Open the Yubico Authenticator app (ensure you can see the QR Code in GitHub)

In the Yubico Authenticator, App click File then Scan QR Code

The GitHub details should be added to the Authenticator

Two Factor via authenticator tokens is enabled and now I can see a Keys options,

I clicked Add next to security keys then Register New Device, I gave the key a name then clicked Add.

I added both keys then I Logged out and back in and two factor was enabled by YubiKey 🙂

Add Two Factor Authentication to Bitbucket
todo…

Add Two Factor Authentication to Debian servers in the cloud (SSH)

Read Setup two-factor authenticator protection at login on Ubuntu or Debian

Add Two Factor Authentication to Ubuntu servers in the cloud (SSH)

Read Setup two-factor authenticator protection at login on Ubuntu or Debian

Add Two Factor Authentication to Android mobile (NFC)?
todo (unlock Android with NFC Tap)…

YubiKey Support

There are loads of Yubico support articles here: https://support.1password.com/yubikey/

Yubico Developer Info

A GitHub repository of source code is located here: https://github.com/Yubico

Other developer related pages here

Java Code to use the Yubico YubiKey in software (challenge mode)
todo: I will add this section soon.

Yubico has Java repository that contains a Java library with an accompanying demo server, as well as a JAAS module, to validate YubiKey OTPs (One-Time Passwords).

https://developers.yubico.com/yubico-java-client/

Recovery info and backup
Always where prompted setup backup access codes (or set alternate two-factor login methods) and know how you can disable YubiKey 2FA logins if needed.

Copy Yubico Authenticator credentials to my Backup Key from my Primary Key

My Primary and Secondary YubiKeys have different Authenticator credentials (I need to sync them)

todo: Add infomation on backing up YubiKey data

Set a YubKey Password (Yubico Authenticator App)

You can set a YubiKey Password so limit access to Two Factor Linked Accounts in the Yubico Authenticator. Nice.

- 1. Open the Yubico Authenticator App
  2. Insert your YubiKey
  3. Open the File then Set Password Menu
  4. Click Set Password

Now when you insert the YubiKey you will be prompted for a password Before Two Factor tokens are displayed.

Find a YubiKey Device Quiz

Use this quiz to find the right YubiKey for you: https://www.yubico.com/quiz/

Final Warning

Do not attempt to activate Two Factor Authentication on a system unless you…

A) Have backups of your data
B) Have backup methods of getting into your account(s)

Murphy’s Law: “Anything that can go wrong will go wrong”

You never know when a Two Factor Authentication Key may die or an Authenticator app or a Mac/PC may stop working so always have a backup method just in case.

Issue(s)

Thunderbird email on Google Chrome (accessing GSuite) is not acceptingg the key.

It is prompting…

But it is not recognising the key (no matter how many times I insert or press the key)?

It appears Thunderbird 52 may not support keys yet, May have to wait until release 60.

I installed Thunderbird 63 (BETA) from https://www.thunderbird.net/en-US/channel/

After I installed Thunderbird it asked for my Security Key, accepted it and asked for further permissions.

I can now read my email in Thunderbird with my YubiKey

Conclusion

Thunderbird issues (solved by installing a BETA).

Not all apps have the same menthid (some have Authenticator App only) and some have YubiKey Insert/Touch, some allow one key or multiple keys.

No issues with the YubiKey setup yet, very happy.

The only issue is my Huawei Mate 9 phone is a little flakey at reading NFC (fixed: I just have to tap for 5 seconds)

I have attached the YubiKeys to Dog Chain’s and they live around my neck.

Version History

V0.8.1 authenticator apps
V0.8.0 Draft: Debian/Ubuntu and many other changes
V0.7.0.1 Draft: Issue – Thunderbird Issue Solved
V0.7.0 Draft: Issue – Thunderbird Issue
V0.6.9 Draft: Protected GitHub
V0.6.9 Draft: Unable to Protect Google AdSense and Analytics
V0.6.8 Draft: Protected Google Gmail (https://gmail.com)
V0.6.7 Draft: Protected Google GSuite (https://gsuite.google.com/ and https://admin.google.com/)
V0.6.6 Draft: Protected Google (https://myaccount.google.com/)
V0.6.5 Draft: Protected Twitter
V0.6 Draft: Set a YubKey Password (Yubico Authenticator App)
V0.5 Draft: Sync Authenticator credentials?
v0.4 Draft: Protected Dropbox
v0.3 Draft: Protected 1Password
v0.2 Draft: Protected macOS Login
v0.1 Draft: Protected macOS Screensaver

Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc

October 3, 2018 by Simon

This is a draft post showing how you can monitor the performance of a server (or servers) with NixStats and receive alerts by SMS, Push, Email, Telegram etc

fyi: This is not a paid post, this is just me using the NixStats software to monitor my servers and send alerts.

Finding a good host

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Monitoring Servers

The post below will show you how you can monitor servers online with https://nixstats.com/ and send alerts when resources reach limits or servers fail.

I signed up and started a Nixstats (14 day free trial).

After I created an account I was emailed by Nixtsats with agent install instructions for Linux (1 line). I was also advised to add contacts and to set up alerts.

I logged into the Nixstats settings and set up…

My Timezone
Default reporting period
First name and Surname
Reporting emails
etc

Nixstats Subscription Upgrade

Subscription options

Free (5 monitors, 1 server, 24-hour data retention etc)
Founder (25 monitors, 10 servers, 30-day data retention etc)
Business (100 monitors, 15 servers, 60-day data retention etc)

I enabled the limited founder subscription so I can monitor 10x servers (this deal is too good to miss). I tried creating a status page myself last year and it is terribly hard.

I am now out of the free trial period 🙂 Let’s start monitoring many servers.

I enabled two factor Auth to Nixstats logins

I created a Nixstats API key for future use (watch this space)

I installed the Nixstats agent (the dashboard gave a 1 line command you can run as root to install the agent (on Linux)).

FYI: Command (######################## is a number linked to your account)

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################

1	wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################

Output

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################
--2018-10-02 09:53:56--  https://www.nixstats.com/nixstatsagent.sh
Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8013, 2400:cb00:2048:1::6819:8113, 104.25.128.19, ...
Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8013|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 38708 (38K) [application/octet-stream]
Saving to: 'nixstatsagent.sh'

nixstatsagent.sh               100%[====================================================>]  37.80K  --.-KB/s    in 0.1s

2018-10-02 09:53:56 (338 KB/s) - 'nixstatsagent.sh' saved [38708/38708]

Found Ubuntu ...
Installing ...
Installing Python2-PIP ...
Installing nixstatsagent ...
Generation a server id ...
Got server_id: ######################
Creating and starting service
Created symlink /etc/systemd/system/multi-user.target.wants/nixstatsagent.service -> /etc/systemd/system/nixstatsagent.service.
Created the nixstatsagent service

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################

--2018-10-02 09:53:56-- https://www.nixstats.com/nixstatsagent.sh

Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8013, 2400:cb00:2048:1::6819:8113, 104.25.128.19, ...

Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8013|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 38708 (38K) [application/octet-stream]

Saving to: 'nixstatsagent.sh'

nixstatsagent.sh 100%[====================================================>] 37.80K --.-KB/s in 0.1s

2018-10-02 09:53:56 (338 KB/s) - 'nixstatsagent.sh' saved [38708/38708]

Found Ubuntu ...

Installing ...

Installing Python2-PIP ...

Installing nixstatsagent ...

Generation a server id ...

Got server_id: ######################

Creating and starting service

Created symlink /etc/systemd/system/multi-user.target.wants/nixstatsagent.service -> /etc/systemd/system/nixstatsagent.service.

Created the nixstatsagent service

Server Dashboard

below is a summary of all connected servers ( https://nixstats.com/dashboard/servers ).

Monitor Setup

I set up a number of monitors to monitor ping replies and https traffic

Advanced Monitoring

I can also set the monitor credentials, timeouts, retries, auth methods, max redirects and frequency. If you server blocks login or resource GET attempts you may need to whitelist IP’s. IP’s of monitoring servers are located here https://nixstats.com/whitelist.php

Monitor Summary

The default dashboard is very informative. Feel free to create your own dashboards that focus on your own infastructure or apps.

Individual Server Reports

You can click on a server and monitor it in detail.

Server Memory Graphs

Long-term memory graphs.

Install Optional Nixstats Plugins

Nixstats offers many plugins to monitor software that is installed on your server (e.g NGINX, MySQL, PHP etc).

1) NGINX Monitoring (Plugin)

To enable NGINX monitoring I read https://help.nixstats.com/en/article/monitoring-nginx-50nu7f/

I edited my NGINX sites-enabled config.

sudo nano /etc/nginx/sites-enabled/default

1	sudo nano /etc/nginx/sites-enabled/default

I added the following

server {
    listen 127.0.0.1:8080;
    server_name localhost;
    location /status_page {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }
}

server {

listen 127.0.0.1:8080;

server_name localhost;

location /status_page {

stub_status on;

allow 127.0.0.1;

deny all;

}

I tested, reloaded and restarted NGINX

nginx -t
nginx -s reload
/etc/init.d/nginx restart

nginx -t

nginx -s reload

/etc/init.d/nginx restart

The status page will only be available on the local machine, I tested the page on the local machine

wget -qO- http://127.0.0.1:8080/status_nginx
Active connections: 3
server accepts handled requests
 15 15 31
Reading: 0 Writing: 1 Waiting: 2

wget -qO- http://127.0.0.1:8080/status_nginx

Active connections: 3

server accepts handled requests

15 15 31

Reading: 0 Writing: 1 Waiting: 2

It’s Working.

I edit /etc/nixstats.ini

sudo nano /etc/nixstats.ini

1	sudo nano /etc/nixstats.ini

I remove comments before these lines to enable the plugin

[nginx]
enabled = yes
status_page_url = http://127.0.0.1:8080/status_nginx

[nginx]

enabled = yes

status_page_url = http://127.0.0.1:8080/status_nginx

I ran the following command to see if NGINX monitoring is possible

nixstatsagent --test nginx

1	nixstatsagent --test nginx

Output

nginx:
{
    "accepts": 39,
    "accepts_per_second": 0.0,
    "active_connections": 6,
    "handled": 39,
    "handled_per_second": 0.0,
    "reading": 0,
    "requests": 119,
    "requests_per_second": 0.0,
    "waiting": 5,
    "writing": 1
}

nginx:

{

"accepts": 39,

"accepts_per_second": 0.0,

"active_connections": 6,

"handled": 39,

"handled_per_second": 0.0,

"reading": 0,

"requests": 119,

"requests_per_second": 0.0,

"waiting": 5,

"writing": 1

}

It’s Working

I restarted the nixstatsagent

service nixstatsagent restart

1	service nixstatsagent restart

I can now view NGINX properties like active_connections in my dashboard. 🙂

2) Enable PHP-FPM Monitoring (Plugin)

Looks like a PHP-FPM monitoring was recently added lets set that up too. Read my guide on setting up PHP child workers here.

We’ve added a premade dashboard for PHP-FPM. If you’re not yet monitoring PHP-FPM take a look at the integration guide https://t.co/X4ywRHw9hX pic.twitter.com/aag1fTsr3R
— Nixstats (@nixstats) September 6, 2018

To enable PHP-FPM monitoring I read https://help.nixstats.com/en/article/monitoring-php-fpm-1tlyur6/

I edited my PHP-FPM ini file

sudo nano /etc/php/7.2/fpm/pool.d/www.conf

1	sudo nano /etc/php/7.2/fpm/pool.d/www.conf

I added the following line

pm.status_path = /status_phpfpm

1	pm.status_path = /status_phpfpm

Restart PHP

sudo service php7.2-fpm restart

1	sudo service php7.2-fpm restart

I added the following to /etc/nginx/sites-enabled/default localhost server block added above Note I use php 7.2 below (read more here).

server {
    listen 127.0.0.1:8080;
    server_name localhost;

    location /status_phpfpm {
         access_log off;
         allow 127.0.0.1;
         deny all;
         include fastcgi_params;
         fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
         fastcgi_pass unix:/run/php/php7.2-fpm.sock;
         log_not_found off;
    }
}

server {

listen 127.0.0.1:8080;

server_name localhost;

location /status_phpfpm {

access_log off;

allow 127.0.0.1;

deny all;

include fastcgi_params;

fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

fastcgi_pass unix:/run/php/php7.2-fpm.sock;

log_not_found off;

}

I tested, reloaded and restarted NGINX

nginx -t
nginx -s reload
/etc/init.d/nginx restart

nginx -t

nginx -s reload

/etc/init.d/nginx restart

I restart PHP-FPM

sudo systemctl restart php7.2-fpm

1	sudo systemctl restart php7.2-fpm

Enabled the plugin in /etc/nixstats.ini

[phpfpm]
enabled = yes
status_page_url = http://127.0.0.1:8080/status_phpfpm?json

[phpfpm]

enabled = yes

status_page_url = http://127.0.0.1:8080/status_phpfpm?json

I tested the status page

wget -qO- http://127.0.0.1:8080/status_phpfpm?json

1	wget -qO- http://127.0.0.1:8080/status_phpfpm?json

Output:

{
	"pool":"www",
	"process manager":"static",
	"start time":1538654543,
	"start since":178,
	"accepted conn":28,
	"listen queue":0,
	"max listen queue":0,
	"listen queue len":0,
	"idle processes":49,
	"active processes":1,
	"total processes":50,
	"max active processes":2,
	"max children reached":0,
	"slow requests":0
}

{

"pool":"www",

"process manager":"static",

"start time":1538654543,

"start since":178,

"accepted conn":28,

"listen queue":0,

"max listen queue":0,

"listen queue len":0,

"idle processes":49,

"active processes":1,

"total processes":50,

"max active processes":2,

"max children reached":0,

"slow requests":0

}

I tested the agent

 nixstatsagent --test phpfpm

1	nixstatsagent --test phpfpm

Output:

phpfpm:
{
    "accepted_conn": 51,
    "accepted_conn_per_second": 0.0,
    "active_processes": 1,
    "idle_processes": 49,
    "listen_queue": 0,
    "listen_queue_len": 0,
    "max_active_processes": 2,
    "max_children_reached": 0,
    "max_listen_queue": 0,
    "pool": "www",
    "process_manager": "static",
    "slow_requests": 0,
    "start_since": 318,
    "start_time": 1538654543,
    "total_processes": 50
}

phpfpm:

{

"accepted_conn": 51,

"accepted_conn_per_second": 0.0,

"active_processes": 1,

"idle_processes": 49,

"listen_queue": 0,

"listen_queue_len": 0,

"max_active_processes": 2,

"max_children_reached": 0,

"max_listen_queue": 0,

"pool": "www",

"process_manager": "static",

"slow_requests": 0,

"start_since": 318,

"start_time": 1538654543,

"total_processes": 50

}

I can now query PHP-FPM status values in Nixstats 🙂

Enable MySQL Monitoring (Plugin)

To enable MySQL monitoring I read https://help.nixstats.com/en/article/monitoring-mysql-1frskd8/

I edited the nixstats.ini

sudo nano /etc/nixstats.ini

1	sudo nano /etc/nixstats.ini

I enabled the mysql section in nixstats.ini and added my mysql credentials

[mysql]
enabled=yes
username=mysqluser
password=#######################
host=127.0.0.1
database=mysql
port=3306
socket=null

[mysql]

enabled=yes

username=mysqluser

password=#######################

host=127.0.0.1

database=mysql

port=3306

socket=null

I ran this command to test MySQL querying

nixstatsagent --test mysql
mysql:
Load error: No module named MySQLdb

nixstatsagent --test mysql

mysql:

Load error: No module named MySQLdb

I had an error.

A quick Google revealed I had to install a mysql python module.

sudo apt-get install python-mysqldb

1	sudo apt-get install python-mysqldb

Allow localhost to connect to MySQL

Edit /etc/mysql.cnf and allow all localhost and external connections (I could not bind to localhost and an external IP at the same time)

bind-address    = 0.0.0.0

1	bind-address = 0.0.0.0

TIP: Ensure you have firewalled access to your MySQL server, never open it up without protection.

Let’s try again

nixstatsagent --test mysql

1	nixstatsagent --test mysql

Output

mysql:
{
    "aborted_clients": 0,
    "aborted_connects": 0,
    "binlog_cache_disk_use": 0,
    "binlog_cache_use": 0,
    "bytes_received": 0,
    "bytes_sent": 0,
    "com_delete": 0,
    "com_delete_multi": 0,
    "com_insert": 0,
    "com_insert_select": 0,
    "com_load": 0,
    "com_replace": 0,
    "com_replace_select": 0,
    "com_select": 0,
    "com_update": 0,
    "com_update_multi": 0,
    "connections": 0,
    "created_tmp_disk_tables": 0,
    "created_tmp_files": 0,
    "created_tmp_tables": 0,
    "key_read_requests": 0,
    "key_reads": 0,
    "key_write_requests": 0,
    "key_writes": 0,
    "max_used_connections": 3.0,
    "open_files": 14.0,
    "open_tables": 316.0,
    "opened_tables": 0,
    "qcache_free_blocks": 1.0,
    "qcache_free_memory": 16760152.0,
    "qcache_hits": 0,
    "qcache_inserts": 0,
    "qcache_lowmem_prunes": 0,
    "qcache_not_cached": 0,
    "qcache_queries_in_cache": 0,
    "qcache_total_blocks": 1.0,
    "questions": 0,
    "select_full_join": 0,
    "select_full_range_join": 0,
    "select_range": 0,
    "select_range_check": 0,
    "select_scan": 0,
    "slave_open_temp_tables": 0.0,
    "slow_launch_threads": 0,
    "slow_queries": 0,
    "sort_range": 0,
    "sort_rows": 0,
    "sort_scan": 0,
    "table_locks_immediate": 0,
    "table_locks_waited": 0,
    "threads_cached": 2.0,
    "threads_connected": 1.0,
    "threads_created": 0,
    "threads_running": 1.0,
    "uptime": 35837.0
}

mysql:

{

"aborted_clients": 0,

"aborted_connects": 0,

"binlog_cache_disk_use": 0,

"binlog_cache_use": 0,

"bytes_received": 0,

"bytes_sent": 0,

"com_delete": 0,

"com_delete_multi": 0,

"com_insert": 0,

"com_insert_select": 0,

"com_load": 0,

"com_replace": 0,

"com_replace_select": 0,

"com_select": 0,

"com_update": 0,

"com_update_multi": 0,

"connections": 0,

"created_tmp_disk_tables": 0,

"created_tmp_files": 0,

"created_tmp_tables": 0,

"key_read_requests": 0,

"key_reads": 0,

"key_write_requests": 0,

"key_writes": 0,

"max_used_connections": 3.0,

"open_files": 14.0,

"open_tables": 316.0,

"opened_tables": 0,

"qcache_free_blocks": 1.0,

"qcache_free_memory": 16760152.0,

"qcache_hits": 0,

"qcache_inserts": 0,

"qcache_lowmem_prunes": 0,

"qcache_not_cached": 0,

"qcache_queries_in_cache": 0,

"qcache_total_blocks": 1.0,

"questions": 0,

"select_full_join": 0,

"select_full_range_join": 0,

"select_range": 0,

"select_range_check": 0,

"select_scan": 0,

"slave_open_temp_tables": 0.0,

"slow_launch_threads": 0,

"slow_queries": 0,

"sort_range": 0,

"sort_rows": 0,

"sort_scan": 0,

"table_locks_immediate": 0,

"table_locks_waited": 0,

"threads_cached": 2.0,

"threads_connected": 1.0,

"threads_created": 0,

"threads_running": 1.0,

"uptime": 35837.0

}

Nice,

I restart MySQL

sudo systemctl restart mysql

1	sudo systemctl restart mysql

I restart my Nixstats service

service nixstatsagent restart

1	service nixstatsagent restart

Now let’s monitor MySQL in Nixstats

I can now view MySQL metrix

Status Page

Nixstats allows you to create a status page ( https://nixstats.com/pages/overview ) where you can add any servers or monitors to that page. This stats page is truly awesome, it builds a live status page based on data coming from your installed agents.

You can even set up a custom subdomain that points to a Nixstats hosted status page too (e.g https://status.yourdomain.com).

FYI: An SSL certificate on your staus page may take a few hours to set up. Don’t panic if it is not instantatly available.

Nice.

This saves doing it yourself. The status page will look like it running on your server.

Status Page

You can create a status page that automatically aggregates collected data from monitors and displays them in a nice layout.

This is great, I used to do my own status pages but not anymore.

Alerts

I added a contact so I could receive alerts. I could then add my mobile, email and PushOver key (to receive push notifications) and Telegram Bot API token.

Test Alerts

I sent a test alert to each service against the contact.

I activated a Pushover licence on my Android device for about $7.49 AUD (one off) to ensure I keep getting Push Notifications.

Nixstats have links that show you how you can create a Telegram Bot and Pushover.net account.

Pushover will cost about $5 USD one off per device (see faq).

I created the following alerts

Alert: Disk Usage higher than 90%

Alert: Load greater than 90 per cent for 1 minute

Alert: Less than 5 percent memory free.

Summary of alerts.

I also added a CPU reached 95% one for 5 mins alert too (but it’s not pictured above)

I forgot to specify alert recipients and methods for each alert so I edited each alert and added the contact.

Now it’s time to test the alerts.

I shut down a server to test alerts

shutdown -h now

1	shutdown -h now

Alerts to my defined Email, SMS, Push and Telegram are working a treat 🙂

After I rebooted the server I also received alerts about the server being back up.

The status page showed the server that was offline too.

Nice

Troubleshooting

I had an issue instaling the agent on Debian

I ran the following command

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh #######################
--2018-10-02 00:41:38--  https://www.nixstats.com/nixstatsagent.sh
Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8113, 2400:cb00:2048:1::6819:8013, 104.25.129.19, ...
Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8113|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'nixstatsagent.sh' not modified on server. Omitting download.

nixstatsagent.sh: line 508: [: Installer exited with error code 0. See nixstatsagent.log for details.: integer expression expected

   An error occured, please check the install log file (nixstatsagent.log)!

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh #######################

--2018-10-02 00:41:38-- https://www.nixstats.com/nixstatsagent.sh

Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8113, 2400:cb00:2048:1::6819:8013, 104.25.129.19, ...

Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8113|:443... connected.

HTTP request sent, awaiting response... 304 Not Modified

File 'nixstatsagent.sh' not modified on server. Omitting download.

nixstatsagent.sh: line 508: [: Installer exited with error code 0. See nixstatsagent.log for details.: integer expression expected

An error occured, please check the install log file (nixstatsagent.log)!

Contents of nixstatsagent.log

cat nixstatsagent.log
Ign:1 http://deb.debian.org/debian stretch InRelease
Hit:2 http://deb.debian.org/debian-security stretch/updates InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Ign:5 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic InRelease
Ign:7 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic Release
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Hit:9 https://packages.sury.org/php stretch InRelease
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Err:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
  404  Not Found
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Reading package lists...
W: The repository 'http://ppa.launchpad.net/ondrej/php/ubuntu cosmic Release' does not have a Release file.
E: Failed to fetch http://ppa.launchpad.net/ondrej/php/ubuntu/dists/cosmic/main/binary-amd64/Packages  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
nixstatsagent.sh: line 118: apt-get upgrade returned error code 100. Please see nixstatsagent.log for details.: command not found

cat nixstatsagent.log

Ign:1 http://deb.debian.org/debian stretch InRelease

Hit:2 http://deb.debian.org/debian-security stretch/updates InRelease

Hit:3 http://deb.debian.org/debian stretch-updates InRelease

Hit:4 http://deb.debian.org/debian stretch Release

Ign:5 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic InRelease

Ign:7 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic Release