and

Setting up a Raspberry PI as a DNS Sinkhole to block ads and Trackers

April 15, 2020 by Simon

What is PiHole (Version 5)?

fyi: I updated this post 3 months after I created it as I killed my 32GB Raspberry Pi Micro SD card when I pulled the power (before a storm) without shutting down the Raspberry PIU first. Always shutdown the pi before removing the power. I have a 16GB Micro SD card that I will use instead.

I am following my guide to re setup up my PiHole.

Snip from WikiPedia: “Pi-hole is a Linux network-level advertisement and Internet tracker blocking application which acts as a DNS sinkhole (and optionally a DHCP server), intended for use on a private network. It is designed for use on embedded devices with network capability, such as the Raspberry Pi, but it can be used on other machines running Linux and cloud implementations. Pi-hole has the ability to block traditional website advertisements as well as advertisements in unconventional places, such as smart TVs and mobile operating system advertisements.”

What is a Raspberry PI?

A Raspberry PI is an inexpensive (5V Volt, 2 Amp) ARM based computer that can run off the power from a USB cable.

Here is a photo of my Raspberry Pi 3B+ with an Adafruit LCD Screen

My Raspberry Pi has the following specifications

4 x 1.4GHz 64-bit (quad-core processor)
1GB LPDDR2 SDRAM
Dual-band wireless LAN
Bluetooth 4.2/BLE,
Faster Ethernet
Extended 40-pin GPIO header
Full-size HDMI 4 USB 2.0 ports
5V/2.5A DC power input

My screen has the following specifications (purchased from Pakronics)

3.5″ display with 480×320 16-bit colour pixels
Resistive touch overlay

I plugged in a full sized USB Keyboard, Mouse and HDMI cable.

SD Card Choice

Read my guide to download and write an Raspberry Pi Operating System to an SD card.

I would not put a cheap/slow MicroSD card in the Raspberry PI, aim for at least a UHS (1) or UHS (3) speed SD card for the best bang for buck.

fyi: I bought a new 32GB Samsung UHS 1 Ultra Micro SD card and it died after 12 hours of use. I replaced it with another 32GB No name brand CLASS 10 SD Card I had laying around.

(after I killed my 32GB Micro SD card I have chosen a 16GB Micro SD card as it is all I have spare)

Raspberry Setup

I download and saved the Raspian (Full) Operating System to a SD Card and inserted it into my Raspberry PI 3B+ (view the guide here on preparing an Operating System on a SD card).

I used the American 110-240V AC to 5.25V 2500ma DC power supply (with a US to AUS adaptor) that came with the Adafruit Screen. It had a Micro USB connection on one end.

It did not work though (I just had a flashing red light on the Raspberry Pi).

I had an Australian 240V AC to 5V 2500ma DC power supply to Micro USB. from a previous project and it worked (the Raspberry Pi Started up).

I also have a number of Moki brand 240V to USB (1A and 2.4A) adapters.

I will use the 2.4mA plug. I know my Adafruit screen uses 100mA so this will do.

I plugged the HDMI cable into my Monitor and set up the HDMI as a Picture in Picture output so I can see my Main 4K screen (Display Port) and the Raspberry Pi HDMI input at the same time.

My First Raspberry Pi Boot

Mmmm my 4K screen with a 1080P HDMI picture in picture image (from the Raspberry Pi).

The Raspberry Pi booted fast and a welcome screen appeared

Apologies in advance, photos below are bad (I don’t have a HDMI capture card).

I clicked Next to setup the Raspberry PI

I set my timezone and language

I set a password

I skipped connecting to WiFi (I want pure Ethernet)

I was prompted to update the software (I clicked Next)

Setup is complete

I rebooted the Raspberry Pi

Second Boot

I changed further configuration by clicking the Raspberry Pi start button then Preferences then Raspberry PI Configuration

Screenshot of the Raspberry Pi menu showing Raspberry Pi Configuration

I changed the hostname to “raspberrypihole”, set Boot to CLI , Login as “pi“, and set Wait for network.

Update: After my Samsung SD card died I re setup my PI with a no name brand SD card and entered the name “raspberrypihole“

Under display I reviewed the display options

I enabled SSH, SPI and I2C.

Enable SSH, SPI and I2C features screenshot.

I increased the GPU memory to 132GB

Time for a Reboot

SSH Access

I do not want to leave a keyboard, mouse and screen connected once I finish setting it up so I setup a SSH connection to the Raspberry Pi.

TIP: Putty is a free program for SSH connections.

I SSH’ed (more information on SSH below) to the Raspberry Pi and ran these commands to update it’s software and firmware.

sudo apt-get update  && sudo apt-get upgrade

Output

[email protected]:~ $ sudo apt-get update  && sudo apt-get upgrade
Hit:1 http://archive.raspberrypi.org/debian buster InRelease
Hit:2 http://raspbian.raspberrypi.org/raspbian buster InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

The program “htop” is good for viewing system resources.

Now it’s time to look at the Adafruit screen and case.

I put the Raspberry PI in the Adafruit Case

I purchased this kit for the Raspberry Pi, the LCD screen just connects to the Raspberry Pi GPIO pins. It has a Adafruit LCD screen and a case for my Raspberry Pi 3B+

Photo of a Raspberry Pi 3B+, LCD screen and case

The screen just connects onto the GPIO pins

The LCD screen allows you to use pins below the screen.

GPIO pins documentation from https://www.raspberrypi.org/documentation/usage/gpio/

GPIO pin documentation from https://www.raspberrypi.org/documentation/usage/gpio/

A nice stack 🙂

Photo showing the LCD screen connected to the Pi

The case clips are hard to clip over the Raspberry Pi (Don’t force it or you might break for Pi)

Photo showing a case clip over the raspberry pi board

The case clip near the GPIO pins is on

Photo showing the case clip near the GPIO pins.

The clip near the power plug was off because the Raspberry Pi was not positioned correctly

Photo of the Raspberry PI off center of the lugs

After 30 minutes I carefully put the Raspberry Pi and LCD screen into the Case.

Photo showing the LCD screen in the case.

Side of the case with USB and Ethernet and USB porws exposed.

Photo of the Case exposing the USB and Ethernet port

HDMI, Power and Audio plugs are visible and lined up 🙂

Photo showing HDMI, Power and Audio plugs

The screen is visible through the case

The screen dips down on one side, I might have to prop it up (hot glue gun) a bit inside later

Photo showing the LCD screen dips to one side

SSH Connections to the Raspberry PI

I created an SSH connection to my Raspberry PI with MobaXterm (review here) and connected to it.

I ran the “ifconfig” command to get a list of all network interfaces.

I ran “ifconfig” to list all network interfaces.

I ran these commands to update my Raspberry PI Software

sudo apt-get update
sudo apt-get upgrade
sudo apt full-upgrade
sudo apt -y dist-upgrade

I updated the Pi Firmware too (this is dangerous, only update if you have issues).

sudo rpi-update

I rebooted and connected to the Raspberry Pi and ran this command to get the Ethernet and wireless mac address.

The first interface is my Ethernet adopter the second if the WiFi adaptor.

ifconfig |grep ether
ether b8:27:eb:d9:00:86 txqueuelen 1000
ether b8:27:eb:8c:55:d3 txqueuelen 1000

The first Mac address is my Ethernet address on The Raspberry PI and the second is WiFi.

I logged into my router (Telstra DJA0230) and clicked Advanced then Local Network. I could see my DHCP range was from 192.168.0.2 to 192.168.0.254, I shortened this to 192.168.0.2 to 192.168.0.200 (so I can set a static IP Address for the Raspberry PI) then I set a Static IP address for the Raspberry pi to 192.168.0.201.

I rebooted the Raspberry PI and checked the IP address

I logged into my Router (at https://192.168.0.1)

When my Samsung SD card died I had to re-setup a new SD card but the IP address came across as the mac address stayed the same (as it was the same hardware), I did, however, change the name of the Static IP hostname in my home router to match the new name “raspberrypihole” (not “pihole”)

I set a static IP for this Ethernet address and defined 192.168.0.201 as the IP address.

Setting Up PiHole on the RaspBerry PI

I SSH’ed to my Raspberry Pi (with the new IP address) and ran this command

Now its time to install Pi Hole onto My Raspberry Pi

wget -O basic-install.sh https://install.pi-hole.net
sudo bash basic-install.sh

I was presented with “This installer will transform your device into a network-wide ad blocker! “

I was presented with “This installer will transform your device into a network-wide ad blocker! “

PiHole is free, but powered by your donations (consider donating)

I donated. Thanks PiHole Team.

This will pay for itself in no time.

Static IP address is required.

I chose to just have PiHole work on Ethernet (and not Wifi)

I was prompted to set my upstream DNS provider.

I selected all default blacklist lists.

I allowed PiHole to use IPv4 and IPv6.

My IP and Gateway was displayed on the screen.

Final warning about setting a static IP address.

The PiHole IPv6 address is show

Install a admin interface (Yes)

Install lighthttpd (Yes)

I chose to log all DNS queries.

I oped to allow the viewing of all logged data. This is less secure but I can reduce this later.

PiHole is now setting up

Installation took about 10 minutes

A PiHole admin URL and Password was displayed (write this down)

I loaded the PiHole initial admin screen (http://192.168.0.201/admin/) and it was a bit empty.

PiHole Interface

I logged into my PiHole (at http://192.1768.0.201/admin/) with the password provided during setup.

The Raspberry Pi Pi Hole service was up and waiting for connections

I have Zero traffic going through the PiHole.

Before I add computers on my network to the PiHole I had better uninstall the nextdns.io (my blog post about NextDNS.io here) as the Pi will now be the main DNS blocking Sinkhole in our house.

On my Windows 10 PC I added the DNS server for the PiHole in IPV4 and IPV6.

I obtained the PiHole IPV4 and IPV6 addresses (1) PiHole Admin, 2) Pi Hole Settings Page, 3) IP Address)

PiHole IPV4 and IPV6 addresses.

I added the Pi Holes IPV4 IP address to my Windows 10 IP Settings.

I added the PIHole DNS to the IPV4 and IPV6 on my Windows 10 Ethernet adaptor

I added the Pi Holes IPV6 IP address to my Windows 10 IP Settings.

After 20 hours or rining computers through the Pi-Hole Admin interface I loaded the PiHole Admin Interface (at http://192.168.0.201/admin/index.php) was reporting stats.

I can view stats for Protocol and answered queries

I can also see stats for permitted and blocked domains

Default Block Lists

I can also see the source blocked domains

Add 3rd party block lists

I added these block lists to my PiHole list of sites to block (Thanks Jol)

https://v.firebog.net/hosts/Easyprivacy.txt
https://v.firebog.net/hosts/Prigent-Ads.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
https://hostfiles.frogeye.fr/multiparty-trackers-hosts.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt
https://v.firebog.net/hosts/Airelle-trc.txt
https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
https://gist.githubusercontent.com/anudeepND/adac7982307fec6ee23605e281a57f1a/raw/5b8582b906a9497624c3f3187a49ebc23a9cf2fb/Test.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
https://hosts-file.net/ad_servers.txt
https://hosts-file.net/emd.txt
https://hosts-file.net/exp.txt
https://hosts-file.net/grm.txt
https://hosts-file.net/psh.txt
https://isc.sans.edu/feeds/suspiciousdomains_Medium.txt
https://mirror1.malwaredomains.com/files/justdomains
https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt
http://someonewhocares.org/hosts/hosts
https://phishing.army/download/phishing_army_blocklist_extended.txt
https://ransomwaretracker.abuse.ch/downloads/CW_C2_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/LY_C2_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/TC_C2_DOMBL.txt
https://ransomwaretracker.abuse.ch/downloads/TL_C2_DOMBL.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/win10/spy.txt
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://raw.githubusercontent.com/ZeroDot1/CoinBlockerLists/master/hosts
https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://v.firebog.net/hosts/AdguardDNS.txt
https://v.firebog.net/hosts/Airelle-hrsk.txt
https://v.firebog.net/hosts/Easylist.txt
https://v.firebog.net/hosts/Easyprivacy.txt
https://v.firebog.net/hosts/Prigent-Ads.txt
https://v.firebog.net/hosts/Prigent-Malware.txt
https://v.firebog.net/hosts/Prigent-Phishing.txt
https://v.firebog.net/hosts/Shalla-mal.txt
https://v.firebog.net/hosts/static/SamsungSmart.txt
https://v.firebog.net/hosts/static/w3kbl.txt
https://www.malwaredomainlist.com/hostslist/hosts.txt
https://www.squidblacklist.org/downloads/dg-malicious.acl
http://sysctl.org/cameleon/hosts
https://zerodot1.gitlab.io/CoinBlockerLists/hosts
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
http://www.joewein.net/dl/bl/dom-bl.txt
http://www.networksec.org/grabbho/block.txt

I updated all block lists at http://192.168.0.201/admin/gravity.php

666,862 domains on my block list 🙂

I now have the Pi Hole blocking 666,862 domains, this number will increase as sites are added to the remote lists, nice.

LCD Screen Setup

I followed thin guide to setup the screen.

I can this code from the pi (logged in as root)

cd ~
wget https://raw.githubusercontent.com/adafruit/Raspberry-Pi-Installer-Scripts/master/adafruit-pitft.sh
chmod +x adafruit-pitft.sh
sudo ./adafruit-pitft.sh

I was prompted to choose a screen

Select configuration:
1. PiTFT 2.4", 2.8" or 3.2" resistive (240x320)
2. PiTFT 2.2" no touch (240x320)
3. PiTFT 2.8" capacitive touch (240x320)
4. PiTFT 3.5" resistive touch (320x480)
5. PiTFT Mini 1.3" or 1.54" display (240x240)
6. MiniPiTFT 1.14" display (240x135) - WARNING! CUTTING EDGE! WILL UPGRADE YOUR KERNEL TO LATEST
7. Quit without installing

SELECT 1-7:

I entered “3” for PiTFT 2.8″ capacitive touch (240×320)

I then was prompted to set the rotation of the screen

Select rotation:
1. 90 degrees (landscape)
2. 180 degrees (portait)
3. 270 degrees (landscape)
4. 0 degrees (portait)

SELECT 1-4:

I entered “3” for 270 degrees (landscape).

I was prompted to allow the console to appear on the screen

Would you like the console to appear on the PiTFT display? [y/n]
y

Install Summary

I rebooted

Reboot [y/n]
y

I edited /boot/config.txt and changed these values

framebuffer_width=320
framebuffer_height=240

Installing PADD to display PiHole stats on the LCD

I followed this guide to install PADD (the software that displays the PiHole stats on the LCD screen)

cd ~
wget -N https://github.com/jpmck/PADD/files/4320681/padd.txt
mv padd.txt paddsimon.sh
chmod +x paddsimon.sh

Making PADD starts at boot

Edit this file

sudo nano  ~/.bashrc

and add the following to the end of the file

# Run PADD
# If we're on the PiTFT screen (ssh is xterm)
if [ "$TERM" == "linux" ] ; then
  while :
  do
    /root/paddsimon.sh
    sleep 0.2
  done
fi

I rebooted the PI.

sudo showdown -r now

How to Update the PiHole from the CLI

I ran the following command to update the PiHole block lists

pihole -g

Output…

  [i] Pi-hole blocking is enabled
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: mirror1.malwaredomains.com (justdomains)
  [✓] Status: No changes detected

  [i] Target: sysctl.org (hosts)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_tracking.txt)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_ad.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (ad_servers.txt)
  [✓] Status: No changes detected

  [i] Target: raw.githubusercontent.com (ytadblock.txt)
  [✓] Status: Retrieval successful

  [i] Target: v.firebog.net (Easyprivacy.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Prigent-Ads.txt)
  [✓] Status: No changes detected

  [i] Target: gitlab.com (notrack-blocklist.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (spy.txt)
  [✓] Status: Retrieval successful

  [i] Target: www.github.developerdan.com (ads-and-tracking-extended.txt)
  [✓] Status: Retrieval successful

  [i] Target: hostfiles.frogeye.fr (firstparty-trackers-hosts.txt)
  [✓] Status: Retrieval successful

  [i] Target: hostfiles.frogeye.fr (multiparty-trackers-hosts.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (android-tracking.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (SmartTV.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (AmazonFireTV.txt)
  [✓] Status: Retrieval successful

  [i] Target: v.firebog.net (Airelle-trc.txt)
  [✓] Status: No changes detected

  [i] Target: bitbucket.org (Mandiant_APT1_Report_Appendix_D.txt)
  [✓] Status: No changes detected

  [i] Target: gist.githubusercontent.com (Test.txt)
  [✓] Status: Retrieval successful

  [i] Target: gitlab.com (notrack-malware.txt)
  [✓] Status: Retrieval successful

  [i] Target: hosts-file.net (emd.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (exp.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (grm.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (psh.txt)
  [✓] Status: No changes detected

  [i] Target: isc.sans.edu (suspiciousdomains_Medium.txt)
  [✓] Status: Retrieval successful

  [i] Target: mirror.cedia.org.ec (immortal_domains.txt)
  [✓] Status: No changes detected

  [i] Target: someonewhocares.org (hosts)
  [✓] Status: No changes detected

  [i] Target: phishing.army (phishing_army_blocklist_extended.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (CW_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (LY_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (RW_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (TC_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: ransomwaretracker.abuse.ch (TL_C2_DOMBL.txt)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (spy.txt)
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (hosts)
  [✗] Status: Not found
  [✗] List download failed: no cached list available

  [i] Target: reddestdream.github.io (minimalhosts)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_malvertising.txt)
  [✓] Status: Retrieval successful

  [i] Target: v.firebog.net (AdguardDNS.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Airelle-hrsk.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Easylist.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Prigent-Malware.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Prigent-Phishing.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (Shalla-mal.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (SamsungSmart.txt)
  [✓] Status: No changes detected

  [i] Target: v.firebog.net (w3kbl.txt)
  [✓] Status: No changes detected

  [i] Target: www.malwaredomainlist.com (hosts.txt)
  [✓] Status: No changes detected

  [i] Target: www.squidblacklist.org (dg-malicious.acl)
  [✗] Status: Connection Timed Out (Cloudflare)
  [✗] List download failed: no cached list available

  [i] Target: zerodot1.gitlab.io (hosts)
  [✓] Status: No changes detected

  [i] Target: zeustracker.abuse.ch (blocklist.php?download=domainblocklist)
  [✓] Status: Retrieval successful

  [i] Target: www.joewein.net (dom-bl.txt)
  [✓] Status: Retrieval successful

  [i] Target: www.networksec.org (block.txt)
  [✓] Status: Retrieval successful
  [i] Received empty file: using previously cached list

  [✓] Consolidating blocklists
  [✓] Extracting domains from blocklists
  [i] Number of domains being pulled in by gravity: 1178534
  [✓] Removing duplicate domains
  [i] Number of unique domains trapped in the Event Horizon: 954486
  [i] Number of whitelisted domains: 2
  [i] Number of blacklisted domains: 0
  [i] Number of regex filters: 0
  [✓] Parsing domains into hosts format
  [✓] Cleaning up stray matter

  [✓] Force-reloading DNS service
  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

I can view all possible command line options by running

pihole /?

Output..

Usage: pihole [options]
Example: 'pihole -w -h'
Add '-h' after specific commands for more information on usage

Whitelist/Blacklist Options:
  -w, whitelist       Whitelist domain(s)
  -b, blacklist       Blacklist domain(s)
  --wild, wildcard     Wildcard blacklist domain(s)
  --regex, regex       Regex blacklist domains(s)
                        Add '-h' for more info on whitelist/blacklist usage

Debugging Options:
  -d, debug           Start a debugging session
                        Add '-a' to enable automated debugging
  -f, flush           Flush the Pi-hole log
  -r, reconfigure     Reconfigure or Repair Pi-hole subsystems
  -t, tail            View the live output of the Pi-hole log

Options:
  -a, admin           Web interface options
                        Add '-h' for more info on Web Interface usage
  -c, chronometer     Calculates stats and displays to an LCD
                        Add '-h' for more info on chronometer usage
  -g, updateGravity   Update the list of ad-serving domains
  -h, --help, help    Show this help dialog
  -l, logging         Specify whether the Pi-hole log should be used
                        Add '-h' for more info on logging usage
  -q, query           Query the adlists for a specified domain
                        Add '-h' for more info on query usage
  -up, updatePihole   Update Pi-hole subsystems
                        Add '--check-only' to exit script before update is perfo                                                     rmed.
  -v, version         Show installed versions of Pi-hole, Web Interface & FTL
                        Add '-h' for more info on version usage
  uninstall           Uninstall Pi-hole from your system
  status              Display the running status of Pi-hole subsystems
  enable              Enable Pi-hole subsystems
  disable             Disable Pi-hole subsystems
                        Add '-h' for more info on disable usage
  restartdns          Restart Pi-hole subsystems
  checkout            Switch Pi-hole subsystems to a different Github branch
                        Add '-h' for more info on checkout usage

After 1 Week

After 1 week stats were rolling into the PIHole.

40% of all traffic was being blocked.

I could see blocked and allowed domain calls

I can white list domains if they are blocked.

I white listed fearby.com and events.gfe.nvidia.com

Done

This is what it looks like done

Nice

Did it block Ad’s

Mostly Yes. Not all advertisements are blocked but most are.

Some YouTube Advertisements seem to get through but I am seeing far less Advertisements in web pages

Using Python to use buttons on the PiTFT Plus 320×240 TFT Touchscreen with a PiHole

Read this guide to make the buttons word: Using Python to use buttons on the PiTFT Plus 320×240 TFT Touchscreen with a PiHole

Troubleshooting

If you receive an update about updating languages on your first boot while updating you can manually update all software by running this after you first reboot in a Terminal window.

sudo apt-get update && apt-get full-upgrade

If your Micro SD cad is filling up you can run to free some space

sudo apt clean

I needed to white list “events.gfe.nvidia.com” to allow my video card drivers to upodate.

Cooling

The Raspberry Pi is running cool at 47c (even though it is in a tight space).

I might add a heat pipe to it and have an external fan. I will thermal epoxy the hat pipe to the Pi CPU and run it outside to a external heat sink and fan.

eBay purchase for a fan, thermal epoxy and heatpipes.

I have many spare heat sinks laying around.

I will update when the part’s arrive.

Update: I did not end up adding extra cooling, there was no need in summer.

Rotating the screen

I did exit my /boot/config.txt to rotate my LCD Screen orientation

Buttons

Read this guide to see how I setup a Python script to make my buttons work.

Do edit your /boot.config.txt to configure your screen rotation (if need be) and to check if the LCD screen is setup (by Adafruit)

framebuffer_width=240
framebuffer_height=320

Backup and Restore PiHole Settings

I used the PiHole Backup feature (at http://192.168.0.201/admin/settings.php?tab=teleporter) to backup all of my PiHole Settings to a zip file.

Handy Links

Handy Guide: https://learn.adafruit.com/pi-hole-ad-pitft-tft-detection-display/pitft-configuration to configure the LCD Screen

Schematics of the screen: https://learn.adafruit.com/assets/25555

Donate to PiHole: https://pi-hole.net/donate/

Raspberry Pi GPIO Pins: https://www.raspberrypi.org/documentation/usage/gpio/

v 2.3 Updating to PiHole 5.1.2

How to backup and restore a MySQL database on Windows and Linux

April 21, 2019 by Simon

Why backup and restore

This is a quick guide demonstrating how you can backup and restore a MySQL database on Windows and Linux using Adminer.

You may need to know how to backup a restore a database for a number of reasons..

e.g

Send the database to someone to debug or give feedback while learning.
Move the database from a local machine to the cloud
Move the database from cloud vendor A to cloud vendor B
etc.

Having a backup of the VM is good but having a backup of the database too is better. I use UpCloud for hosting my VM’s and setting backups is easy. But I cannot download those backups.

Murphy’s Law

“If anything can go wrong, it will”

The most important reason for taking a backup and knowing how to restore it is for disaster recovery reasons.

Backup (the easiest way) with Adminer

Adminer is a free PHP based IDE for MySQL and other databases. Simply install Adminer and save the file on your local computer or remote web server directory.

FYI: The Adminer author Jakub Vrana has a patron page, I am a patron of this awesome software.

Snip from Adminers website. “Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch andMongoDB.”

TIP: The file would be publicly accessible to anyone so don’t save it to a common area, obfuscate the file, protect it of delete the file when you are done using it.

Once Adminer is installed load it in a web browser, login with your MySQL credentials. Once you login you will see all databases and an Import and Export menu.

Adminer main screen, all databases and import and export menu.

tbtest is a simple database with one table and 4 fields (ID, Key, Value and Modified)

.Click Export to open the export screen.

Export screen showing a list of databases and export options

Click Export, a SQL file will be generated (this is the export of the database).

Here is a save of the file:
https://fearby.com/wp-content/uploads/export.txt

Exported view of https://dev.mysql.com/doc/workbench/en/wb-admin-export-import-management.html

Its that simple.

If I add a binary blob file to the table and upload a PNG file lets see how the export looks.

Screenshot o the new table with a blog field in Adminer UI

Let export the database again in Adminer and check out the output. I used Sublime Text editor to view the export file.

New Export shows the binary file in the Backup SQL file

Restore (the easiest way) with Adminer

OK lets delete the tbtest database and then restore it with Adminer. I used Adminer to delete (DROP) the database.

Database “dbtest” deleted.

Now lets create a blank database to restore to (same name).

Database created.

Now lets import the database backup using Adminer.

Click Import, select the backup file and un-tick Stop on errors.

Import screenshot, dxtest selectded, Restore file selected, stop on errors disabled

TIP: The 2MB next the the choose file button is defined by your web server and PHP configuration. If you are trying to import a larger database (e.g 80MB) first increase the limits in your web server and PHP (via php.ini).

The Import (restore should take seconds)

The database was imported from a backup, all tables and records imported just fine.

Bonus methods.

On Ubuntu use this guide to backup from the command line. If you use the Oracle MySQL Workbench read this.

I hope this helps someone.

Updating NGINX to the development branch to get more frequent updates and features over the stable branch

November 20, 2018 by Simon

Updating NGINX to the development branch (on Ubuntu) to get more frequent updates and features over the stable branch

Aside

I have a number of guides on moving away from CPanel, Setting up VM’s on UpCloud, AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. View all recent posts here https://fearby.com/all/

Now on with the post

Warning

Backup your Nginx and Server before making any changes. The Nginx development branch is quite stable but anything can happen. If your site is mission-critical then stay on the stable branch.

Nginx Branches

By default, you will most likely get the stable branch of Nginx when instaling and updating Nginx. I have been running the stable version for the last few years but was made aware of a DDoS vulnerability in Nginx.

Here is a good write-up on development merges into the stable branch.

Nginx Updates

Widely-used #Nginx server releases versions 1.15.6 and 1.14.1 to patch two HTTP/2 implementation vulnerabilities that might cause excessive memory consumption (CVE-2018-16843) & CPU usage (CVE-2018-16844), allowing a remote attacker to perform #DoS attackhttps://t.co/1Z3JoghoBr pic.twitter.com/qQ3pOFD1Lk
— The Hacker News (@TheHackersNews) November 9, 2018

I was aware recently of a DDoS bug affecting Nginx and the recommendation was to update ot Nginx 1.15.6 development branch (or 1.14.1 stable branch).

A few days ago no 1.14.1 update was available but a 1.15.6 was, should I switch to the development branch to get updates earlier?

Reminder to update your #nginx installations to the 1.14.1 stable or the 1.15.6 mainline versions for critical security patches released this week. #NGINXPlus customers, see instructions for updating based on the patch released 10/30 https://t.co/KitsOWIJkb
— NGINX, Inc. (@nginx) November 8, 2018

Recent Nginx Changes

Here are the recent changes to Nginx: http://nginx.org/en/CHANGES

Changes with nginx 1.15.6                                        06 Nov 2018

    *) Security: when using HTTP/2 a client might cause excessive memory
       consumption (CVE-2018-16843) and CPU usage (CVE-2018-16844).

    *) Security: processing of a specially crafted mp4 file with the
       ngx_http_mp4_module might result in worker process memory disclosure
       (CVE-2018-16845).

    *) Feature: the "proxy_socket_keepalive", "fastcgi_socket_keepalive",
       "grpc_socket_keepalive", "memcached_socket_keepalive",
       "scgi_socket_keepalive", and "uwsgi_socket_keepalive" directives.

    *) Bugfix: if nginx was built with OpenSSL 1.1.0 and used with OpenSSL
       1.1.1, the TLS 1.3 protocol was always enabled.

    *) Bugfix: working with gRPC backends might result in excessive memory
       consumption.


Changes with nginx 1.15.5                                        02 Oct 2018

    *) Bugfix: a segmentation fault might occur in a worker process when
       using OpenSSL 1.1.0h or newer; the bug had appeared in 1.15.4.

    *) Bugfix: of minor potential bugs.


Changes with nginx 1.15.4                                        25 Sep 2018

    *) Feature: now the "ssl_early_data" directive can be used with OpenSSL.

    *) Bugfix: in the ngx_http_uwsgi_module.
       Thanks to Chris Caputo.

    *) Bugfix: connections with some gRPC backends might not be cached when
       using the "keepalive" directive.

    *) Bugfix: a socket leak might occur when using the "error_page"
       directive to redirect early request processing errors, notably errors
       with code 400.

    *) Bugfix: the "return" directive did not change the response code when
       returning errors if the request was redirected by the "error_page"
       directive.

    *) Bugfix: standard error pages and responses of the
       ngx_http_autoindex_module module used the "bgcolor" attribute, and
       might be displayed incorrectly when using custom color settings in
       browsers.
       Thanks to Nova DasSarma.

    *) Change: the logging level of the "no suitable key share" and "no
       suitable signature algorithm" SSL errors has been lowered from "crit"
       to "info".


Changes with nginx 1.15.3                                        28 Aug 2018

    *) Feature: now TLSv1.3 can be used with BoringSSL.

    *) Feature: the "ssl_early_data" directive, currently available with
       BoringSSL.

    *) Feature: the "keepalive_timeout" and "keepalive_requests" directives
       in the "upstream" block.

    *) Bugfix: the ngx_http_dav_module did not truncate destination file
       when copying a file over an existing one with the COPY method.

    *) Bugfix: the ngx_http_dav_module used zero access rights on the
       destination file and did not preserve file modification time when
       moving a file between different file systems with the MOVE method.

    *) Bugfix: the ngx_http_dav_module used default access rights when
       copying a file with the COPY method.

    *) Workaround: some clients might not work when using HTTP/2; the bug
       had appeared in 1.13.5.

    *) Bugfix: nginx could not be built with LibreSSL 2.8.0.


Changes with nginx 1.15.2                                        24 Jul 2018

    *) Feature: the $ssl_preread_protocol variable in the
       ngx_stream_ssl_preread_module.

    *) Feature: now when using the "reset_timedout_connection" directive
       nginx will reset connections being closed with the 444 code.

    *) Change: a logging level of the "http request", "https proxy request",
       "unsupported protocol", and "version too low" SSL errors has been
       lowered from "crit" to "info".

    *) Bugfix: DNS requests were not resent if initial sending of a request
       failed.

    *) Bugfix: the "reuseport" parameter of the "listen" directive was
       ignored if the number of worker processes was specified after the
       "listen" directive.

    *) Bugfix: when using OpenSSL 1.1.0 or newer it was not possible to
       switch off "ssl_prefer_server_ciphers" in a virtual server if it was
       switched on in the default server.

    *) Bugfix: SSL session reuse with upstream servers did not work with the
       TLS 1.3 protocol.


Changes with nginx 1.15.1                                        03 Jul 2018

    *) Feature: the "random" directive inside the "upstream" block.

    *) Feature: improved performance when using the "hash" and "ip_hash"
       directives with the "zone" directive.

    *) Feature: the "reuseport" parameter of the "listen" directive now uses
       SO_REUSEPORT_LB on FreeBSD 12.

    *) Bugfix: HTTP/2 server push did not work if SSL was terminated by a
       proxy server in front of nginx.

    *) Bugfix: the "tcp_nopush" directive was always used on backend
       connections.

    *) Bugfix: sending a disk-buffered request body to a gRPC backend might
       fail.


Changes with nginx 1.15.0                                        05 Jun 2018

    *) Change: the "ssl" directive is deprecated; the "ssl" parameter of the
       "listen" directive should be used instead.

    *) Change: now nginx detects missing SSL certificates during
       configuration testing when using the "ssl" parameter of the "listen"
       directive.

    *) Feature: now the stream module can handle multiple incoming UDP
       datagrams from a client within a single session.

    *) Bugfix: it was possible to specify an incorrect response code in the
       "proxy_cache_valid" directive.

    *) Bugfix: nginx could not be built by gcc 8.1.

    *) Bugfix: logging to syslog stopped on local IP address changes.

    *) Bugfix: nginx could not be built by clang with CUDA SDK installed;
       the bug had appeared in 1.13.8.

    *) Bugfix: "getsockopt(TCP_FASTOPEN) ... failed" messages might appear
       in logs during binary upgrade when using unix domain listen sockets
       on FreeBSD.

    *) Bugfix: nginx could not be built on Fedora 28 Linux.

    *) Bugfix: request processing rate might exceed configured rate when
       using the "limit_req" directive.

    *) Bugfix: in handling of client addresses when using unix domain listen
       sockets to work with datagrams on Linux.

    *) Bugfix: in memory allocation error handling.

Development branch changes are made every few weeks and stable branch changes are made less often.

Updating Nginx

Normally you update Nginx bu running an update and upgrade

apt-get update && apt-get upgrade

Restart Nginx for good measure

/etc/init.d/nginx restart

Checking NGINX Version

nginx -v
nginx version: nginx/1.14.1

Changing your repository to the development branch

I changed ot the development branch by running

sudo add-apt-repository ppa:nginx/development

Update and upgrade Nginx

apt-get update && apt-get upgrade

Restart Nginx for good measure

/etc/init.d/nginx restart

Checking NGINX Version

nginx -v
nginx version: nginx/1.16.6

Removing the stable Nginx repository

Run this command to remove the stable branch of Nginx

sudo add-apt-repository -r ppa:nginx/stable

Check to see if the development branch is listed

grep -r --include '*.list' '^deb ' /etc/apt/sources.list* |grep nginx
/etc/apt/sources.list.d/nginx-ubuntu-development-bionic.list:deb http://ppa.launchpad.net/nginx/development/ubuntu bionic main

Good luck and I hope this guide helps someone

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.0 Initial post

Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker

November 7, 2018 by Simon

This is how I checked the compatibility of my WordPress theme and plugin (code) with PHP Compatibility Checker

Aside

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. PHP is my programming language of choice.

Now on with the post

Snip from: https://wordpress.org/plugins/php-compatibility-checker/

What is PHP Compatibility Checker

> The WP Engine PHP Compatibility Checker can be used by any WordPress website on any web host to check PHP version compatibility.

> This plugin will lint theme and plugin code inside your WordPress file system and give you back a report of compatibility issues for you to fix. Compatibility issues are categorized into errors and warnings and will list the file and line number of the offending code, as well as the info about why that line of code is incompatible with the chosen version of PHP. The plugin will also suggest updates to themes and plugins, as a new version may offer compatible code.

> This plugin does not execute your theme and plugin code, as such this plugin cannot detect runtime compatibility issues.
Please note that linting code is not perfect. This plugin cannot detect unused code-paths that might be used for backwards compatibility, and thus might show false positives. We maintain a whitelist of plugins that can cause false positives. We are continuously working to ensure the checker provides the most accurate results possible.
This plugin relies on WP-Cron to scan files in the background. The scan will get stuck if the site’s WP-Cron isn’t running correctly. Please see the FAQ for more information.

Install PHP Compatibility Checker

PHP Compatibility Checker

I instaled by SSH’ing to my server and opening my WP Plugins folder

cd /www-root/wp-content/plugins/

I grabbed the latest download URL from here (hover over the download button), at the time of writing this was the latest version: https://downloads.wordpress.org/plugin/php-compatibility-checker.1.4.6.zip

I downloaded the plugin on my server (then unzipped it and deleted the zip)

wget https://downloads.wordpress.org/plugin/php-compatibility-checker.1.4.6.zip
unzip php-compatibility-checker.1.4.6.zip
rm php-compatibility-checker.1.4.6.zip

Enable PHP Compatibility Checker Plugin

I enabled the plugin

I clicked on the following message

> You have just activated the PHP Compatibility Checker. Start scanning your plugins and themes for compatibility with the latest PHP versions now!

I already have PHP 7.2 installed so let’s scan my site. PHP 7.3 will be available in December and it is already being tested in beta.

PHP Versions

Site Scanning

PHP Compatibility Checker site scanning is very business-like

PHP Compatability Checker Scan Results

2 of 22 plugins I use were not PHP 7.2 compatible (WordFence and WP Meta SEO)?

I read on twitter that Wordfence may be a false positive.

Clicking toggle details reveal why the scan failed. A Two Factor Auth plugin was all OK.

Your results will hopefully be…

> PHP 7.2 compatible

Of if errors exist it should explain why it did not pass.

FILE: /www-root/wp-content/plugins/wp-meta-seo/jutranslation/jutranslation.php
> —————————————————————————————-
> FOUND 1 ERROR AFFECTING 1 LINE
> —————————————————————————————-
> 251 | ERROR | The function is_countable() is not present in PHP version 7.2 or earlier
> —————————————————————————————-

I can’t wait for PHP 7.3 scanning. I will update this post in December 2018 after PHP 7.3 is released.

Good luck and I hope this guide helps someone

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.0 Initial post

Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App

October 28, 2018 by Simon

Here is a quick guide to show you how to add two-factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA authenticator app

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line.

Why Secure WordPress

WordPress CMS is a widely targeted CMS for hackers. View the official WordPress stats on WordPress Version/PHP and MySQL Version. View WordPress vulnerabilities here.

Read the Sucuri 2017 report on reported WordPress Hacks here (spoiler 34,371 infected websites in 2017).

Plugins exist to secure and scan WordPress. Read my blog post here on the now-retired Gravityaity Scan plugin and the awesome WordFence security plugin.

You (and hackers) can scan your site with https://wpscans.com/ or other open-source tools like wp-scan from OWASP ZAP. If you manage a WordPress site I’d recommend you install Kali Linux to scan your site.

Running a wp scan in Kali Linux is easy.

wpscan --url https://fearby.com --debug-output 2> ~/Desktop/wpscan.txt

The output from the Kali Linux wpscan tool

What are Hardware YubiCo YubiKeys

Read my guide here to see what YubiCo YubiKeys are and how to use them.

Get the Two-Factor Plugin for WordPress Plugin

Plugin: https://en-au.wordpress.org/plugins/two-factor/

Two-Factor

Plugin Page at WordPress.org

The source code for this plugin is available (nice): https://github.com/georgestephanis/two-factor. This plugin was updated 2 weeks ago (nice).

Downloading the Plugin

FYI: I do not allow downloading or updating of plugins in WordPress (via FTP), I prefer SSH manual downloading. FTP plugin installation and updating are not allowed on my site.

I got the latest download URL (e.g. https://downloads.wordpress.org/plugin/two-factor.zip) by copying the URL from the download button above.

I connected to my server via SSH and navigated to my WordPress plugin folder

cd /your-www-root/wp-content/plugins

I download the plugin.

[email protected]:/your-www-root/wp-content/plugins# wget https://downloads.wordpress.org/plugin/two-factor.zip
--2018-10-28 14:44:27--  https://downloads.wordpress.org/plugin/two-factor.zip
Resolving downloads.wordpress.org (downloads.wordpress.org)... 198.143.164.250
Connecting to downloads.wordpress.org (downloads.wordpress.org)|198.143.164.250|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 47882 (47K) [application/octet-stream]
Saving to: 'two-factor.zip'

two-factor.zip                             100%[=======================================================================================>]  46.76K  --.-KB/s    in 0.001s

2018-10-28 14:44:27 (37.1 MB/s) - 'two-factor.zip' saved [47882/47882]

I extracted the plugin zip file

[email protected]:/your-www-root/wp-content/plugins# unzip two-factor.zip
Archive:  two-factor.zip
   creating: two-factor/
   creating: two-factor/assets/
  inflating: two-factor/assets/banner-1544x500.png
  inflating: two-factor/assets/banner-772x250.png
  inflating: two-factor/assets/icon-128x128.png
  inflating: two-factor/assets/icon-256x256.png
  inflating: two-factor/class.two-factor-core.php
   creating: two-factor/includes/
  inflating: two-factor/includes/function.login-header.php
   creating: two-factor/includes/Google/
  inflating: two-factor/includes/Google/u2f-api.js
   creating: two-factor/includes/Yubico/
  inflating: two-factor/includes/Yubico/U2F.php
   creating: two-factor/providers/
  inflating: two-factor/providers/class.two-factor-backup-codes.php
  inflating: two-factor/providers/class.two-factor-dummy.php
  inflating: two-factor/providers/class.two-factor-email.php
  inflating: two-factor/providers/class.two-factor-fido-u2f-admin-list-table.php
  inflating: two-factor/providers/class.two-factor-fido-u2f-admin.php
  inflating: two-factor/providers/class.two-factor-fido-u2f.php
  inflating: two-factor/providers/class.two-factor-provider.php
  inflating: two-factor/providers/class.two-factor-totp.php
   creating: two-factor/providers/css/
  inflating: two-factor/providers/css/fido-u2f-admin.css
   creating: two-factor/providers/js/
  inflating: two-factor/providers/js/fido-u2f-admin-inline-edit.js
  inflating: two-factor/providers/js/fido-u2f-admin.js
  inflating: two-factor/providers/js/fido-u2f-login.js
  inflating: two-factor/readme.md
  inflating: two-factor/readme.txt
  inflating: two-factor/two-factor.php
  inflating: two-factor/user-edit.css

Enable the Plugin

Don’t forget to update the plugin in WordPress.

Once the plugin is enabled I can setup Two-factor authentication

Edit your Users

To setup two-factor authentication open your WordPress users screen (/wp-admin/users.php).

Notice the Two-Factor column

Edit your desired user to enable two-factor login options

Scroll down to Two Factor Options header, you will see a QR code that you can scan with your two-factor authentication app (e.g Google Authenticator or YubiCo Authenticator).

Always generate and save backup codes in case you lose your YubiKeys or authenticator app.

You can enable authentication methods as required.

Add the code to your Authenticator app. I will add mine to my Yubico Authenticator app that requires the insertion of a physical YubiKey. I can read my YubiKey via NFC and use my mobile phone to generate one time passwords too. Read here to learn about YubiKey 2FA (touch) devices. I have secured my Ubuntu/Debian and macOSX with these keys,

TIP: Don’t forget to save the user after editing.

Add the YubiKey 2FA (touch) to WordPress logins.

While editing a user click Register New Key under Security Keys

Add your primary and backup YubiKey as required (I added both of mine).

Enable all desired 2FA options

Email (OFF)
Time based One-Time Password (Authenticator App) (ON)
FIDO Universal 2nd Factor (U2F) – YubiKey Insertion and touch (ON)
Backup Codes (ON)

TIP: Don’t forget to save the user after editing.

Users Table

Aim to set up every user who has access to your WordPress to use 2FA.

Mobile 2FA login

I tested logos via mobile and I was prompted to tab my YubiKey to my phone. Nice.

What happens at login?

When One Time Password is enabled as the primary authentication method I am prompted for a one-time password after entering my username and password. I then need to insert my YubiKey (or tap the YubiKey to my phone (via NFC)) to generate a one time password.

When FIDO is enabled I need to insert my YubiKey and press the button.

Conclusion

I can now secure my WordPress site with 2FA protections without expensive security plugins.

I hope this guide helps someone.

More

Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software

October 4, 2018 by Simon

This post aims to show you how you can use a Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and other software and services.

Background

Although I am a developer I do like security related topics and I try and do as much as I can to secure my systems and applications. Reading the Multi-Factor Authentication Wikipedia page has all the details on Multi-Factor authentication.

I have been a big fan of 1Password to generate strong and unique passwords for separate accounts for a while now. Read my guide on upgrading from a standalone 1Password licence to a 1Password subscription. I love generating unique and complex passwords with 1Password.

But what happens if someone gets access to my 1password vault? Yubico has a catalogue of support services that I can use Yubikeys with to have, 1password is one supported service 🙂

I want to add Yubico protections with these services.

macOS Logins (DONE)
macOS Screensavers (DONE)
1Password (DONE)
Dropbox (DONE)
Twitter (DONE)
Google (DONE)
Google GSuite (DONE, WAITING TO VERIFY)
Google GMail (DONE)
Google Analytics and AdSense (DONE)
Github (DONE)
Thunderbird Email (DONE)
Debian servers in the cloud (SSH) (DONE)
Ubuntu servers in the cloud (SSH) (DONE)
Securing WordPress (DONE)

Etc

Final Warning

Do not attempt to activate Two Factor Authentication on a system unless you…

A) Have backups of your data
B) Have backup methods of getting into your account(s)

Murphy’s Law: “Anything that can go wrong will go wrong”

You never know when a Two Factor Authentication Key may die or an Authenticator app or a Mac/PC may stop working so always have a backup method just in case.

General

General Yubico YubiKey Setup guides https://www.yubico.com/setup/

Buying a Yubico YubiKey

International visitors can buy a YubiKey from the official store here. Australian readers can buy a key locally here. I grabbed 2x YubiKey YubiKey Neo 4 (with NFC) for $50 USD (about $75 AUD) each.

This blog post will aim to show how you can set up a primary key and backup key for use on macOS and other apps to add hardware-based two-factor authentication to logins.

Authenticator Apps

You can use Google Authenticator, Yubico Authenticator or freeOTP from https://freeotp.github.io

Plugging the YubiKey into macOS Mojave

First I read this guide: https://www.yubico.com/works-with-yubikey/catalog/macos/

1) I plugged in my Yubico Neo key into my USB slot.
2) I closed the Keyboard setup window that appeared (I guess the YubiKey is a kind of a keyboard to allow inserting of challenge-response character streams into apps and websites).

3) I followed the basic troubleshooting page and confirmed that the key was being detected (yes it was.)

4) I followed this guide to test U2F functionality and this guide to test OTP functionality. Web pages and Google Chome can talk to the plugged-in YubiKey(s).

I was prompted to register a UTF deice (and create an account)

I was prompted to (insert) and touch my Yubico key.

Google Chome asked for some permissions first.

FYI: Chrome 67 is recommended to securely allow the reading of UbiKey’s from web pages. Only allow sites you trust access to your USB devices and use a modern browser.

Success, Chrome could now see my YubiKey and my device was now verified.

Technical data is available to let you know what is going on in the background. I am not going to break down how this works but Yubico has in-depth whitepapers and documentation if you are interested.

Nice

Configuring OSX

I logged into my Mac with the account that I was going to secure.

I performed a complete time machine backup before proceeding. If you lock yourself out you will need to restore OSX from a Time Machine backup.

I Read the “Using Yubico Pluggable Authentication Module (PAM) with Challenge-Response” login guide: https://www.yubico.com/wp-content/uploads/2016/07/yubico_YubiKeyMacLoginGuide_en.pdf

I downloaded the Download the YubiKey Manager

I downloaded the yubikey-manager from here so I could configure the keys to use “HMAC-SHA1 Challenge-Response”.

Oops, I downloaded the wrong tool, good to know this one exists though.

I will update what this tool does in future (update firmware?)

I Downloaded the Yubikey Personalization Tool

I went back to the Yubico download page and downloaded the Personalization tool.

Many options are available here.

It’s time to configure a primary and backup (duplicate YubiKey) for use with macOS etc.

Enable Challenge-Response

I opened the YubiKey Personalization Tool, Inserted my primary key, clicked the Settings tab, and in the Logging Settings group, selected Log configuration output and Yubico format.

I then clicked on the Challenge Response Tab, clicked the HMAC-SHA1 button, selected Configuration Slot 2, ticked “Program Multiple YubiKeys“, changed the “Parameter Generation Scheme = Same for all Keys“, Selected “Fixed 64 byte input” under “HMAC-SHA1 Parameters” and generated a new key (wrote it down).

Under “Configuration Protection” then I selected Enable Protection” I then visited here and generated a 6 digit string to convert to hex array (with spaces (e.g: “70 61 73 73 77 64”)).

Warning: If you set an access code and later forget it, you cannot make any programming changes to this YubiKey. You would need to buy another YubiKey.

I clicked on Write Configuration

If you chose Configuration Slot 1 you will receive a warning about not saving over Configuration Slot 1 due to Yubico VIP/Symantec, I personally do not trust Symantec or the https://vip.symantec.com/ service due to Symantec issuing non-compliant certificates for use on websites. Yubico allows you to swap configuration slots if want to keep the configuration data.

On the output of the first write, I was prompted to save a file. I saved this to “secretkey.csv” onto the Desktop.

When the write to my primary key was successful, I ejected it then inserted my backup key and wrote the same configuration data to it too (on Configuration Slot 2).

Testing the HMAX-SHA1 Challenge

I open the YubiKey Personalization Tool, then click the Tools tab and click Challenge Response. Choose Configuration Slot 2, I selected HMAC-SHA1. I typed a sample input challenge (e.g “hello world”) and clicked Perform.

I noticed the Yubico key touch panel was flashing. I pressed the button, then a response appeared below the input textbox. I copied this response text then insert your second key and perform the same test so I could compare the responses (they should be the same). They were.

If the responses don’t match rewrite the configuration to your primary and secondary keys and ensure the same key and secret was used for both keys.

FYI: I rewrote configuration a few times until I got it right.

Installing the Pluggable Authentication Module (PAM) on macOS

I re-read the Mac login guide here as I don’t want to lock myself out of my Mac.

I opened the Yubico Software Download page here and clicked Computer Login Tools and downloaded the PAM for Mac.

I installed the PAM package and verified the package installation with this command.

ls -al /usr/local/lib/security

Output:

Text Output:

> drwxr-xr-x 3 root wheel 96 9 Oct 10:29 .

> drwxrwxr-x 74 simon admin 2368 9 Oct 10:29 ..

> -rwxr-xr-x 1 root wheel 143172 20 Apr 21:13 pam_yubico.so

Backup macOS

Again I ensured my Mac was backed up with Time Machine.

I logged in to my Mac with the account I wanted to be protected with the Yubico YubiKeys.

I ran the following command in terminal

mkdir –m0700 –p ~/.yubico

I double checked that my Yubico key(s) were set up for challenge response (above).

I inserted my Uubico key and ran this command

ykpamcfg -2

Feel free to read the “ykpamcfg” manual here. The yubico-pam source code is located here.

Output:

The contents of “/Users/simon/.yubico/challenge-#######” looked like (I replaced 232 random chars with #’s below). The filename ended with my keys serial number.

v2:########################################################################################################################################################################################################################################:10000:2

Next, I was supposed to copy the challenge output from ykpamcfg to /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER] with this command..

sudo cp /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER] /Users/[USERNAME]/.yubico

But I had this error.

No such file or directory

Weird as the source file existed?? macOS issues?

I Opened /Users/[USERNAME]/.yubico/challenge-[YUBIKEY SERIAL NUMBER] in the nano editor (sudo elevated process) and saved the file to /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER].

I reopened my terminal and verified the contents of /var/root/.yubico/challenge-[YUBIKEY SERIAL NUMBER]. The file is now there.

Permissions on the file is “-rw——-“. Good.

I inserted my second backuP key and re-ran “ykpamcfg -2” and copied the file to “/Users/simon/.yubico”

I verified the file contents

sudo cd /var/root/.yubico/
ls -al

Output

Text Output:

> drwxr-xr-x 4 root wheel 128 9 Oct 09:50 .
> drwxr-x— 12 root wheel 384 9 Oct 09:39 ..
> -rw-r–r– 1 root wheel 244 9 Oct 09:50 challenge-#######
> -rw-r–r– 1 root wheel 244 9 Oct 09:42 challenge-#######

Snip from: https://www.yubico.com/wp-content/uploads/2016/07/yubico_YubiKeyMacLoginGuide_en.pdf

“Program at least two YubiKeys when implementing a requirement for authentication with a YubiKey on your Mac. If you configure only one YubiKey and something happens to the YubiKey, you must restore the Mac from a Time Machine backup that you created before editing the authorization file before you can log back in to your account. ”

Reading the guide regarding multiple accounts (setting up a Key for each login). I have 5 logins on my Mac but when this works I will disable the other accounts from logging in.

Enable the use of the Yubico key when the screensaver is deactivated on macOS

I opened a terminal and edited “/etc/pam.d/screensaver ” (I use the easier nano editor)

sudo nano /etc/pam.d/screensaver

I added this line

auth       required       /usr/local/lib/security/pam_yubico.so mode=challenge-response

auth[7 spaces]required[7 spaces]/usr/local/lib/security/pam_yubico.so mode=challenge-response

I saved the file ( [CTRL+O], [CTRL+X] ) and exited nano.

I tested my screensaver and no extra protection was provided (the screensaver just exited).

I rebooted, still no change?

I reinstalled the PAM module.

Silly me, I needed to enable the password on the screensaver to then activate the /etc/pam.d/screensaver entries.

I enabled the screensaver passwords

I am now prompted to enter my password and inset and tap my Yubico Key on screensaver exit (on both keys). Awesome.

Next, I need to enable this at macOS login.

Enable the use of the Yubico key at macOS Login

I edited /etc/pam.d/authorization file with nano in the terminal

sudo nano /etc/pam.d/authorization

I added the same line as was added to the file /etc/pam.d/screensaver

auth       required       /usr/local/lib/security/pam_yubico.so mode=challenge-response

auth[7 spaces]required[7 spaces]/usr/local/lib/security/pam_yubico.so mode=challenge-response

I saved the file ( [CTRL+O], [CTRL+X] ) and exited nano.

Now let’s log out and test this.

It’s working.

Excellent

Add Two Factor Authentication to 1Password

Here is a guide on using the Yubico YubiKey with 1Password. This directed me to https://support.1password.com/yubikey/

I downloaded the Yubico Authenticator app on macOS and installed it.

After I inserted my primary Key I received a “No Credentials Found”message.

I logged into https://my.1password.com/signin and clicked My Profile.

I clicked More Actions then Turn On Two-Factor Authentication

I added the generated QR code details to the Android Authenticator and macOS Yubico Authenticator app. At first, I could not scan the QR code in macOS (was Mojave blocking this?), I manually entered the details (after confirming them from the Android app QR code scan).

Details:

Issuer: 1Password
Account Name: my.1password.com
Secret Key: ###################
Time: 30
Algorithm: SHA-1
Period: 30
Digits: 6

Now, 1Password web and the desktop app are asking for the 2-factor code (generated in the Yubico Authenticator app after I insert my YubioKey).

Nice

I logged off and I was not prompted for my Two Factor code?

Snip from: https://support.1password.com/two-factor-authentication/

“Your 1Password account is now protected by two-factor authentication. From now on, you’ll need to enter a six-digit authentication code from your authenticator app when you sign in to 1Password on a new device.”

I logged in to 1Password from Google Chrome on Android and indeed I was prompted for a two-factor auth code form the Yubico Authenticator app (with a KubiKey inserted).

Add Two Factor Authentication to Dropbox
I read https://www.yubico.com/works-with-yubikey/catalog/dropbox-personal/. Dropbox also has setup instructions here.

I logged into Dropbox and went to Settings then Security then clicked Add next to Security Keys

I started the Wizard, entered my Dropbox password, then inserted my YubiKey.

Name the Key

I added my Primary and Backup Key(s)

I logged out and back in and no Security Key prompt?

I am using Chrome and had cleared past browsers from the Dropbox list of web browsers at https://www.dropbox.com/account/security

I discovered that I need to set the primary authentication method to Use Mobile App (My Bad, it would be nice if Dropbox set this as default after I added the keys).

I added the Dropbox QR code to the Yuboico Authenticator app

I was asked to enter a 6 digit code from my Yubico Authenticator app to verify the working link. I inserted my YubiKey into my machine to show the code.

Now Dropbox is configured 🙂

Success

I now have to insert my primary key when logging into Dropbox

I need to find a way to copy my Authenticator credentials to my Backup Key from my Primary key

Add Two Factor Authentication to Twitter

I read https://www.yubico.com/works-with-yubikey/catalog/twitter/ (Setup Instructions)

1) Login to Twitter

2) Open your Settings and Look For Security

3) Click Start

4) Enter Your Password

5) Accept and enter any SMS codes if you set up SMS Two Factor codes via SMS

6) Click “Review your login verification methods”

7) Click “Setup Key”

8) Insert Your YubiKey and follow the prompts to activate it.

9) Now the key will be requoted to log in to Twitter

Testing Two Factor Login to Twitter

I logged out of and back into Twitter but the SMS Two Factor Authentication method was still active?

I tried to disable the SMS method in Twitter but two factor was disabled altogether and the registered key was deleted. I re-added my key 🙁

I solved this by choosing “Choose a different verification method” when logging in then choosing “Use your security key“, Twitter then accessed my YubiKey and further login attempts used the key instead of SMS 🙂 I could use an Authenticator code but they YubiKey touch method is quicker.

Done

It would be nice if Twitter allowed multiple keys to be used to log in?

Add Two Factor Authentication to Google, Google cloud, Gsuite etc

I read https://www.yubico.com/works-with-yubikey/catalog/google-accounts (Instructions https://myaccount.google.com/).

Adding two Factor authentication details to Google was not easily accessible at Google so I Googled (lol) this https://support.yubico.com/support/solutions/articles/15000006418-using-your-yubikey-with-google

I loaded: https://myaccount.google.com/signinoptions/two-step-verification/enroll-welcome

I clicked Get Start

I clicked Choose Another Option (not SMS Two factor)

Clicked Security Key

As prompted I inserted my key and allowed access to it.

I named the Key

I repeated the steps and added my 2nd key.

Done

I logged out my https://myaccount.google.com and logged back in and I was prompted to insert my YubiKey

Nice

I did try and login to my google GSuite account at https://admin.google.com but it did not prompt me to insert a key. I will do this next.

Add Two Factor Authentication to GSuite

I logged into the GSuite admin interface at https://admin.google.com/ I generated some backup codes in case I need them in the future.

I checked my main admin user account and I could see the 2 google security keys synced through from Google.

I then searched GSuite for “Two Factor” and loaded the “Enforcement” Page

I enabled “Turn On Enforcement Now”

I enabled “Only Security Keys”

I logged out and back into https://gsuite.google.com/ TWICE and no security key prompt.

Silly me: I forgot to click save at the bottom of the screen and it appears there is a 24-hour delay?

Add Two Factor Authentication to GMail

This is already done (above), GSuite email takes up to 24 hours to become active, GMail is instant.

Add Two Factor Authentication to Google Analytics

I can’t see an option to turn Two Factor Auth on in Google Analytics 🙂

I did send feedback to the Google Analytics team.

Add Two Factor Authentication to Google Adsense

I can’t see an option to turn Two Factor Auth on in Google Adsense either 🙂

I did send feedback to the Google AdSense team.

Add Two Factor Authentication to Github

I logged into Github, opened my Settings and clicked Security then Enable two-factor authentication

Click Setup using an app save the recovery codes.

Open the Yubico Authenticator app (ensure you can see the QR Code in GitHub)

In the Yubico Authenticator, App click File then Scan QR Code

The GitHub details should be added to the Authenticator

Two Factor via authenticator tokens is enabled and now I can see a Keys options,

I clicked Add next to security keys then Register New Device, I gave the key a name then clicked Add.

I added both keys then I Logged out and back in and two factor was enabled by YubiKey 🙂

Add Two Factor Authentication to Debian servers in the cloud (SSH)

Read Setup two-factor authenticator protection at login on Ubuntu or Debian

Add Two Factor Authentication to Ubuntu servers in the cloud (SSH)

Read Setup two-factor authenticator protection at login on Ubuntu or Debian

YubiKey Support

There are loads of Yubico support articles here: https://support.1password.com/yubikey/

Yubico Developer Info

A GitHub repository of source code is located here: https://github.com/Yubico

Other developer related pages here

Securing WordPress

Read this guide on Securing WordPress with 2FA (YubiKey insertion or Authenticator app).

I found a good WordPress plugin to handle 2FA logn methods.

I am prompted to insert my YubiKey after logging into WordPress.

Nice

Java Code to use the Yubico YubiKey in software (challenge mode)

todo: I will add this section soon.

Yubico has Java repository that contains a Java library with an accompanying demo server, as well as a JAAS module, to validate YubiKey OTP’s (One-Time Passwords).

https://developers.yubico.com/yubico-java-client/

PHP Code to use the Yubico YubiKey in software (challenge mode)

todo: I will add this section soon.

Yubico has PHP library ad source code but it has not been updated in 3 years. I cannot get this working on PHP 7.2.

https://github.com/Yubico/php-yubico

Using Yubico YubiKeys as 2fA with one-time Passwords.

The YubiKeys can be used to store and generate one time passwords.

Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc

October 3, 2018 by Simon

This is a draft post showing how you can monitor the performance of a server (or servers) with NixStats and receive alerts by SMS, Push, Email, Telegram etc

fyi: This is not a paid post, this is just me using the NixStats software to monitor my servers and send alerts.

Finding a good host

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Monitoring Servers

The post below will show you how you can monitor servers online with https://nixstats.com/ and send alerts when resources reach limits or servers fail.

I signed up and started a Nixstats (14 day free trial).

After I created an account I was emailed by Nixtsats with agent install instructions for Linux (1 line). I was also advised to add contacts and to set up alerts.

I logged into the Nixstats settings and set up…

My Timezone
Default reporting period
First name and Surname
Reporting emails
etc

Nixstats Subscription Upgrade

Subscription options

Free (5 monitors, 1 server, 24-hour data retention etc)
Founder (25 monitors, 10 servers, 30-day data retention etc)
Business (100 monitors, 15 servers, 60-day data retention etc)

I enabled the limited founder subscription so I can monitor 10x servers (this deal is too good to miss). I tried creating a status page myself last year and it is terribly hard.

I am now out of the free trial period 🙂 Let’s start monitoring many servers.

I enabled two factor Auth to Nixstats logins

I created a Nixstats API key for future use (watch this space)

I installed the Nixstats agent (the dashboard gave a 1 line command you can run as root to install the agent (on Linux)).

FYI: Command (######################## is a number linked to your account)

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################

Output

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh ########################
--2018-10-02 09:53:56--  https://www.nixstats.com/nixstatsagent.sh
Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8013, 2400:cb00:2048:1::6819:8113, 104.25.128.19, ...
Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8013|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 38708 (38K) [application/octet-stream]
Saving to: 'nixstatsagent.sh'

nixstatsagent.sh 100%[====================================================>] 37.80K –.-KB/s in 0.1s

2018-10-02 09:53:56 (338 KB/s) – ‘nixstatsagent.sh’ saved [38708/38708]

Found Ubuntu …
Installing …
Installing Python2-PIP …
Installing nixstatsagent …
Generation a server id …
Got server_id: ######################
Creating and starting service
Created symlink /etc/systemd/system/multi-user.target.wants/nixstatsagent.service -> /etc/systemd/system/nixstatsagent.service.
Created the nixstatsagent service

Server Dashboard
below is a summary of all connected servers ( https://nixstats.com/dashboard/servers ).

Monitor Setup

I set up a number of monitors to monitor ping replies and https traffic

Advanced Monitoring

I can also set the monitor credentials, timeouts, retries, auth methods, max redirects and frequency. If you server blocks login or resource GET attempts you may need to whitelist IP’s. IP’s of monitoring servers are located here https://nixstats.com/whitelist.php

Monitor Summary

The default dashboard is very informative. Feel free to create your own dashboards that focus on your own infrastructure or apps.

Individual Server Reports

You can click on a server and monitor it in detail.

Server Memory Graphs

Long-term memory graphs.

Install Optional Nixstats Plugins

Nixstats offers many plugins to monitor software that is installed on your server (e.g NGINX, MySQL, PHP etc).

1) NGINX Monitoring (Plugin)

To enable NGINX monitoring I read https://help.nixstats.com/en/article/monitoring-nginx-50nu7f/

I edited my NGINX sites-enabled config.

sudo nano /etc/nginx/sites-enabled/default

I added the following

server {
    listen 127.0.0.1:8080;
    server_name localhost;
    location /status_page {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }
}

I tested, reloaded and restarted NGINX

nginx -t
nginx -s reload
/etc/init.d/nginx restart

The status page will only be available on the local machine, I tested the page on the local machine

wget -qO- http://127.0.0.1:8080/status_nginx
Active connections: 3
server accepts handled requests
 15 15 31
Reading: 0 Writing: 1 Waiting: 2

It’s Working.

I edit /etc/nixstats.ini

sudo nano /etc/nixstats.ini

I remove comments before these lines to enable the plugin

[nginx]
enabled = yes
status_page_url = http://127.0.0.1:8080/status_nginx

I ran the following command to see if NGINX monitoring is possible

nixstatsagent --test nginx

Output

nginx:
{
    "accepts": 39,
    "accepts_per_second": 0.0,
    "active_connections": 6,
    "handled": 39,
    "handled_per_second": 0.0,
    "reading": 0,
    "requests": 119,
    "requests_per_second": 0.0,
    "waiting": 5,
    "writing": 1
}

It’s Working

I restarted the nixstatsagent

service nixstatsagent restart

I can now view NGINX properties like active_connections in my dashboard. 🙂

2) Enable PHP-FPM Monitoring (Plugin)

Looks like a PHP-FPM monitoring was recently added lets set that up too. Read my guide on setting up PHP child workers here.

We’ve added a premade dashboard for PHP-FPM. If you’re not yet monitoring PHP-FPM take a look at the integration guide https://t.co/X4ywRHw9hX pic.twitter.com/aag1fTsr3R
— Nixstats (@nixstats) September 6, 2018

To enable PHP-FPM monitoring I read https://help.nixstats.com/en/article/monitoring-php-fpm-1tlyur6/

I edited my PHP-FPM ini file

sudo nano /etc/php/7.2/fpm/pool.d/www.conf

I added the following line

pm.status_path = /status_phpfpm

Restart PHP

sudo service php7.2-fpm restart

I added the following to /etc/nginx/sites-enabled/default localhost server block added above Note I use php 7.2 below (read more here).

server {
    listen 127.0.0.1:8080;
    server_name localhost;

location /status_phpfpm {
access_log off;
allow 127.0.0.1;
deny all;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass unix:/run/php/php7.2-fpm.sock;
log_not_found off;
}
}

I tested, reloaded and restarted NGINX

nginx -t
nginx -s reload
/etc/init.d/nginx restart

I restart PHP-FPM

sudo systemctl restart php7.2-fpm

Enabled the plugin in /etc/nixstats.ini

[phpfpm]
enabled = yes
status_page_url = http://127.0.0.1:8080/status_phpfpm?json

I tested the status page

wget -qO- http://127.0.0.1:8080/status_phpfpm?json

Output:

{
	"pool":"www",
	"process manager":"static",
	"start time":1538654543,
	"start since":178,
	"accepted conn":28,
	"listen queue":0,
	"max listen queue":0,
	"listen queue len":0,
	"idle processes":49,
	"active processes":1,
	"total processes":50,
	"max active processes":2,
	"max children reached":0,
	"slow requests":0
}

I tested the agent

 nixstatsagent --test phpfpm

Output:

phpfpm:
{
    "accepted_conn": 51,
    "accepted_conn_per_second": 0.0,
    "active_processes": 1,
    "idle_processes": 49,
    "listen_queue": 0,
    "listen_queue_len": 0,
    "max_active_processes": 2,
    "max_children_reached": 0,
    "max_listen_queue": 0,
    "pool": "www",
    "process_manager": "static",
    "slow_requests": 0,
    "start_since": 318,
    "start_time": 1538654543,
    "total_processes": 50
}

I can now query PHP-FPM status values in Nixstats 🙂

Enable MySQL Monitoring (Plugin)

To enable MySQL monitoring I read https://help.nixstats.com/en/article/monitoring-mysql-1frskd8/

I edited the nixstats.ini

sudo nano /etc/nixstats.ini

I enabled the mysql section in nixstats.ini and added my mysql credentials

[mysql]
enabled=yes
username=mysqluser
password=#######################
host=127.0.0.1
database=mysql
port=3306
socket=null

I ran this command to test MySQL querying

nixstatsagent --test mysql
mysql:
Load error: No module named MySQLdb

I had an error.

A quick Google revealed I had to install a mysql python module.

sudo apt-get install python-mysqldb

Allow localhost to connect to MySQL

Edit /etc/mysql.cnf and allow all localhost and external connections (I could not bind to localhost and an external IP at the same time)

bind-address    = 0.0.0.0

TIP: Ensure you have firewalled access to your MySQL server, never open it up without protection.

Let’s try again

nixstatsagent --test mysql

Output

mysql:
{
    "aborted_clients": 0,
    "aborted_connects": 0,
    "binlog_cache_disk_use": 0,
    "binlog_cache_use": 0,
    "bytes_received": 0,
    "bytes_sent": 0,
    "com_delete": 0,
    "com_delete_multi": 0,
    "com_insert": 0,
    "com_insert_select": 0,
    "com_load": 0,
    "com_replace": 0,
    "com_replace_select": 0,
    "com_select": 0,
    "com_update": 0,
    "com_update_multi": 0,
    "connections": 0,
    "created_tmp_disk_tables": 0,
    "created_tmp_files": 0,
    "created_tmp_tables": 0,
    "key_read_requests": 0,
    "key_reads": 0,
    "key_write_requests": 0,
    "key_writes": 0,
    "max_used_connections": 3.0,
    "open_files": 14.0,
    "open_tables": 316.0,
    "opened_tables": 0,
    "qcache_free_blocks": 1.0,
    "qcache_free_memory": 16760152.0,
    "qcache_hits": 0,
    "qcache_inserts": 0,
    "qcache_lowmem_prunes": 0,
    "qcache_not_cached": 0,
    "qcache_queries_in_cache": 0,
    "qcache_total_blocks": 1.0,
    "questions": 0,
    "select_full_join": 0,
    "select_full_range_join": 0,
    "select_range": 0,
    "select_range_check": 0,
    "select_scan": 0,
    "slave_open_temp_tables": 0.0,
    "slow_launch_threads": 0,
    "slow_queries": 0,
    "sort_range": 0,
    "sort_rows": 0,
    "sort_scan": 0,
    "table_locks_immediate": 0,
    "table_locks_waited": 0,
    "threads_cached": 2.0,
    "threads_connected": 1.0,
    "threads_created": 0,
    "threads_running": 1.0,
    "uptime": 35837.0
}

Nice,

I restart MySQL

sudo systemctl restart mysql

I restart my Nixstats service

service nixstatsagent restart

Now let’s monitor MySQL in Nixstats

I can now view MySQL metrix

Status Page

Nixstats allows you to create a status page ( https://nixstats.com/pages/overview ) where you can add any servers or monitors to that page. This stats page is truly awesome, it builds a live status page based on data coming from your installed agents.

You can even set up a custom subdomain that points to a Nixstats hosted status page too (e.g https://status.yourdomain.com).

FYI: An SSL certificate on your staus page may take a few hours to set up. Don’t panic if it is not instantatly available.

Nice.

This saves doing it yourself. The status page will look like it running on your server.

Status Page

You can create a status page that automatically aggregates collected data from monitors and displays them in a nice layout.

This is great, I used to do my own status pages but not anymore.

Alerts

I added a contact so I could receive alerts. I could then add my mobile, email and PushOver key (to receive push notifications) and Telegram Bot API token.

Test Alerts

I sent a test alert to each service against the contact.

I activated a Pushover licence on my Android device for about $7.49 AUD (one off) to ensure I keep getting Push Notifications.

Nixstats have links that show you how you can create a Telegram Bot and Pushover.net account.

Pushover will cost about $5 USD one off per device (see faq).

I created the following alerts

Alert: Disk Usage higher than 90%

Alert: Load greater than 90 per cent for 1 minute

Alert: Less than 5 percent memory free.

Summary of alerts.

I also added a CPU reached 95% one for 5 mins alert too (but it’s not pictured above)

I forgot to specify alert recipients and methods for each alert so I edited each alert and added the contact.

Now it’s time to test the alerts.

I shut down a server to test alerts

shutdown -h now

Alerts to my defined Email, SMS, Push and Telegram are working a treat 🙂

After I rebooted the server I also received alerts about the server being back up.

The status page showed the server that was offline too.

Nice

Troubleshooting

I had an issue instaling the agent on Debian

I ran the following command

wget --no-check-certificate -N https://www.nixstats.com/nixstatsagent.sh && bash nixstatsagent.sh #######################
--2018-10-02 00:41:38--  https://www.nixstats.com/nixstatsagent.sh
Resolving www.nixstats.com (www.nixstats.com)... 2400:cb00:2048:1::6819:8113, 2400:cb00:2048:1::6819:8013, 104.25.129.19, ...
Connecting to www.nixstats.com (www.nixstats.com)|2400:cb00:2048:1::6819:8113|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'nixstatsagent.sh' not modified on server. Omitting download.

nixstatsagent.sh: line 508: [: Installer exited with error code 0. See nixstatsagent.log for details.: integer expression expected

An error occurred, please check the install log file (nixstatsagent.log)!

Contents of nixstatsagent.log

cat nixstatsagent.log
Ign:1 http://deb.debian.org/debian stretch InRelease
Hit:2 http://deb.debian.org/debian-security stretch/updates InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Ign:5 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic InRelease
Ign:7 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic Release
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Hit:9 https://packages.sury.org/php stretch InRelease
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Ign:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Ign:8 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main all Packages
Err:10 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main amd64 Packages
  404  Not Found
Ign:11 http://ppa.launchpad.net/ondrej/php/ubuntu cosmic/main Translation-en
Reading package lists...
W: The repository 'http://ppa.launchpad.net/ondrej/php/ubuntu cosmic Release' does not have a Release file.
E: Failed to fetch http://ppa.launchpad.net/ondrej/php/ubuntu/dists/cosmic/main/binary-amd64/Packages  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
nixstatsagent.sh: line 118: apt-get upgrade returned error code 100. Please see nixstatsagent.log for details.: command not found

I asked the Nixstats chat help and I was advised I had a dead repository (I removed this (editing the dead repo in the appropriate file in /etc/apt/) and all was ok)

I had trouble testing my Telegram alerts but it was my fault as I forgot to follow the bot account I created. Telegram does not allow message from a user (bot) unless you follow them.

A chat with the Nixstats staff sorted me out. Thanks, Nixstats chat team.

I had an issue with a missing python mysql package

Load error: No module named MySQLdb

I solved it by instaling python-mysqldb

sudo apt-get install python-mysqldb

Nixstats Help

Nixstats have a help subdomain: https://help.nixstats.com/en/

Error Logs Plugin

I did ask Nixstats on Twitter and they said they are working on a logging plugin, I can’t wait for that.

We’re launching a closed beta for Logging at Nixstats. Contact us to get setup! You can search and tail log files across all your servers! pic.twitter.com/FIeip2SOUw
— Nixstats (@nixstats) October 4, 2018

I now have access to beta log features and can see log tabs in Nixstats

I had or check the version of my rsyslogd

rsyslogd -v
rsyslogd 8.32.0, compiled with:
        PLATFORM:                               x86_64-pc-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        systemd support:                        Yes
        Number of Bits in RainerScript integers: 64

I edited: /etc/rsyslog.d/31-nixstats.conf

I pasted

##########################################################
### Rsyslog Template for Nixstats ###
##########################################################

$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down

template(name=”NixFormat” type=”string”
string=”<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [[email protected] tag=\”rsyslog\”] %msg%\n”
)

action(type=”omfwd” protocol=”tcp” target=”log.nixstats.com” port=”514″ template=”NixFormat”)
#################END CONFIG FILE#########################

I restarted the rsyslog service

sudo service rsyslog restart

Live Log Output

I can see a live log from (unknown) logs.

I can see the firewall blocking access to certain ports.

Search logs

Blacklist Checking (Beta)

Nixstats tweeted “We just launched a new blacklist check feature. Monitor your IP and hostname reputation. Free during the Beta!”

I enabled it.

I received a Blacklist notification

I requested removal at junkmailfilter.com

Thanks, Nixstats

Conclusion

This is one of the best software packages I have seen in a while. I have developed status

pages in the past (no more). Great work Nixstats.

Please check out Nixstats today, they are awesome. Signup for a free account and consider the limited time founder plan (it’s a bargain).

Nixstats Live chat support is awesome

Server Plug

If you need a server, consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.3 Blacklist beta

V1.2 Logs beta

v1.1 Logging Tweet

v1.0 Initial Post

Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..

July 28, 2018 by Simon

This post is for Telstra (to help a 10+-week investigation into Telstra NBN issues).

This post has been split up to make this page load faster: Read [Part 1] [Part 2] [Part 3] [Part 4] [Part 5]

Unresolved Issues with Telstra

Telstra promised Fibre to the premises but delivered fibre to the node (deceptive bait and switch?)
Data stability issues remain.

Status

TIO deadline (16th of August) passed 7 days ago (no resolution). Still waiting for assistance for the TIO (at least 6-week wait from the TIO ticket creation)
Telstra deadline passed (23rd August) with no resolution.
Escalating Federally (Federal cannot help until the TIO case is closed, Telstra will not close TIO case)
Telstra added a 2-week deadline to transfer the phone line (phone line back).
Have I been blacklisted by Telstra 13 22 00 support number (and reps says talk to your case manager)
Data stability issues remain

I was already with NBN (FTTN, ISP Name redacted) on a 50/20 Megabit plan and Telstra promised fibre to the house (FTTP) instead of FTTN and this is what happened (still unfolding…).

Telstra NBN (FTTN)

You can read my change over from ADSL 2 to NBN posts here. I was paying for a 50 Megabit Down and 20 Megabit up plan and it was not very after 5 months. I am not into torrenting unless downloading essential Raspberry Pi images or Linux ISO images. I needed a faster and more reliable internet connection and phone line.

My landline was down for 30 days and because I had a stroke recently (all good touch wood) a working phone line is very important. My previous provider said they can’t escalate issues unless the data line is completely down. Some days the internet was fine other days it was terrible.

I was sent a number of routers from my previous ISP to try an diagnose the dead landline and bad NBN speeds.

I was sick of the poor speeds of FTTN NBN. I was frequently talking to my previous NBN provider and had to perform daily router reboots. Some days NBN 50 Megabit seemed like being on a modem on a good day or ADSL1 on a bad day.

All support calls with my previous ISP support usually went one of two ways

We cannot escalate this issue (dead landline/slow data speeds) to NBN because your router is “syncing” (all be it slowly and with a dead landline).
We will send you a new router (mostly with experimental firmware?).

The landline stopped working and we got used to rebooting the router multiple times a day.

NBN is having troubles?

NBN woes signal boom in fixed wireless broadband internet (1 in l 6 NBN connections fail.

1st Telstra Inquiry

On the 30th June 2018, I chatted with Telstra to investigate Telstra NBN options.

General Info
Duration (actual chatting time)	00:40:54
Operator	TelstraOperator (real name redacted)

Telstra Chat Transcript (*** = “*** removed delayed replies for readability ***”. Some numbers have been pre redated by Telstra?, Telstra Typo’s not corrected.)

P.S The real name provided by the Telstra Operator has been replaced with TelstraOperator (as I promised to hide names when asking for permission to post the conversation on my blog).

fyi: Alison and myself (Simon) are married.

> Info: Thank you for choosing Telstra 24×7 Live Chat. A representative will be with you shortly. At the conclusion of your chat please take a few moments to give us some feedback on your experience today.
> Info: All agents are currently assisting others. Thank you for your patience.
> Info: You are now chatting with TelstraOperator.
> TelstraOperator: Welcome to Telstra, this is TelstraOperator one of the sales representatives. How may I help you today?
> TelstraOperator: Hello Alison how are you?
> Alison: (***Asked about switching NBN to Telstra***)
> Alison: Do you we need to leave the current provider or can we transfer?
> TelstraOperator: Could you please provide me your account /home number with complete name and Date of birth? I will quickly have a look.
> …
> Alison: (*** I provided my addresss ***)
> Alison: (*** I provided my landline ***)
>TelstraOperator: I Thought you are existing Telstra customer
> Alison: Telstra mobile but (Previous ISP Name) phone
> Alison: We want the phone and nbn to be on Telstra with mobile number
> …
> TelstraOperator: Thank you for the address, and yes while palcing the order you will get a option to enter your deatails, so our team will cancel your current broadband plan and switch to Telstra.
> Alison: Nice
> TelstraOperator: Thank you, (Please allow me a moment while i check the same in our system.)
> Alison: Ok
> Alison: I have an optus phone I am willing to switch ove too
> Alison: Over
> TelstraOperator: Greta, yes you can switch your number as well with us.
> TelstraOperator: After running the serviceability check it was found that Telstra will be able to provide you service via NBN which is the fastest fiber optic based technology we offer, and with our NBN plan you will get 50 mbps speed .
> Alison: Are you sure it’s fibre optic?
> TelstraOperator: Great! Based on the information you have provided today, I would like to offer you _ our $79 plan for ***** gb data per month + new smart modem free , and pay as you go calls, however if you want the cals so you can add our calling pack any time for $10 unlimited loclam national ,Aus mobile calls .
> TelstraOperator: And yes its a fibre optic .
> Alison: Our nbn at present is copper vdsl
> Alison: sounds great
> Alison: Do we pay extra to upgrade our house line to fibre from copper?
> TelstraOperator: Thank you, and like to
> TelstraOperator: iNFORM*
> TelstraOperator: Like to inform*
> Alison: What deals can we get with a landline phone, nbn 50/20/***** and 1 or 2 mobiles a month?
> Alison: Sounds good, thanks
> TelstraOperator: After sign up our $79 plan our team will come to your place to set up our service for you,and there is only one time fee of $99 for activtion feeand this fee will be add on your firts bill , so no need to pay any thing right now , and this fee will cover all like set up , line connection technical team visiting etc.
> TelstraOperator: And yes on $79 plan you can add mobile plan as well any time .
> …
> Alison: Can you double check please the fibre optic, is this to the house or nbn cabinet?
> TelstraOperator: Sur
> TelstraOperator: sure*
> Alison: Can I share these awesome details on my blog? So sick of (*** Previous ISP Name ***) NBN
> TelstraOperator: Thank you for waiting, as I have checked and confirm with my team as well, so yes fibre optic its availabl.
> TelstraOperator: Order Processing You will have to place an order from the Telstra website itself, I will help you with the process whenever you are ready.
> Alison: fibre to the house?
> …
> TelstraOperator: Yes , You can connect to the nbn™ network via Fibre to the Curb technology.
> TelstraOperator: Fibre.
> Alison: Oh, so it’s not fibre to the house but copper from the house to the node then fibre the exchange from the node
> TelstraOperator: Yes, but do noty worry.
> TelstraOperator: As I have inform you
> Alison: yes fibre or copper from the house? Huge differences.
> TelstraOperator: That after signup the plan our team will come to your place to do all teh set up from your streret to your house
> TelstraOperator: With a fibre .
> Alison: Sorry to be a pain and ask for clarification but it’s copper from the house to the node not fibre. Answers being blogged here for other to read https://fearby.com/article/upgrade-adsl2-fttn-nbn-national-broadband-network-australia-journey/
> TelstraOperator: I understand it’s been an inconvenience and I’m sorry., but let me explain .
> Alison: Not sure it’s legal for you to say I can get fibre nbn when it’s copper to the node?
> TelstraOperator: As I have inform that our team will set up the service from your street to your home, it means our team will do the new cabling from your strtyee to your home, of fibre .
> Alison: FttN or FttC
> TelstraOperator: For thaat reason only there is one time fee of $99 .
> Alison: Awesome
> Alison: Do we pay for the fibre from our house to the street? Our driveway is 80m long
> …
> Alison: Nice, based on your advice of new fibre and price, we will signup ASAP, do you have a reference number we can use to lock this in?
> TelstraOperator: Thank you we can sign up the plan now as well
> TelstraOperator: And after sign u p the plan our team will call you with i 24 hours for the confirmatoin.
> TelstraOperator: Order Processing You will have to place an order from the Telstra website itself, I will help you with the process whenever you are ready.
> Alison: I need to confirm we can exit our nbn plan from the current provider first
> Alison: do you have a reference number for this chat?
> Info: The chat transcript will be sent to: [email protected] at the end of your chat.
> : ThatTelstraOperator’s fine , no worries please take your time,and once your ready so please feel free to chat back with us
> TelstraOperator: I hope I have replied all your queries successfully?
> Alison: It has, awesome news tekstea
> Alison: telstra
> TelstraOperator: Thank you,I have a small favour to ask.
> TelstraOperator: It has been a pleasure serving you, please take a few moments to give us some feedback on your experience today. Please Initiate the End Chat Session Now, by clicking the (+) sign, and Select the ‘End Chat Conversation’ and you can also get a copy of our chat transcript by selecting ‘Email Transcript’ button.
> TelstraOperator: Thank you,I am glad I was able to assist. Is there anything else I could help you with today?
> Alison: Yep, can I share this info on my personal blog? happy to blur out names.
> TelstraOperator: yes of cource you camn
> Info: The chat transcript will be sent to: [email protected] at the end of your chat.
> TelstraOperator: can*
> …

Summary

TelstraOperator: Great! Based on the information you have provided today, I would like to offer you _ our $79 plan for ***** gb data per month + new smart modem free , and pay as you go calls, however if you want the cals so you can add our calling pack any time for $10 unlimited loclam national ,Aus mobile calls .
TelstraOperator: And yes its a fibre optic .
Alison: Do we pay extra to upgrade our house line to fibre from copper?
Alison: Can you double check please the fibre optic, is this to the house or nbn cabinet?
TelstraOperator: Thank you for waiting, as I have checked and confirm with my team as well, so yes fibre optic its availabl.
Alison: fibre to the house?
TelstraOperator: As I have inform that our team will set up the service from your street to your home, it means our team will do the new cabling from your strtyee to your home, of fibre .
Alison: Do we pay for the fibre from our house to the street? Our driveway is 80m long
Alison: Nice, based on your advice of new fibre and price, we will signup ASAP, do you have a reference number we can use to lock this in?

I was left thinking that I was being offered fibre to the house (FTTP – fibre laid up my driveway).

Telstra promise of Fibre (FTTN)?

Street to the house cabling (Fibre).

Fibre to the house sounds awesome.

Exiting my contact with my previous ISP

I asked my previous ISP to release me from the contract I was in (given the ongoing issues)

My previous ISP called and said that I can leave the contract 🙂

2nd Telstra Inquiry (and switchover to FTTN)

On the 3rd of July 2018, I contacted Telstra Chat to confirm fibre again and to switch over if it was fibre.

General Info
Duration (actual chatting time)	00:47:30
Operator	TelstraOperator (real name redacted)

Telstra Chat Transcript (*** = “*** removed delayed replies for readability ***”. Some numbers have been pre redated by Telstra?, Telstra Typo’s not corrected.)

P.S The real name provided by the Telstra Operator has been replaced with TelstraOperator (as I promised to hide names when asking for permission to post the conversation on my blog).

> Info: Thank you for choosing Telstra 24×7 Live Chat. A representative will be with you shortly. At the conclusion of your chat please take a few moments to give us some feedback on your experience today.
> Info: You are now chatting with TelstraOperator.
> TelstraOperator: Welcome to Telstra, this is TelstraOperator one of the sales representatives. How may I help you today?
> TelstraOperator: Hello Simon , how are you?
> Simon Fearby: Hello, (Previous ISP Name Redacted) just said I can exit a 50/20 NBN contract early so I can switch to Telsta NBN (Connect Plus)
> Simon Fearby: Shall I cancel with (Previous ISP Name Redacted) then signup with Telstra to make things smoother or transfer and have Telstra pull the number etc?
> TelstraOperator: That’s good to know, and good choice of plan, Sure I Telstra sure provide teh service,and Telstra will cancel yourt current (Previous ISP Name Redacted) plan as well .
> TelstraOperator: While placing the order you will get a option to enter your (Previous ISP Name Redacted) details
> Simon Fearby: So I should signup with Telstra, enter address and number and then have Telstra pull services then call (Previous ISP Name Redacted) to ensure all is cancelled?
> Simon Fearby: Nice.
> TelstraOperator: Yes corret .
> TelstraOperator: Can I have your complete address to perform a serviceability check for you?
> Simon Fearby: I cant wait, Telstra’s service sounds great.
> Simon Fearby: (*** I provided my addresss ***)
> TelstraOperator: Thank you, in our $79 plan you will gte ***** gb data per month+ new smart modem free , no calls included , however if you want the calls so you can add our $10 calling pack for ulnimited local, natioanl ,Aus mobile calls .
> Simon Fearby: Thanks
> Simon Fearby: We dont make many calls. 🙂
> Simon Fearby: So we signup online?
> TelstraOperator: After running the serviceability check it was found that Telstra will be able to provide you service via NBN which is the fastest fiber optic based technology we offer, and with our NBN $79 plan you will gte 50 mbps speed.
> Simon Fearby: Do we pay extra for fibre to be installed?
> TelstraOperator: Once the order will be placed, soon you will receive an email and SMS about the package details. The plan/bundle will be activated in your account within 5 to 7 business working days based upon cabling work required . The necessary devices will be sent within 5-7 business working days, and if you want in urgent so while placing the order you can select connect me soon option.
> TelstraOperator: NO , There is only one time cost for $99 for activtion fee, and thsi fee will be add on your First bill so no need to pay any thing right now , and this fee will cover all like set up, line connectiojn , technical team visiting etc.
> Simon Fearby: Thanks for your information, this has made me decide switch 🙂
> TelstraOperator: Great, Order Processing You will have to place an order from the Telstra website itself, I will help you with the process whenever you are ready.
> Simon Fearby: Will do tonight.
> Simon Fearby: Thanks TelstraOperator
> TelstraOperator: Sure, but I like to suggest to go right now
> TelstraOperator: Because I will be there online for you
> Simon Fearby: go and get connected?
> TelstraOperator: So if you stuck in ay question so i can help you
> …
> TelstraOperator: So you can sign up the plan any time .
> Simon Fearby: Just to confirm we dont need to pay for the fibre optic trench form the cabinet to the house up our driveway?
> TelstraOperator: No need to pay any extra
> TelstraOperator: The $99 wil cover all.
> TelstraOperator: will *
> Simon Fearby: Im happy to switch now, just got the ok.
> TelstraOperator: Awesome, so shall we go a head to sign up the $79 plan ?
> Simon Fearby: Yep.
> TelstraOperator: Sure captain .
> TelstraOperator: https://www.telstra.com.au/broadband/plans-bundles
> Simon Fearby: Lets do this
> …
> TelstraOperator: please visit the above link to sign up the $79 p;lan
> Simon Fearby: done
> TelstraOperator: After visit the link, please enter your address.
> Simon Fearby: I can self install
> TelstraOperator: Yes
> : You have to select self install option.
> TelstraOperator: When you click ‘select’, it will take you to the online order form. You will have to fill the details and pass me the Order Reference Number that is generated after the order is placed successfully for further processing.
> Simon Fearby: doing now 🙂
> Simon Fearby: Whats “Registered Priority Assistance customer.”?
> TelstraOperator: Sure,please take your time, and once you done, please pass me the order id munnuimber for the further process, and Thank you. I am there online in case you need any help.
> TelstraOperator: We offer Priority Assistance which is a free service designed to help customers who have (or are living with someone who has) a diagnosed life threatening medical condition and whose life may be at risk without access to a fully operational phone service. Eligible customers can register for Priority Assistance and we will install a Priority Assistance customer’s first home phone line at their address as quickly as possible.
> Simon Fearby: just confirming phone bundle
> TelstraOperator: Yes
> Simon Fearby: can i select the $10 unlimited calls option later?
> TelstraOperator: yes of cource
> Simon Fearby: “Yes, with another service provider and I want to switch to Telstra” = (Previous ISP Name Redacted)?
> TelstraOperator: you can add the add on stuiff any time .
> TelstraOperator: Stuff*
> Simon Fearby: nice
> TelstraOperator: Yes correct ,and enter your (Previous ISP Name Redacted) details.
> Simon Fearby: “Connection Type” = Cable Other?
> TelstraOperator: please select other .
> TelstraOperator: As part of your service, your name, address and phone number will be published in the printed and online White Pages directories and available from Directory Assistance. If you don’t want your details published, we offer a private number service called Silent Line free per month. We’ll also block your number so that when you call others they won’t be able to see your number. It’s important to know that a Silent Line alone won’t stop all telemarketing or unwanted calls. Would you like a silent Line?
> TelstraOperator: As part of the Telstra family it’s important to know all the best ways you can make changes and view your account online. Your first stop for all things related to your account is My Account found at: https://www.my.telstra.com.au. Here you can view your accounts, current invoices and make change requests to your accoun
> TelstraOperator: The whole process takes anywhere from 5 to 7 business days to get you connected after you sign-up.
> TelstraOperator: You won’t be charged for any excess broadband usage, however your speed will be shaped to 256 kbps for that month or until you do a data top-up.
> TelstraOperator: Your Telstra Technician will call when they’re on their way. Please allow around 4 to 5 hours for your appointment. You must have : an authorised representative on your account on site and available during your appointment(s), access to all work areas (including any required work permits), access to any service configuration emails that we’ve sent you, access to any relevant usernames and passwords. If your installation requires any non-standard services or equipment, your technician will discuss your options and let you know any extra costs before commencing work.
> TelstraOperator: We offer the option to connect to the nbn network if you’re confident with installing your own devices and if your home set-up is fairly simple, this would be the ‘Self Install’ option. If a standard Professional Installation/ tech visit is required, there will be a charge of $240 (one off). The $240/standard Professional Installation/ tech visit fee is available on a 12 month service repayment option. Additional costs will apply if you have complex cabling requirement
> Simon Fearby: DONE.
> TelstraOperator: There must be a working phone line/cable in the property for Telstra to provide services. If cabling is required, additional charges may incur.
> TelstraOperator: https://www.telstra.com/content/dam/tcom/personal/help/pdf/cis-personal/bundles-c/personal-critical-information-summary-telstra-plans-bundles-MOSC2160.pdf
> TelstraOperator: The above link its for our $79 Critiacl info summary , so you can go throuigh it any time .
> Simon Fearby: how do i pay?
> TelstraOperator: you will get a bil after 30 days .
> Simon Fearby: What about $99 setup fee?
> TelstraOperator: It will add on your First bill.
> TelstraOperator: And from second montyh it will be $79 per month .
> Simon Fearby: Nice.
> Simon Fearby: When will the fibre be connected to the house and router turn up?
> TelstraOperator: After sig up the plan our team will come to your place to set up our service for you, and our team will call you with in 24 hours for the confirmation.
> Simon Fearby: Thanks
> TelstraOperator: Thank you, once you done please pass me the order number /
> Simon Fearby: Today?
> Simon Fearby: So I check email?
> Simon Fearby: ########
> Simon Fearby: What’s next? We’ll process your request within 24-48 hours and contact you to clarify details of your order if required. Your order Order reference number: NA########
> TelstraOperator: NO , OUR team will create a pass word them you can checked the email .
> Simon Fearby: thanks, so that is?
> TelstraOperator: Fantastic,thank you for the order id number, soon you will receive an email and SMS about the package details. and one call fro our team with in 24 hours for the confirmation.
> …
> TelstraOperator: Just to summarise what you have purchased today our $79 copnnet plus plan with NBN
> TelstraOperator: My account As part of the Telstra family it’s important to know all the best ways you can make changes and view your account online. Your first stop for all things related to your account is My Account found at: https://www.my.telstra.com.au. Here you can view your accounts, current invoices and make change requests to your account.
> TelstraOperator: CrowdSupport If you have further questions about your services in the future, I also recommend visiting our crowdsupport page at: https://crowdsupport.telstra.com.au/. This is a digital community providing support and answers to a list of commonly asked questions by our customers. Have a browse and ask a question, there are no bad questions.
> Simon Fearby: Yes, with pay as you go phone.
> TelstraOperator: I hope I have replied all your queries successfully?
> TelstraOperator: Yes correct .
> Simon Fearby: Thanks, can I tell my friends on blog how awesome Telstra are?
> Simon Fearby: https://fearby.com
> TelstraOperator: YES please .
> Simon Fearby: Thanks
> …
> TelstraOperator: It has been a pleasure serving you, please take a few moments to give us some feedback on your experience today. Please Initiate the End Chat Session Now, by clicking the (+) sign, and Select the ‘End Chat Conversation’ and you can also get a copy of our chat transcript by selecting ‘Email Transcript’ button.
> …
> TelstraOperator: Thank you for visiting Telstra, have a nice day.

Summary

> TelstraOperator: After running the serviceability check it was found that Telstra will be able to provide you service via NBN which is the fastest fiber optic based technology we offer, and with our NBN $79 plan you will gte 50 mbps speed.
> Simon Fearby: Do we pay extra for fibre to be installed?
> Simon Fearby: Just to confirm we dont need to pay for the fibre optic trench form the cabinet to the house up our driveway?
> Simon Fearby: When will the fibre be connected to the house and router turn up?

Again I was expecting Fibre to the house.

Continue onto Part 2

WiFi Research

I found this Telstra Wifi Page:
https://www.telstra.com.au/broadband/extras/getwifi

I Downloaded the Telstra Home Dashboard App https://play.google.com/store/apps/details?id=com.telstra.wifidiag

House Plan

My Wifi Speeds

Related Links

ACCC report on NBN speed show some customers aren’t getting what they paid for.
NBN Co considering mechanisms to slow down heavy NBN users (Gamers)? Will Netflix users be next? This sounds like modem dial-up days and low quotas again 🙁
Better Internet For Rural, Regional And Remote Australia (BIRRR) volunteers helping fix NBN connections – IT Wire
NBN backflips on 1 gigabit speed promise
Telstra’s full-year profit falls 8pc to $3.6 billion as NBN rollout erodes business
Telstra offers to compensate 42,000 customers for slow NBN speeds
NBN came in like a wrecking ball, and worse is to come
TIO Complaint Handling Procedure
Telstra expects $300m income hit from NBN delays
NBN Co’s build costs rise $2bn, 1.2m premises delayed

Revision History

V8.2 TIO case closed.

V8.1 upload limit removed by Telstra Specialist.

V8.0 Telstra confirmed that NBN visited last week and fixed a fault. Also 14Mb upload OK ok.

V8.0 slow upload speeds remain

V7.9 slow upload speed

V7.8 unscheduled technician visit while I was out.

V7.7 third plan restart?

V7.6 remote fix scheduled.

V7.5 technician visit booked.

V7.4 Update 4th October 2018.

V7.3 the TIO called and asked that the remaining issue is? (I said dropouts, slow uploads and fluctuating speeds). New Telstra case manager will be assigned.

V7.2 slow speeds

V7.1 Upload speeds now 14Mb (20% lower)

V7.0 Created Part 5 (Part 4 was too slow to edit on mobile)

V6.9 the Internet is going down a lot tonight, 6x already.

V6.8 NBN went down, I called Telstra and the automated voice said they can’t find a NBN plan attached to our landline.

V6.7 data slow again?

V6.6 landline back, data stable, no FTTP. Case over?

V6.5 landline and data are back but no services are listed in our account.

V6.4 Telstra landline is still dead, data-poor. Talked to Teksta and they cancelled our contract and will reconnect in 6 days.

V6.5 silent case manager.

V6.5 reconnection date changed to 27/9/2018.

V6.4 reconnection date changed to 1/10/2018

V6.3 dead landline still.

V6.2 dead landline and unstable speeds again.

V6.1 router is working video, why send a tech.

V6.0 Local Telstra technician tried to visit (Telstra did not call him to cancel the call). Again Telstra need a working CRM.

V5.9 I requested a new case manager.

V5.8 Telstra cancelled todays technician visit (news to me), Telstra rescheduled to next month.

V5.7 Telstra CEO blocked me.

V5.6 I advised Telstra of the days I will be home for NBN to visit again. Waiting for confirmation.

V5.5 TIO said they cant help for a few more weeks, Telstra NBN visit appointment issues.

V5.4 Being booted of NBN so they can reconnect us with the old number.

V5.3 Mandatory Telstra visit required to port a landline.

V5.2 Telstra replied to the query about $40/mobthly landline BILL (needed to port our landline)

V5.1 TIO office and asked Telstra to send a bill.

V5.0 Re-adding advertisements to this blog post to help pay for my mobile data.

V4.9 Splitting the guide into 4 parts (it is too long and I can’t edit on my phone).

V4.8 CEO comments

V4.7 microchip details invalid.

V4.6 two more weeks to wait.

V4.5 two months on

V4.4 deadlines passed, escalating.

V4.3 FTTN running very slowly.

V4.2 my desire for FTTP is my fault?

V4.1 Error porting number.

V4.0 50/20 FTTN is back up.

V3.9 replacement router received.

V3.8 Donated $10 USD to the Electronic Frontier Foundation – The leading nonprofit defending digital privacy, free speech, and innovation (to cover any revenue gained from an ad’s in the post sidebar (no ads are in the header or content).

V3.7 Telstra CEO replied on Twitter.

V3.6 TIO added link to Complaint Handling Procedure.

V3.5 mobile quota update.

V3.4 Weekend with no Internet or phone.

V3.3 dead router, no internet or phone.

V3.2 todo factory reset router.

V3.1 No outages nearby

V3.0 No Internet.

V2.9 TIO resolution time expired.

V2.8 TIO complaint update.

V2.7 no reply from Telstra today, added profit fall link.

V2.6 early update on the final day of TIO complaint

V2.5 3pm speed test added.

V2.4 Telstra called.

v2.3 Added WiFi MAp and Telstra call details.

V2.2 added 100/40 FTTN 6am speed test

V2.1 added 100/40 FTTN night speeds.

V2.0 added 100 Mbit FTTN

V1.9 Telstra were in touch.

V1.8 added lunchtime speed test.

V1.7 added backflip link on 1gb speeds

v1.6 Instability and dropouts tonight

v1.5 Complaint is back to the Executive Customer Contact Team 🙁

v1.4 NBN Co to slow down heavy NBN users?

v1.3 added requested speed test

v1.2 speed test falling well below 50Mb.

v1.1 Telstra received my TIO complaint.

v1.0 Escalated to ACCC and Telecommunication Ombudsman

v0.99 update 2nd August 2018

v0.98 added Links section

v0.97 Escalated to Telstra “Escalated Complaints Group”

v0.96 Next update in 2 days.

v0.95 escalated to Executive Complaints Team

v0.94 added Telstra link

v0.93 added 2nd phone update.

v0.92 added update re phone line.

v0.91 added nbn issues article

v0.9 Draft Post

Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins

July 23, 2018 by Simon

This is a quick post that shows how I upgraded to Wordfence Premium to get real-time defence feeds, malware scanner and two-factor authentication for WordPress logins

Aside

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Now on with the post.

What is Wordfence

WordFence is a free WordPress plugin (install guide here) that helps protect your WordPress site by logging and blocking bad events. I was a big fan of the Wordfence sister program called GravityScan (before it was retired)

Read my review of the free Wordfence plugin here.

I was using Wordfence free to

Whitelist logins for known IP’s (read my guide on whitelisting IPs here)
Block known bad IPs from the Wordfence global network (but with a 30-day delay)
Create a firewall
Rate limiting page requests
Scan my site for malware
Ability to see past failed logins (and ban them)
Ability to block/ban users who try and login form new IP’s
Force strong WordPress account passwords
Set ban thresholds
Have I Been Pwned breached password checks
Much more

Install and set up Wordfence (Free)

Read my guide here to learn how to setup Wordfence (Free).

Malware Infections

Your website is often scanned and ranked for safety by sites like Norton Safe Web, Google, Trend Micro, Kaspersky Virus Desk, SiteGuarding etc along with search engines. Having malicious files on your site will affect your site Search EnginOptimizationio (SEO).

I had a 5-year-old scan of a subdomain (that was hosted on a CPanel Host). The subdomain had false positives for malware.

Working to remove the false positive was a lengthy process.

You should aim to stay off the radar or many site scanning, check VirusTotal often to keep your self-updated as to the status of your website. Wordfence will hopefully detect real malware issues automatically in the future.

https://sitecheck.sucuri.net/ is a good site that can aggregate your sites safety ratings.

WordfFence Free v Premium

Wordfence Premium

Prices (USD)

WordFence Premium

Read about some benefits of Wordfence Premium here.

Real-time firewall rules and malware signatures
Global Wordfence premium IP blacklist
Priority server processing for premium customers
Two Factor Authentication (only if you don’t use whitelisting I found out)

Buying a Wordfence Premium API Key

Login to https://www.wordfence.com/dashboard/
Click Buy More API Keys
Enter your Payment Details

>Thanks, your card information has been updated. You can now go to your API Key Manager and create and manage your Wordfence API keys.

Now you can buy an API key and copy and paste the API ey o to your Wordfence plugin.

Wordfence Firewall

Wordfence does a great job at showing failed/successful, top blocked IP’s

Wordfence Malware Scanner

Wordfence premium has schedulable scans with real-time malware signatures

Scan Progress

Testing the scanner

Wordfence says “A Wordfence scan examines all files on your WordPress website looking for malicious code, backdoors, shells that hackers have installed, known malicious URLs and known patterns of infections.”

I created an eicar.txt test file (information on eicar here (slightly modified so I don’t get tagged again b virus scanners)) to test the Wordfence malware scanner

echo 'X5O!P#removed#X54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' > /www-root/eicar.txt
sudo chown www-data:www-data eicar.txt

I enable scanning of files outside of WordPress

I rescanned my site with Wordfence

Result: Nothing??

I logged a support ticket to see if this is right?

Update: Wordfence support replied and said “Thanks for writing in. We do detect the EICAR test file, but scans don’t scan file types that aren’t dangerous on a site by default, since scans would waste a lot of time on files that aren’t exploitable.“

I disagree a virus is a virus.

Wordfence says “A Wordfence scan examines all files on your WordPress website looking for malicious code, backdoors, shells that hackers have installed, known malicious URLs and known patterns of infections.”

I guess “all” does not mean “all”?

Wordfence support said EICAR files are detected if I rename the file to php. I renamed the file and to enabled “Scan images, binary, and other files as if they were executable“.

I started a new scan

> Scan Failed
>The scan has failed because we received an unexpected response from the Wordfence servers. This may be a temporary error, though some sites may need adjustments to run scans reliably

🙁

I scanned my system with ClamAV and it found the EICAR file.

clamscan -r --bell -i /www-root

Result:

/www-root/eicar.txt: Eicar-Test-Signature FOUND

ClamAV found the virus.

Setting up Two Factor Authentication (work in progress)

Add your desired user and number

Click Enable User

Wait for the text message and activation code (on your phone)

Enter the activation code and press Activate

The two-factor authentication should be activated

List of two-factor authorization enabled users.

I logged out of WordPress and logged back in but the two-factor auth did not work, I logged a support Ticket with my theme maker and WordFence.

Update: Wordfence Support “Wordfence > Tools > Two Factor Authentication options there is an option for Enable Separate Prompt for Two Factor Code which you could disable and try.“

This fix did not work. I sent a 2nd diagnostics report to Wordfence.

Wordfence support said

>When our two-factor authentication feature allows you to login bypassing the need to enter the authentication code it is typically because of these possible reasons:

> 1) The user has whitelisted their IP address in the advanced firewall option “Whitelisted IP addresses that bypass all rules“.

>2) Another plugin, or possibly a theme, that creates non-standard WordPress behaviour such as user role and capabilities modification, or that modifies the login flow process in some way.

It appears my IP whitelist was disabling the two-factor auth feature 🙁

I’d rather keep the two-factor auth along with keeping the whitelist (just in case my whitelist IP is known and used).

Refund

I asked Wordfence for a refund (given)

Conclusion

Pros

Protects and blocks bad logins
Real-time blocked IP and malware feeds

Cons

Almost $140 Australian dollars a year
A scan does not detect eicar.txt test virus files (ticked logged), renamed to eicar.php and still no luck.
Two-factor auth (authenticator and SMS) does not work (ticket logged)
Wordfence support resolve/close support tickets with no confirmation from the user.
Two Factor Auth is disabled if you whitelist IPs 🙁

Is Premium worth it? Yes if you want “Real-time firewall rules and malware signatures” (and don’t whitelist your IP).

I hope this guide helps someone.

Please consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.4 Updated conclusion and Wordfence refund

v1.3 added whitelist 2FA info

v1.2 added replied from Wordfence support re EICAR and Two Factor Auth.

v1.1 Added Pros and Cons section

v1.0 Initial Post

Setup a dedicated Debian subdomain (VM), Install MySQL 14 and connect to it from a WordPress on a different VM

July 21, 2018 by Simon

This is how I set up a dedicated Debian subdomain (VM), Installed MySQL 14 and connected to it from a WordPress installation on a different VM

Aside

Buy a domain name here

Now on with the post.

Fearby.com

I will be honest, fearby.com is my play server where I can code, learn about InfoSec and share (It’s also my stroke rehab blog).

There is no faster way to learn than actually doing. The problem is my “doing” usually breaks the live site from time to time (sorry).

I really need to set up a testing environment (DEV-TEST-LIVE or GREEN-BLUE) server(s). GREEN-BLUE has advantages as I can always have a hot spare ready. All I need to do is toggle DNS and I can set the GREEN or BLUE server as the live server.

But first I need to separate my database from my current fearby.com server and setup up a new web server. Having a Green and Blue server that uses one database server will help with near real-time production website switches.

Dedicated Database Server

I read the following ( Should MySQL and Web Server share the same server? ) at Percona Database Performance Blog. Having a separate database server should not negatively impact performance (It may even help improve speeds).

Deploy a Debian VM (not Ubuntu)

I decided to set up a Debian server instead of Ubuntu (mostly because of the good focus on stability and focus on security within Debian).

I logged into the UpCloud dashboard on my mobile phone and deployed a Debian server in 5 mins. I will be using my existing how to setup Ubuntu on UpCloud guide (even though this is Debian).

TIP: Sign up to UpCloud using this link to get $25 free UpCloud VM credit.

Deploy a Debian server setup steps:

Login to UpCloud and go to Create server.
Name your Server (use a fully qualified domain name)
Add a description.
Choose your data centre (Chicago for me)
Choose the server specs (1x CPU, 50GB Disk, 2GB Memory, 2TB Traffic for me)
Name the Primary disk
Choose an operating system (Debian for me)
Select an SSH Key
Choose misc settings
Click Deploy server

After 5 mins your server should be deployed.

After Deploy

Setup DNS

Login to your DNS provider and create DNS records to the new IP’s (IPv4 and IPv6) provided by UpCloud. It took DNS 12 hours to replicate to my in Australia.

Setup a Firewall (at UpCloud)

I would recommend you set up a firewall at UpCloud as soon as possible (don’t forget to add the recommended UpCloud DNS IP’s and any whitelisted IP’s your firewall).

Block everything and only allow

Port 22: Allow known IP(s) of your ISP or VPN.
Port 53: Allow known UpCloud DNS servers
Port 80 (ALL)
Port 443 (ALL)
Port 3306 Allow your WordPress site and known IP(s) of your ISP or VPN.

Read my post on setting up a whitelisted IP on an UpCloud VM… as it is a good idea.

UpCloud thankfully has a copy firewall feature that is very handy.

After I set up the firewall I SSH’ed into my server (I use vSSH on OSX buy you could use PUTTY).

I updated the Debian system with the following command

sudo apt update

Get the MySQL Package

Visit http://repo.mysql.com/ and get the URL of the latest apt-config repo deb file (e.g “mysql-apt-config_0.8.9-1_all.deb”). Make a temp folder.

mkdir /temp
cd /temp

Download the MySQL deb Package

wget http://repo.mysql.com/mysql-apt-config_0.8.9-1_all.deb

Install the package

sudo dpkg -i mysql-apt-config_0.8.9-1_all.deb

Update the system again

sudo apt update

Install MySQL on Debian

sudo apt install mysql-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libaio1 libatomic1 libmecab2 mysql-client mysql-common mysql-community-client mysql-community-server psmisc
The following NEW packages will be installed:
libaio1 libatomic1 libmecab2 mysql-client mysql-common mysql-community-client mysql-community-server mysql-server psmisc
0 upgraded, 9 newly installed, 0 to remove and 1 not upgraded.
Need to get 37.1 MB of archives.
After this operation, 256 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://repo.mysql.com/apt/debian stretch/mysql-5.7 amd64 mysql-community-client amd64 5.7.22-1debian9 [8886 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 mysql-common all 5.8+1.0.2 [5608 B]
Get:3 http://deb.debian.org/debian stretch/main amd64 libaio1 amd64 0.3.110-3 [9412 B]
Get:4 http://deb.debian.org/debian stretch/main amd64 libatomic1 amd64 6.3.0-18+deb9u1 [8966 B]
Get:5 http://deb.debian.org/debian stretch/main amd64 psmisc amd64 22.21-2.1+b2 [123 kB]
Get:6 http://deb.debian.org/debian stretch/main amd64 libmecab2 amd64 0.996-3.1 [256 kB]
Get:7 http://repo.mysql.com/apt/debian stretch/mysql-5.7 amd64 mysql-client amd64 5.7.22-1debian9 [12.4 kB]
Get:8 http://repo.mysql.com/apt/debian stretch/mysql-5.7 amd64 mysql-community-server amd64 5.7.22-1debian9 [27.8 MB]
Get:9 http://repo.mysql.com/apt/debian stretch/mysql-5.7 amd64 mysql-server amd64 5.7.22-1debian9 [12.4 kB]
Fetched 37.1 MB in 12s (3023 kB/s)
Preconfiguring packages ...
Selecting previously unselected package mysql-common.
(Reading database ... 34750 files and directories currently installed.)
Preparing to unpack .../0-mysql-common_5.8+1.0.2_all.deb ...
Unpacking mysql-common (5.8+1.0.2) ...
Selecting previously unselected package libaio1:amd64.
Preparing to unpack .../1-libaio1_0.3.110-3_amd64.deb ...
Unpacking libaio1:amd64 (0.3.110-3) ...
Selecting previously unselected package libatomic1:amd64.
Preparing to unpack .../2-libatomic1_6.3.0-18+deb9u1_amd64.deb ...
Unpacking libatomic1:amd64 (6.3.0-18+deb9u1) ...
Selecting previously unselected package mysql-community-client.
Preparing to unpack .../3-mysql-community-client_5.7.22-1debian9_amd64.deb ...
Unpacking mysql-community-client (5.7.22-1debian9) ...
Selecting previously unselected package mysql-client.
Preparing to unpack .../4-mysql-client_5.7.22-1debian9_amd64.deb ...
Unpacking mysql-client (5.7.22-1debian9) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../5-psmisc_22.21-2.1+b2_amd64.deb ...
Unpacking psmisc (22.21-2.1+b2) ...
Selecting previously unselected package libmecab2:amd64.
Preparing to unpack .../6-libmecab2_0.996-3.1_amd64.deb ...
Unpacking libmecab2:amd64 (0.996-3.1) ...
Selecting previously unselected package mysql-community-server.
Preparing to unpack .../7-mysql-community-server_5.7.22-1debian9_amd64.deb ...
Unpacking mysql-community-server (5.7.22-1debian9) ...
Selecting previously unselected package mysql-server.
Preparing to unpack .../8-mysql-server_5.7.22-1debian9_amd64.deb ...
Unpacking mysql-server (5.7.22-1debian9) ...
Setting up libatomic1:amd64 (6.3.0-18+deb9u1) ...
Setting up psmisc (22.21-2.1+b2) ...
Setting up mysql-common (5.8+1.0.2) ...
update-alternatives: using /etc/mysql/my.cnf.fallback to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Setting up libmecab2:amd64 (0.996-3.1) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up libaio1:amd64 (0.3.110-3) ...
Processing triggers for systemd (232-25+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up mysql-community-client (5.7.22-1debian9) ...
Setting up mysql-client (5.7.22-1debian9) ...
Setting up mysql-community-server (5.7.22-1debian9) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service -> /lib/systemd/system/mysql.service.
Setting up mysql-server (5.7.22-1debian9) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Processing triggers for systemd (232-25+deb9u4) ...

Secure MySQL

You can secure the MySQL server deployment (set options as needed)

sudo mysql_secure_installation

Enter password for user root:
********************************************
VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No: No
Using existing password for root.
Change the password for root ? ((Press y|Y for Yes, any other key for No) : No

... skipping.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : Yes
Success.

Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : No

... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.

Remove test database and access to it? (Press y|Y for Yes, any other key for No) : Yes
- Dropping test database...
Success.

- Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : Yes
Success.

All done!

Install NGINX

I installed NGINX to allow Adminer MySQL GUI to be used

I ran these commands to install NGINX.

sudo apt update
sudo apt upgrade
sudo apt-get install nginx

I edited my NGINX configuration as required.

Set a web server root
Set desired headers
Optimized NGINX (see past guides here, here and here)

I reloaded NGINX

sudo nginx -t
sudo nginx -s reload
sudo systemctl restart nginx

Install PHP

I followed this guide to install PHP on Debian.

sudo apt update
sudo apt upgrade

sudo apt install ca-certificates apt-transport-https
wget -q https://packages.sury.org/php/apt.gpg -O- | sudo apt-key add -
echo "deb https://packages.sury.org/php/ stretch main" | sudo tee /etc/apt/sources.list.d/php.list

sudo apt update
sudo apt install php7.2
sudo apt install php-pear php7.2-curl php7.2-dev php7.2-mbstring php7.2-zip php7.2-mysql php7.2-xml php7.2-cli php7.2-common

Install PHP FPM

apt-get install php7.2-fpm

Increase Upload Limits

You may need to temporarily increase upload limits in NGINX and PHP before you can restore a WordPress database. My feabry.com blog is about 87MB.

Add “client_max_body_size 100M;” to “/etc/nginx/nginx.conf”

Add the following to “/etc/php/7.2/fpm/php.ini”

post_max_size = 100M
upload_max_filesize = 100M

Restore a backup of my MySQL database in MySQL

You can now use Adminer to restore your blog to MySQL. Read my post here on Adminer here. I used Adminer to move my WordPress site from CPanel to a self-managed server a year ago.

First login to your source server and export your desired database then login to the target server and import the database.

Firewall Check

Don’t forget to allow your WordPress site’s 2x Public IP’s and 1x Private IP to access port 3306 in your UpCloud Firewall.

How to check open ports on your current server

sudo netstat -plunt

Set MySQL Permissions

Open MySQL

mysql --host=localhost --user=root --password=***************************************************************************

I ran these statements to grant the user logging in on the nominate IP’s access to MySQL.

mysql>
GRANT ALL ON databasenmae.* TO [email protected] IDENTIFIED BY '***********sql*user*password*************';
GRANT ALL ON databasenmae.* TO [email protected] IDENTIFIED BY '***********sql*user*password*************';
GRANT ALL ON databasenmae.* TO [email protected] IDENTIFIED BY '***********sql*user*password*************';
GRANT ALL ON databasenmae.* TO [email protected] IDENTIFIED BY '***********sql*user*password*************';

Reload permissions in MySQL

FLUSH PRIVILEGES;

Allow access to the Debian machine from known IP’s

Edit “/etc/host.allow”

Additions (known safe IP’s that need access to this MySQL remotely).

mysqld : IPv4Server1PublicAddress : allow
mysqld : IPv4Server1PrivateAddress : allow
mysqld : IPv4Server2PublicAddress : allow
mysqld : IPv4Server1PrivateAddress : allow

mysqld : ALL : deny

Tell MySQL to listen on

Edit “/etc/mysql/my.cnf”

Added..

[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/English
bind-address = DebianServersIntenalIPv4Address

I guess you could change the port to something random???

Restart MySQL

sudo service mysql restart

Install a second local firewall on Debian

Install ufw

sudo apt-get instal ufw

Do add the IP of your desired server or VPN to access SSH

sudo ufw allow from 123.123.123.123 to any port 22

Do add the IP of your desired server or VPN to access WWW

sudo ufw allow from 123.123.123.123 to any port 80

Now add the known IP’s (e.g any web servers public (IPv4/IPv6) or Private IP’s) that you wish to grant access to MySQL (e.g the source website that used to have MySQL)

sudo ufw allow from 123.123.123.123 to any port 3306

Do add UpCloud DNS Servers to your firewall

sudo ufw allow from 94.237.127.9 to any port 53
sudo ufw allow from 94.237.40.9 to any port 53
sudo ufw allow from 2a04:3544:53::1 to any port 53
sudo ufw allow from 2a04:3540:53::1 to any port 53

Add all other rules as needed (if you stuff up and lock your self out you can login to the server with the Console on UpCloud)

Restart the ufw firewall

sudo ufw disable
sudo ufw enable

Prevent MySQL starting on the source server

Now we can shut down MySQL on the source server (leave it there just in case).

Edit “/etc/init/mysql.conf”

Comment out the line that contains “start on ” and save the file

and run

sudo systemctl disable mysql

Reboot

shutdown -r now

Stop and Disable NGINX on the new DB server

We don’t need NGINX running now the database has been imported with Adminer.

Stop and prevent NGINX from starting up on startup.

/etc/init.d/nginx stop
sudo update-rc.d -f nginx disable
sudo systemctl disable nginx

Check to see if MySQL is Disabled

service mysql status
* mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; disabled; vendor preset: enabled)
Active: inactive (dead)

Yep

Test access to the database server in PHP code

Add to dbtest.php

<em>SELECT guid FROM wp_posts</em>()<br />
<ul><?php

//External IP (charged after quota hit)
//$servername = 'db.yourserver.com';

//Private IP (free)
//$servername = '10.x.x.x';

$username = 'username';
$password = '********your*password*********';
$dbname = 'database';

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

$sql = 'SELECT guid FROM wp_posts';
$result = $conn->query($sql);

if ($result->num_rows > 0) {
    // output data of each row
    while($row = $result->fetch_assoc()) {
        echo $row["guid"] . "<br>";
    }
} else {
    echo "0 results";
}
$conn->close();
?></ul>
Done

Check for open ports.

You can install nmap on another server and scan for open ports

Install nmap

sudo apt-get install nmap

Scan a server for open ports with nmap

You should see this on a server that has access to see port 3306 (port 3306 should not be visible by non-whitelisted IP’s). Port 3shouldoudl not be seen via everyone.

sudo nmap -PN db.yourserver.com

Starting Nmap 7.40 ( https://nmap.org ) at 2018-07-20 14:15 UTC
Nmap scan report for db.yourserver.com (IPv4IP)
Host is up (0.0000070s latency).
Other addresses for db.yourserver.com (not scanned): IPv6IP
Not shown: 997 closed ports
PORT     STATE SERVICE
3306/tcp open  mysql

You should see something like this on a server that has access to see port 80/443 (a web server)

sudo nmap -PN yourserver.com

Starting Nmap 7.40 ( https://nmap.org ) at 2018-07-20 14:18 UTC
Nmap scan report for db.yourserver.com (IPv4IP)
Host is up (0.0000070s latency).
Other addresses for db.yourserver.com (not scanned): IPv6IP
Not shown: 997 closed ports
PORT     STATE SERVICE
80/tcp   open  http
443/tcp   open  https

I’d recommend you use a service like https://pentest-tools.com/network-vulnerability-scanning/tcp-port-scanner-online-nmap# to check for open ports. https://hackertarget.com/tcp-port-scan/ is a great tool too.

https://www.infobyip.com/tcpportchecker.php is also a free port checker that you can use to verify individual closed ports.

Hardening MySQL and Debian

Read: https://www.debian.org/doc/manuals/securing-debian-howto/ap-checklist.en.html

Configuring WordPress use the dedicated Debian VM

On the source server that used to have MySQL edit your wp-config.php file for WordPress.

Remove

define('DB_HOST', 'localhost');

add (read the update below, I changed the DNS IP to the Private IP to have free traffic)

//Oriinal localhost
//define('DB_HOST', 'localhost');

//New external host via DNS (Charged after quota hit)
//define('DB_HOST', 'db.fearby.com');

//New external host via Private IP (Free)
define('DB_HOST','10.x.x.x');

Restart NGINX

sudo nginx -t
sudo nginx -s reload
sudo systemctl restart nginx

Restart PHP-FPM

service php7.2-fpm restart

Conclusion

Nice, I seem to have shaved off 0.3 seconds in load times (25% improvement)

Update: Using a Private IP or Public IP between WordPress and MySQL servers

After I released this blog post (version 1.0 with no help from UpCloud) UpCloud contacted me and said the following.

Hello Simon,

I notice there's no mention of using the private network IPs. Did you know that we automagically assign you one when you deploy with our templates. The private network works out of the box without additional configuration, you can use that communicate between your own cloud servers and even across datacentres.

There's no bandwidth charge when communicating over private network, they do not go through public internet as well. With this, you can easily build high redundant setups.

Let me know if you have any other questions.

--
Kelvin from UpCloud

I will have updated my references in this post and replace the public IP address (that is linked to DNS record for db.fearby.com) and instead use the private ip address (e.g 10.x.x.x), your servers private IP address is listed against the public IPv$ and IPv6 address.

I checked that the local ufw firewall did indeed allow the private IP access to MySQL.

sudo ufw status numbered |grep 10.x.x.x
[27] 3306                       ALLOW IN    10.x.x.x

On my new Debian MySQL server, I edited the file /etc/mysql/my.cnf and changed the IP to the private IP and not the public IP.

Now it looked like

[mysqld]
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 3306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
language        = /usr/share/mysql/English
bind-address    = 10.x.x.x

(10.x.x.x my Debian servers private IP)

On my WordPress instance, I edited the file /www-root/wp-config.php

I added the new private host

//Oriinal localhost
//define('DB_HOST', 'localhost');

//New external host via DNS (Charged after quota hit)
//define('DB_HOST', 'db.fearby.com');

//New external host via Private IP (Free)
define('DB_HOST','10.x.x.x');

(10.x.x.x my Debian servers private IP)

Alos on Debian/MySQL ensure you have granted access to the private IP of the WordPress server

Edit /etc/host.allow

Add

mysqld : 10.x.x.x : allow

Restart MySQL

sudo systemctl restart mysql

TIP: Enable UpCloud Backups

Do setup automatic backups (and or take manual backups). Backups are an extra charge but are essential IMHO.

Troubleshooting

If you can’t access MySQL log back into MySQL

mysql --host=localhost --user=root --password=***************************************************************************

and run

GRANT ALL PRIVILEGES ON *.* TO [email protected]'%' IDENTIFIED BY '***********sql*user*password*************''; FLUSH PRIVILEGES;

Reboot

Lower Upload Limits

Don’t forget to lower file upload sizes in NGINX and PHP (e.g 2M) now that the database has been restored.

I hope this guide helps someone.

TIP: Sign up to UpCloud using this link to get $25 free UpCloud VM credit.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.6 Changed Public IP use to private IP use to ensure we are not charged when the serves sage goes over the quota

v1.5 Fixed 03 type (should have been 0.3)

v1.4 added disable nginx info

v1.3 added https://www.infobyip.com/tcpportchecker.php

v1.1 added https://hackertarget.com/tcp-port-scan/

v1.0 Initial Post

and

What is PiHole (Version 5)?

What is a Raspberry PI?

SD Card Choice

Raspberry Setup

My First Raspberry Pi Boot

Second Boot

SSH Access

I put the Raspberry PI in the Adafruit Case

SSH Connections to the Raspberry PI

Setting Up PiHole on the RaspBerry PI

PiHole Interface

Default Block Lists

Add 3rd party block lists

666,862 domains on my block list 🙂

LCD Screen Setup

Installing PADD to display PiHole stats on the LCD

Making PADD starts at boot

How to Update the PiHole from the CLI

After 1 Week

Done

Did it block Ad’s

Using Python to use buttons on the PiTFT Plus 320×240 TFT Touchscreen with a PiHole

Troubleshooting

Cooling

Rotating the screen

Buttons

Backup and Restore PiHole Settings

Handy Links

Why backup and restore

Murphy’s Law

Backup (the easiest way) with Adminer

Restore (the easiest way) with Adminer

Bonus methods.

Footer

Popular

Security

Code

Tech

Wordpress

General