with

How to install Windows 10 Pro alongside an OSX partition with Apple Boot Camp

April 20, 2018 by Simon Leave a Comment

This guide will show you how to install Windows 10 Pro alongside an OSX partition with Apple Boot Camp.

I have a number of guides on moving away from CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. I use OSX to develop and occasionally need to use Windows to achieve some development tasks, this is how you can install Windows on an Apple PC.

Apple Boot Camp

Apple Bootcamp software can be found here: https://support.apple.com/en-au/HT201468

Bootcamp Steps.

You can download a Windows 10 ISO here: https://www.microsoft.com/en-au/software-download/windows10

You will need to plug in a 16GB or larger USB key to use as an install medium.

Download Boot Camp

Download AppBoot Camp here: https://support.apple.com/en-au/boot-camp

Start Apple Boot Camp, click Continue, Tick all options, select the Windows 10 ISO file (beware Bootcamp will select the first ISO file it finds, ensure your USB is selected, Click Continue (tip: You may need to er format and prepare the USB key 10 times, Apple Bootcamp is not the most reliable program), It may take 6 hours to copy files pre Window Setup).

Choose your Windows partition size (at least 200GB is ideas)

When the Bootcamp Wizard is complete you can reboot into Windows (automatic at the end of the wizard)

Enter the desired details in each step (e.g Local, Windows Key, Windows Version, Partition and Country etc)

Loads more steps like Keyboard Layout, Domain, Microsoft Account, Password and Cortana and Privacy etc.

Windows is now installed, Apple Bootcamp will set up appropriate drivers for your Mac,

While this is happening I will install Google Chrome Canary.

Bootcamp has now finished setting up drivers.

Windows 10 Start menu (I prefer OSX’s simplicity)

You can set up your prefered Startup disk in the Apple System Preferences or press the Option key on startup and choose a partition to boot.

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.0 Initial post

Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare

April 14, 2018 by Simon Leave a Comment

This post will show you how you can setup Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM) Signatures and Domain-based Message Authentication, Reporting, and Conformance (DMARC) on your GMail (G Suite) email to limit spam and increase security.

I have a number of guides on moving away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. I use Google G Suite to send and receive emails that are linked to my domain (even via the command line) using multiple domains (with aliases).

Buy how can you extend your email security and limit spam?

Enter..

Sender Policy Framework (SPF) for Authorizing Use of Domains in Email

Background: SPF summary from the RFC document from the Internet Engineering Task Force (IETF).

“Email on the Internet can be forged in a number of ways. In particular, existing protocols place no restriction on what a sending host can use as the “MAIL FROM” of a message or the domain given on the SMTP HELO/EHLO commands. This document describes version 1 of the Sender Policy Framework (SPF) protocol, whereby ADministrative Management Domains (ADMDs) can explicitly authorize the hosts that are allowed to use their domain names, and a receiving host can check such authorization.”

Google has a guide in setting up SPF records for your G Suite account.

Scan your site for SPF, DKIM and DMARC configuration(s).

Gmail has a test site where you can check your site SPF, DKIM and DMARC etc: https://toolbox.googleapps.com/apps/checkmx/

How to set up an SPF Record

I followed this guide to set up an SPF record on my G Suite account. I use Cloudflare for my DNS provider so I’ll make my DNS changes there.

Update: Google instructions were wrong, use a TXT record and not a SPF record.

Backing up OSX or an Ubuntu server with Backblaze B2 Cloud Storage from the Command Line

March 14, 2018 by Simon Leave a Comment

This computer will show you can back up computer or server with Backblaze B2 Cloud Storage from the Command Line n OSX and Ubuntu.

This post is still being written. I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Also, I have blogged about how you can add block storage to a Vultr server, backup and restore snapshots , syncing files with rsync along with using GitHub and Bitbucket but what do you do if you need to backup large amounts of data?

Backblaze has a Cloud storage solution that costs as low as $0.005c a GB (a month), The first 10G is free. Backblaze say “From bytes to petabytes Backblaze B2 is the lowest cost high-performance cloud storage in the world. ”

Back Blaze have open sourced internal drive enclosure designs and drive failure stats and it’s time I gave them a try.

Goto https://www.backblaze.com

Create or sign in.

After you login got the dashboard.

Click Backblaze B2 Cloud Storage

Create a Bucket

Name the bucket (long names with a GUID are good).

You can rename the bucket here and change public/private and or upload/download files manually.

The first thing I did was limit the versions of files under the lifecycle settings for the bucket.

Now I created a series of subfolders to store files from different servers (I could have used many buckets but one bucket will do).

I can upload files via the Backblaze bucket GUI if I needed to.

Back Blaze has a command line tool for uploading: https://www.backblaze.com/b2/docs/quick_command_line.html

Install Steps

Backblaze state “The B2 command-line tool is available from the Python Package Index (PyPI) using the standard pip installation tool. Your first step is to make sure that you have either Python 2 (2.6 or later) or Python 3 (3.2 or later) installed.”

I have Python 2.7 installed

python --version
Python 2.7.10

1 2	python --version Python 2.7.10

Install PIP

sudo easy_install pip
Password:
Searching for pip
Best match: pip 1.5.6
Processing pip-1.5.6-py2.7.egg
pip 1.5.6 is already the active version in easy-install.pth
Installing pip script to /usr/local/bin
Installing pip2.7 script to /usr/local/bin
Installing pip2 script to /usr/local/bin

Using /Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg
Processing dependencies for pip
Finished processing dependencies for pip

sudo easy_install pip

Password:

Searching for pip

Best match: pip 1.5.6

Processing pip-1.5.6-py2.7.egg

pip 1.5.6 is already the active version in easy-install.pth

Installing pip script to /usr/local/bin

Installing pip2.7 script to /usr/local/bin

Installing pip2 script to /usr/local/bin

Using /Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg

Processing dependencies for pip

Finished processing dependencies for pip

I ran into issues updating b2 CLI

sudo pip install --upgrade b2
Requirement already up-to-date: b2 in /Library/Python/2.7/site-packages
Requirement already up-to-date: arrow>=0.8.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: logfury>=0.1.2 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: requests>=2.9.1 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: six>=1.10 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: tqdm>=4.5.0 in /Library/Python/2.7/site-packages (from b2)
Requirement already up-to-date: futures>=3.0.5 in /Library/Python/2.7/site-packages (from b2)
Downloading/unpacking python-dateutil from https://pypi.python.org/packages/bc/c5/3449988d33baca4e9619f49a14e28026399b0a8c32817e28b503923a04ab/python_dateutil-2.7.0-py2.py3-none-any.whl#md5=5a86a548fe776cc079bf4a835473e3f8 (from arrow>=0.8.0->b2)
  Downloading python_dateutil-2.7.0-py2.py3-none-any.whl (207kB): 207kB downloaded
Installing collected packages: python-dateutil
  Found existing installation: python-dateutil 1.5
    Uninstalling python-dateutil:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

sudo pip install --upgrade b2

Requirement already up-to-date: b2 in /Library/Python/2.7/site-packages

Requirement already up-to-date: arrow>=0.8.0 in /Library/Python/2.7/site-packages (from b2)

Requirement already up-to-date: logfury>=0.1.2 in /Library/Python/2.7/site-packages (from b2)

Requirement already up-to-date: requests>=2.9.1 in /Library/Python/2.7/site-packages (from b2)

Requirement already up-to-date: six>=1.10 in /Library/Python/2.7/site-packages (from b2)

Requirement already up-to-date: tqdm>=4.5.0 in /Library/Python/2.7/site-packages (from b2)

Requirement already up-to-date: futures>=3.0.5 in /Library/Python/2.7/site-packages (from b2)

Downloading/unpacking python-dateutil from https://pypi.python.org/packages/bc/c5/3449988d33baca4e9619f49a14e28026399b0a8c32817e28b503923a04ab/python_dateutil-2.7.0-py2.py3-none-any.whl#md5=5a86a548fe776cc079bf4a835473e3f8 (from arrow>=0.8.0->b2)

Downloading python_dateutil-2.7.0-py2.py3-none-any.whl (207kB): 207kB downloaded

Installing collected packages: python-dateutil

Found existing installation: python-dateutil 1.5

Uninstalling python-dateutil:

Cleaning up...

Exception:

Traceback (most recent call last):

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main

status = self.run(options, args)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run

requirement_set.install(install_options, global_options, root=options.root_path)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install

requirement.uninstall(auto_confirm=True)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall

paths_to_remove.remove(auto_confirm)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove

renames(path, new_path)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames

shutil.move(old, new)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move

copytree(src, real_dst, symlinks=True)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree

raise Error, errors

Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/zoneinfo-2010g.tar.gz'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/zoneinfo'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/relativedelta.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/parser.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/rrule.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/easter.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tz.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil/tzwin.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil', "[Errno 1] Operation not permitted: '/tmp/pip-jWEHna-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/dateutil'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

I tried installing via the alternative method (with no luck)

git clone https://github.com/Backblaze/B2_Command_Line_Tool.git
Cloning into 'B2_Command_Line_Tool'...
remote: Counting objects: 5084, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 5084 (delta 1), reused 1 (delta 0), pack-reused 5076
Receiving objects: 100% (5084/5084), 1.25 MiB | 689.00 KiB/s, done.
Resolving deltas: 100% (3622/3622), done.

git clone https://github.com/Backblaze/B2_Command_Line_Tool.git

Cloning into 'B2_Command_Line_Tool'...

remote: Counting objects: 5084, done.

remote: Compressing objects: 100% (8/8), done.

remote: Total 5084 (delta 1), reused 1 (delta 0), pack-reused 5076

Receiving objects: 100% (5084/5084), 1.25 MiB | 689.00 KiB/s, done.

Resolving deltas: 100% (3622/3622), done.

cd B2_Command_Line_Tool/

1	cd B2_Command_Line_Tool/

I tried running the setup script (with no luck)

sudo python setup.py install
setuptools 20.2 or later is required. To fix, try running: pip install "setuptools>=20.2"

1 2	sudo python setup.py install setuptools 20.2 or later is required. To fix, try running: pip install "setuptools>=20.2"

Upgrading setup tools also failed

sudo pip install "setuptools>=20.2"
Downloading/unpacking setuptools>=20.2
  Downloading setuptools-38.5.2-py2.py3-none-any.whl (490kB): 490kB downloaded
Installing collected packages: setuptools
  Found existing installation: setuptools 18.5
    Uninstalling setuptools:
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

sudo pip install "setuptools>=20.2"

Downloading/unpacking setuptools>=20.2

Downloading setuptools-38.5.2-py2.py3-none-any.whl (490kB): 490kB downloaded

Installing collected packages: setuptools

Found existing installation: setuptools 18.5

Uninstalling setuptools:

Cleaning up...

Exception:

Traceback (most recent call last):

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 122, in main

status = self.run(options, args)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/commands/install.py", line 283, in run

requirement_set.install(install_options, global_options, root=options.root_path)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1431, in install

requirement.uninstall(auto_confirm=True)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 598, in uninstall

paths_to_remove.remove(auto_confirm)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/req.py", line 1836, in remove

renames(path, new_path)

File "/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg/pip/util.py", line 295, in renames

shutil.move(old, new)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move

copytree(src, real_dst, symlinks=True)

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree

raise Error, errors

Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-8Vu7xp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]

Storing debug log for failure in /Users/simon/Library/Logs/pip.log

Backing up a Mac via command line with B2

More to come when I can get B2 CLI Installed.

Backing up a an Ubuntu machine via command line with B2

More to come when I can get B2 CLI Installed.

Update

My ticket with Backblaze was automatically closed with this note “If the issue is persisting, it may be easiest to map the installation to the user folder, rather than the system level.”

No ideas how but something to research.

Ask a question or the recommend an article

Revision History

V1.1 ticket closed

v1.0 Initial post

PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API

March 1, 2018 by Simon Leave a Comment

Developed by Simon Fearby https://www.fearby.com to allow PHP developers to integrate haveibeenpwned exposed password checks into their websites sign up’s (or logins). Get the latest version of this code from https://github.com/SimonFearby/phphaveibeenpwned/.

This demonstrates a PHP framework less way (using HTML 5, Javascript and PHP) to validate a password by hashing (SHA1) the password (before the HTML form is submitted). A part of the password hash is checked at https://api.pwnedpasswords.com/range/{[}xxxxx} API (before a decision to save the form data is made). A Password exposed result returns the user to the sign-up form and no match completes the submission process.

SHA is performed on the password entered in the HTML form in Javascript (Your password never leaves the browser) and the PHP submit receiver performs a partial hash check at api.pwnedpasswords.com. Only a fraction of your password hash is sent to api.pwnedpasswords.com and only a partial hash of your password is returned with other partial matches (Making it hard (for anyone listening) to know what password you used).

This demo does not enforce SSL, sanitize, validate any form data or save the password to a database etc. The aim of this page is to demonstrate integration with api.pwnedpasswords.com. This demo displays a password strength meter. signup_submit.php allows you to enable debugging to see what is going on (detected errors are sent back to the submit.php and alerts shown.

Basics

signup.php – Main PHP file with a form with basic Javascript validation.

signup_submit.php – The form submit sends the form data here and calls the pwnedpasswords API

signup_ok.php – Is loaded if the password is not exposed

The initial HTML code was generated with the Platforma GUI web generator. I added to the Javascript and relevant code. the HTML input field types are set to “text” (not “password”) so you can see the passwords. A password SHA1 hash is generated on form submission and the users password never leaves the browser.

A SHA1 hash of the password is updated and displayed (I am using the jsSHA library).

<script type="text/javascript" src="./js/sha/sha1.js"></script>

1	<script type="text/javascript" src="./js/sha/sha1.js"></script>

After a basic HTML Javascript form validation is performed the uses password is replaced with a hash then the form is submitted (to signup_submit.php).

// Generate SHA1 Hash
var shaObj = new jsSHA("SHA-1", "TEXT");
shaObj.update(document.forms["submitform"]["password1"].value);
var passwordhash = shaObj.getHash("HEX");
document.getElementById("sha135").value = passwordhash;

// Generate SHA1 Hash

var shaObj = new jsSHA("SHA-1", "TEXT");

shaObj.update(document.forms["submitform"]["password1"].value);

var passwordhash = shaObj.getHash("HEX");

document.getElementById("sha135").value = passwordhash;

signup_submit.php then takes the password hash, get the first 5 chars and fires up a curl connection to https://api.pwnedpasswords.com/range/$data when the data returns PHP checks the haveibeenpwned API body for matches of the matching password hashed and compared the known hash with the passwords has. Read more about how the API works here.

The PHP function that does the AI check is located here

function sendPostToPwnedPasswordsCom($data) {

    $curl = curl_init();		// Init Curl Object

    if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
    	echo "Data to Send: $data <br />";
    	echo "Sending Data to: https://api.pwnedpasswords.com/range/$data <br />";
    }

    // Set Curl Options: http://php.net/manual/en/function.curl-setopt.php
    curl_setopt($curl, CURLOPT_URL, "https://api.pwnedpasswords.com/range/$data");
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true); 
	curl_setopt($curl, CURLOPT_FRESH_CONNECT, true); 		// TRUE to force the use of a new connection instead of a cached one.
	curl_setopt($curl, CURLOPT_FORBID_REUSE, true); 		// TRUE to force the connection to explicitly close when it has finished processing, and not be pooled for reuse.
	curl_setopt($curl, CURLOPT_TIMEOUT, 10); 
    curl_setopt($curl, CURLOPT_MAXREDIRS, 10); 
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);		// TRUE to follow any "Location: " header that the server sends as part of the HTTP header (note this is recursive, 
    														// PHP will follow as many "Location: " headers that it is sent, unless CURLOPT_MAXREDIRS is set).

	// Make a request to the api.pwnedpasswords.com
    $http_request_result = curl_exec ($curl);
    $http_return_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);

	// api.pwnedpasswords.com Response codes
	/*
	Semantic HTTP response code are used to indicate the result of the search:

	Code	Description
	200	Ok — everything worked and there's a string array of pwned sites for the account
	400	Bad request — the account does not comply with an acceptable format (i.e. it's an empty string)
	403	Forbidden — no user agent has been specified in the request
	404	Not found — the account could not be found and has therefore not been pwned
	429	Too many requests — the rate limit has been exceeded
	*/


	// Change the return code to debug
	//$http_return_code = 429;
	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
      	echo "Return HTTP CODE: $http_return_code <br />";
    }
	
    // What was the http response code from api.pwnedpasswords.com
	if ($http_return_code == 200) {
		// OK (All other return codes direct the user back with an error)
		if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
	    	echo "Return HTTP Data site: " . strlen($http_request_result) . " bytes. <br />";
	    }

	} elseif ($http_return_code == 400) {
		// api.pwnedpasswords.com: API Bad Request
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Bad Request <br />";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIBadRequest&code=" . $http_return_code);
		die();

	} elseif ($http_return_code == 403) {
    	// api.pwnedpasswords.com: API Bad User Agent
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Bad User Agent <br />";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIBadUserAgent&code=" . $http_return_code);
		die();

	} elseif ($http_return_code == 404) {
		// api.pwnedpasswords.com: API User Not Found, not needed in his password hash check but we may as well catch now
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API User Not Found <br />";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIuserNotFound&code=" . $http_return_code);
		die();

    } elseif ($http_return_code == 429) {
    	// api.pwnedpasswords.com: API Too Many Requests
    	if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Too Many Requests</ br>";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIuserTooManyRequests&code=" . $http_return_code);
		die();

    } else {
    	// api.pwnedpasswords.com: API Down

		if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
        	echo "API Down!</ br>";
        }

        header("Location: signup.php?Error=PwnedpasswordsAPIDown&code=" . $http_return_code);
		die();
    } 

    // Tidy up the curl object and return the request api body
    curl_close ($curl); 
	return $http_request_result; 
}

100

function sendPostToPwnedPasswordsCom($data) {

$curl = curl_init(); // Init Curl Object

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "Data to Send: $data <br />";

echo "Sending Data to: https://api.pwnedpasswords.com/range/$data <br />";

}

// Set Curl Options: http://php.net/manual/en/function.curl-setopt.php

curl_setopt($curl, CURLOPT_URL, "https://api.pwnedpasswords.com/range/$data");

curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);

curl_setopt($curl, CURLOPT_FRESH_CONNECT, true); // TRUE to force the use of a new connection instead of a cached one.

curl_setopt($curl, CURLOPT_FORBID_REUSE, true); // TRUE to force the connection to explicitly close when it has finished processing, and not be pooled for reuse.

curl_setopt($curl, CURLOPT_TIMEOUT, 10);

curl_setopt($curl, CURLOPT_MAXREDIRS, 10);

curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true); // TRUE to follow any "Location: " header that the server sends as part of the HTTP header (note this is recursive,

// PHP will follow as many "Location: " headers that it is sent, unless CURLOPT_MAXREDIRS is set).

// Make a request to the api.pwnedpasswords.com

$http_request_result = curl_exec ($curl);

$http_return_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);

// api.pwnedpasswords.com Response codes

Semantic HTTP response code are used to indicate the result of the search:

Code Description

200 Ok — everything worked and there's a string array of pwned sites for the account

400 Bad request — the account does not comply with an acceptable format (i.e. it's an empty string)

403 Forbidden — no user agent has been specified in the request

404 Not found — the account could not be found and has therefore not been pwned

429 Too many requests — the rate limit has been exceeded

// Change the return code to debug

//$http_return_code = 429;

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "Return HTTP CODE: $http_return_code <br />";

}

// What was the http response code from api.pwnedpasswords.com

if ($http_return_code == 200) {

// OK (All other return codes direct the user back with an error)

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "Return HTTP Data site: " . strlen($http_request_result) . " bytes. <br />";

}

} elseif ($http_return_code == 400) {

// api.pwnedpasswords.com: API Bad Request

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "API Bad Request <br />";

}

header("Location: signup.php?Error=PwnedpasswordsAPIBadRequest&code=" . $http_return_code);

die();

} elseif ($http_return_code == 403) {

// api.pwnedpasswords.com: API Bad User Agent

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "API Bad User Agent <br />";

}

header("Location: signup.php?Error=PwnedpasswordsAPIBadUserAgent&code=" . $http_return_code);

die();

} elseif ($http_return_code == 404) {

// api.pwnedpasswords.com: API User Not Found, not needed in his password hash check but we may as well catch now

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "API User Not Found <br />";

}

header("Location: signup.php?Error=PwnedpasswordsAPIuserNotFound&code=" . $http_return_code);

die();

} elseif ($http_return_code == 429) {

// api.pwnedpasswords.com: API Too Many Requests

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "API Too Many Requests</ br>";

}

header("Location: signup.php?Error=PwnedpasswordsAPIuserTooManyRequests&code=" . $http_return_code);

die();

} else {

// api.pwnedpasswords.com: API Down

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "API Down!</ br>";

}

header("Location: signup.php?Error=PwnedpasswordsAPIDown&code=" . $http_return_code);

die();

}

// Tidy up the curl object and return the request api body

curl_close ($curl);

return $http_request_result;

}

You can enable debugging in signup_submit.php if you wish (but if an echo has presented debug data and a header redirect happens it will produce an errror).

// define('ENABLE_DEBUG_OUTPUT', true);
define('ENABLE_DEBUG_OUTPUT', false);

1 2	// define('ENABLE_DEBUG_OUTPUT', true); define('ENABLE_DEBUG_OUTPUT', false);

echo statements are wrapped

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {
    echo "API Bad Request <br />";
}

if (defined('ENABLE_DEBUG_OUTPUT') && true === ENABLE_DEBUG_OUTPUT) {

echo "API Bad Request <br />";

}

signup_submit.php will redirect the browser back to signup.php if an error is found

header("Location: signup.php?Error=PwnedpasswordsAPIuserTooManyRequests&code=" . $http_return_code);
die();

1 2	header("Location: signup.php?Error=PwnedpasswordsAPIuserTooManyRequests&code=" . $http_return_code); die();

A success event (no password hash found) the user is sent to ignup_ok.php

header("Location: signup_ok.php");
die();

1 2	header("Location: signup_ok.php"); die();

signup.php will check for the Error query string

Then display bootstrap alert errors for each error case

if ($error == "PwnedpasswordsAPIuserTooManyRequests") {
    echo '<div class="alert alert-danger" role="alert">';
    echo '<br /><a target="_blank" href="https://api.pwnedpasswords.com">https://api.pwnedpasswords.com</a> reported too many requests to the API from this IP, please wait a few seconds and try again.<br /> (E009)<br />';
    echo '<br />Please consider donating to Troy Hunt <a target="_blank" href="https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/">https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/</a> (<em>developer of <a target="_blank" href="https://haveibeenpwned.com">https://haveibeenpwned.com</a></em>).<br />';
    echo '</div>'; 
}

if ($error == "PwnedpasswordsAPIuserTooManyRequests") {

echo '<div class="alert alert-danger" role="alert">';

echo '<br /><a target="_blank" href="https://api.pwnedpasswords.com">https://api.pwnedpasswords.com</a> reported too many requests to the API from this IP, please wait a few seconds and try again.<br /> (E009)<br />';

echo '<br />Please consider donating to Troy Hunt <a target="_blank" href="https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/">https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/</a> (<em>developer of <a target="_blank" href="https://haveibeenpwned.com">https://haveibeenpwned.com</a></em>).<br />';

echo '</div>';

}

if ($error == "PwnedpasswordsAPIuserTooManyRequests") {
    echo '<div class="alert alert-danger" role="alert">';
    echo '<br /><a target="_blank" href="https://api.pwnedpasswords.com">https://api.pwnedpasswords.com</a> reported too many requests to the API from this IP, please wait a few seconds and try again.<br /> (E009)<br />';
    echo '<br />Please consider donating to Troy Hunt <a target="_blank" href="https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/">https://www.troyhunt.com/donations-why-i-dont-need-them-and-why/</a> (<em>developer of <a target="_blank" href="https://haveibeenpwned.com">https://haveibeenpwned.com</a></em>).<br />';
    echo '</div>'; 
}

if ($error == "PwnedpasswordsAPIuserTooManyRequests") {

echo '<div class="alert alert-danger" role="alert">';

echo '</div>';

}

Sample Errors

Sample Password exposed error.

Sample API Offline alert

If form field needs attention a JavaScript event is written to set the focus etc.

if ($error == "PasswordExposed") {
    echo '<script>';
    echo 'document.getElementById("password1").focus();';
    echo 'document.getElementById("password1").select();';
    echo '</script>';
}

if ($error == "PasswordExposed") {

echo '<script>';

echo 'document.getElementById("password1").focus();';

echo 'document.getElementById("password1").select();';

echo '</script>';

}

If no password hash has been matched with pwnedpasswords the user is directed to signup_ok.php (not very exciting but that’s your jobs to integrate it with your system and harden).

Sample debugging output

Get the code: https://github.com/SimonFearby/phphaveibeenpwned/

More Reading

If you are using Ubuntu don’t forget to set up a free SSL cert, setting up an SSL cert on OSX is also a good idea. I have guides on setting up an Ubuntu server on AWS, Digital Ocean and Vultr. I love Vultr VM hosts and have blogged about setting up WordPress via the CLI, uploading files with SSH, restoring Vultr Snapshots etc.

I hope this guide helps someone.

Ask a question or recommend an article

Revision History

v1.0 Initial post

Scanning WordPress with Gravity scan for free to detect Supply Chain Attacks and WordPress malware

January 5, 2018 by Simon 1 Comment

A recent trend with some WordPress Plugins (and Google Chrome Extensions) is malicious parties will purchase existing plugins (extensions) and inject malicious code into new versions to infect sites and software, this is called “Supply Chain Attacks”. This is a personal unpaid review of Gravity Scan.

Update Feb 2018: Gravity Scan is shutting down 🙁

Recently WordFence wrote a blog post about Supply Chain Attacks found cases where older plugins are being purchased by malicious people in order to infect WordPress sites. WordPress CMS apparently runs 29% of the websites on the internet. Wordfence is a firewall and Gravity Scan is vulnerability scanner, they complement each other.

I have blogged here about setting up WordPress via Command line and setting up an Ubuntu server for as low as 42.5 a month on Vultr.

What can you do to protect your WordPress sites from “Supply Chain Attacks”? First, install the WordFence plugin (I blogged about it here). Wordfence gives you a great set of security settings and reports to keep your site safe. The Wordfence dashboard page on your site is a good place to stay up to date.

WordFence is a Firewall, Gravity scan is a vulnerability and malware scanner. Read more here.

Gravity Scan

Gravity scan is also made by the WordFence people to enable external audits and reports.

At login, you will be prompted to add a domain.

Scan

tip: You may want to whitelist Gravity scan servers. Read my guide about securing Ubuntu in the cloud.

sudo ufw allow from 68.64.48.0/27 to any port 443

1	sudo ufw allow from 68.64.48.0/27 to any port 443

A site scan will automatically be started.

Scan Results

Post Scan Actions

Speed Up future scans by downloading the Install Gravity Scan Accelerator (by clicking “Not instaled” under “Accelerator” in scan results) and follow the instructions to download, upload and verify the accelerator.

Read the Gravity Scan Accelerator Install Instructions here.

tip: I had to run the following command to make the

sudo chown www-data.www-data /www/gravityscan-agent-#############################################.php

1	sudo chown www-data.www-data /www/gravityscan-agent-#############################################.php

I also clicked “Trust Badge” link and added the script code to my site and verified it.

I now have a scan badge in my site footer.

Future scans are all good to go.

New Scan Options

It looks as if the accelerator gives more server-side verifications of checks of WordPress and PHP versions etc.

Go Pro

Gravity Scan also offers a non-free (paid) version where you can enable more options, enable scan schedules and set up SMS alerts and more for $4.95 a month per site.

To be honest I am happy with performing manual scans and I’d rather pay for a premium WordFence subscription first.

The Catch

Hang on Gravity scan requires a Pro membership to see High and Critical issues 🙁

I decided not to go pro to reveal issues.

A few months later

I started receiving scan results with severity Critical (but I can’t see results until I start a trail (and enter payment details)).

Time to start a trial

I started a trial and full details were shown, the critical error was my fault

Critical Issue

This was my fault, I left a previous version of WordPress in a subfolder from when I moved the site to a self-managed server. A quick few Linux commands later (removed) and this was fixed.

High Issue

Publically accessible file (fixed with a chmod command)

Current Scan Results

Remote Scan Options

Daily Scans, Alter levels, Malware, Vulnerability and status checks. Definitely, install the Accelerator as it found my local backup of WordPress.

Pros

Found a publically readable file
Found a past copy of my WordPress site (and all known issues with the old WordPress backup).
Can setup daily remote scans.

Cons

You have to go pro.
Cant read my NGINX version (“Nginx version not detected, Gravityscan is unable to detect any associated vulnerabilities.“). I logged a ticket. Surely they can add a shell to”nginx -v” to the scan accelerator.
No word fence discount bundle?
Gravity scan and Word fence on twitter are slow to respond.

More to come.

More Reading

Hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

Revision History

V.1.5 Gravity Scan shutting down

v1.4 Added remote scan options

v1.3 Pros and Cons and current results.

v1.2 Added more

v1.1 Fixed a few issues

v1.0 Initial Version

How to upload and download files to a server with sftp (over the SSH protocol)

December 7, 2017 by Simon 2 Comments

I have blogged about http://c9.io before and how it makes managing a remote Ubuntu server easier. Recently AWS acquired C9 and integrated it into AWS. This has triggered me to find a more open/free way to connect to my servers. I like AWS but I can tell that C9 will someday block you from talking to non-AWS servers.

C9 Aquisition

C9 on AWS: https://aws.amazon.com/cloud9/?origin=c9io

C9 Aquisition: https://c9.io/announcement

I have blogged about connecting to an AWS EC2 instance with C9 (before AWS acquired C9), The quickest way to set up a scalable development IDE and web server when I was using Digital Ocean Servers, How to buy a new domain (dedicated server from digital ocean) and add an SSL certificate from NameCheap but when I moved to Vultr serves I decided to ditch C9 and save $9 a month. I got used to setting up an SSH connection, using rmate to edit remote files locally with Sublime and SSH’ing into the box with vSSH on OSX but nothing replaced the C9 file management experience.

C9 was a good IDE (but recent price hikes and AWS purchase have made me cautious).

Uploading files without C9 to Ubuntu

Uploading files from an OSX laptop to a remote Ubuntu server is possible with the SCP command

scp ~/Desktop/FilesToUpload.zip username@www.yourserver.com /www/FilesToUpload.zip
< # Enter the remote server password

1 2	scp ~/Desktop/FilesToUpload.zip username@www.yourserver.com /www/FilesToUpload.zip < # Enter the remote server password

Server Actions (to upload the old way, with scp)

#Install Unzip (if you do not have it)
sudo apt-get install unzip

# Unzip the file
cd /www/
unzip /www/FilesToUpload.zip

#Install Unzip (if you do not have it)

sudo apt-get install unzip

# Unzip the file

cd /www/

unzip /www/FilesToUpload.zip

There has to be a better way

I have used the Forklift 3 program before (my review here) and in recent google searches, it was suggested that Forklift best SFTP program on OSX. Yay. This unix.stackexchange.com thread mentions the difference between SFTP and SSH.

Below are the steps to use SFTP in Foftlift 3 on OSX to connect to a Vultr server (you could use Digital Ocean). Ensure you have a working SSH connection setup from your server to your local OSX.

Setting up an SFTP Connection in Forklift 3

Read my Forklift 3 guide here first.

Update your Forklift to the latest version.

Add a Favorite in Forklift 3

Name the connection

In Forklift edit the connection

Specify SFTP and your servers working SSH username and password, and “/” folders

Save.

Double Click the entry to connect to your server.

To upload or download files with Forklift over SFTP, simply drag and drop.

Simple

You can now use Forklift to navigate folders, delete files, set remote file permissions all from a GUI. You can still use SSH CLI and or Sublime.

Quik Edit works too.

Nice, this gives me a convenient GUI way to upload, download and edit files instead of via CSP/SSH/CLI.

Now is there an SFTP plugin for Sublime?

More to come soon.

Donate and make this blog better

Ask a question or recommend an article

Revision History

v1.0 Initial Draft

Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl

December 3, 2017 by Simon 1 Comment

Below is my quick guide to showing how I recover lost files on a removable (or internal drive) with software on Windows. This is not a paid advert but this is what works for me (and has worked with everything I have thrown at it). Tell me what works for you (especially on OSX).

“Helping one person might not change the world but it could change the world for one person” – Buddha.

I hope this guide helps someone, I have never seen someone happier than when I have been able to restore lost files or photos for someone. Not everyone backs up data (read my guide here).

Before you start

Don’t touch (format, access or install software) onto drives and USB drive that you want to restore files from or you may overwrite files on that drive. Doing this will potentially block the ability to restore files. Electronics die, software gets corrupted, this can be complex so If in doubt seek professional help and advice before proceeding. This is advice only and it’s your call with your data, Restoring files on damaged magnetic or USB drives can damage the data storage platter or storage chips, so proceed at your own risk. (legal disclaimer over, good luck).

More on how file systems and storage work

How File Systems Work

How the File Allocation Table Works

Installing DIY Data Recovery

The DIY Data Recovery software can restore images (photos) from a UBS, SD card or files (images or non-images) from data or operating system drives. If you have a system or data drive from a PC you can plug it into another PC running DIY Dat Recovery and scan and restore from the possibly bad drive to a good host drive, never restore files to a bad drive from a bad drive. I am an Apple Mac user now but only know of this Windows method for restoring files, please let me know of an Apple Mac way to restore files on FAT, NTFS and other file systems please.

On a working Windows computer install DIY Data Recovery software from here.

Plug in your dead USB or another drive to restore files from (do not interact with the drive once Windows can see it).

Do not format or modify the drive if prompted (cancel this screen).

From memory, you can follow the steps below on a free trial version but you will need a paid version (serial number) to actually restore files. The free trial will show files that it can restore though. I paid about $60 for the software a few years ago, it is currently 59 EUROS. Enter your serial number if you have one.

After you register it (or continue with the free trial) select the drive to scan for files and click Next.

DIY Data Recovery will scan each sector and try and find files. On a smaller healthy drive, the scan won’t take long, on a larger drive (e.g a 2TB NTFS drive it may take 18+ hours).

In this case, we can see the USB drive has two bad sectors (red squares) at the front of the USB stick in the file system sectors (green squares). This possibly happened when the USB was pulled out when the operating system was writing to the USB stick. If the USB was dying the bad sectors would be at random positions.

Data fragments are listed as blue squares.

When the scan is done click ‘Save” to save the found sectors state (saving the state can save you time later).

Disclaimer: Clicking save does not save lost files it just saves the current state.

You can click the legend button to see the state of each file (Not processed, Presumably valid, Invalid, bad sectors etc)

Tick the files you want to be restored.

FYI: You will see some files you know, some files you forgot and some weird files. Restoring files is hit and miss and you may never see files again (do backup). Benjamin Franklin once said “Failing to plan is planning to fail”, You can also say “Failing to backup is guaranteeing lost data”.

Restoring Files

Create a folder on your working Desktop to restore files to.
Do increase Read and Write cache
Do set longer Timeouts (e.g 1000, 5000, 10000, 60000)

Click “Start copying the selected files” to restore.

Hopefully, you will have files restored, if not seek professional help. I’d recommend keeping the corrupt or source drive in a drawer, you may be able to use other tools to restore more files at a later date.

Donate and make this blog better

Ask a question or recommend an article

Revision History

v1.0 Initial Post

Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin

December 2, 2017 by Simon 4 Comments

Below is my quick blog posts on using the EWWW IO ExactDN CDN plugin in WordPress to set up an ExactDN (Global Dis.cotribution Network (CDN)) to distribute images to my site’s visitors and shrink (optimize) images in posts.

I have blogged before on speeding up WordPress that has involved moving servers away from CPanel domains to self-managed servers (e.g on Digital Ocean or Vultr), using Lazy Load image plugins like BJ Lazy Load, Optimize images automatically in WordPress with EWWW.io and scaling and moving servers closer to your customers.

Today I am going to set up the https://www.ewww.io/resize/ ExactDN (CDN) delivery network. I am paying for this myself (and this is an unbiased review).

FYI: I use Ubuntu servers and not Windows.

Know your starting point.

“If you fail to plan, you are planning to fail!” – Benjamin Franklin

In my original post about Speeding up WordPress, I used the webpagetest.org site to test my sites response times, I was getting an embarrassing 21 seconds load time and 6 second first-byte time. I have worked to speed up WordPress by moving WordPress to a self-managed server (away from CPanel), used the BJ Lazy Load plugin and the awesome ewww.io image optimize plugin, now I get about 4-5 seconds.

I am hoping adding a CDN will make things faster. My blog is delivering over two-thirds images, perfect for a CDN, this is why I am trying this out.

TIP: Check where your customers/readers are located, and how many are New versus Returning customers? Do you need a CDN to deliver content (Images) that are closer to your customers/readers or do you need to move your web server somewhere else? The more you know the more you can help them. Worst case you will be supporting a positive experience (and potentially turning a one time visitor into a returning visitor).

I looked at my Google Analytics data to see where my visitors are. Whether good or bad, they are all over the world (Hello)?

Other data is available in Google Analytics. I can see the last few years growth is growing and I am getting more returning visitors, now is the time to ensure my site is ready for more traffic and returning visitors.

Note: The fall in traffic in the Audience overview (right-hand side of the left image) is the unfinished month (not a reader fall off).

Personally, I set a goal to have a high page bounce rate of 90% be way lower (at present I am at about 80% and falling (good) and my page read time has gone from 40 seconds to 1 minute 40. Every bit you can do will help create a positive experience and help your visitors. I can see from the data above the content is being read, I am building returning visitors and they are geographically spread out. A CDN will be great. After you know where your visitors are it is good to know the times of day that your visitors are hitting your site. Lucky for me it is spread evenly over a 24 hour period.

Data Quantity

FYI: My servers outgoing data (last 30 days), not huge but BJ LazyLoad and image optimization may be helping.

Traffic Forecast

Do you know the forecasted growth of your website?

Measure Before Optimizing

Before I started to optimize my WordPress site (hosted on a shared CPanel server) I had the following Web Page Test score. I tested from Singapore as that’s was where my server was originally (and the closest to me).

TIP: Read more about Performance, First Byte, Start Render and Complete scores at my blog post here.

Web Page Performace Test Results (Before Optimizations):

Load Time: 23.672s
First Byte: 6.743s
Start Render: 11.8300s
Speed Index: 15024
Requests: 132/164 (Document complete v Fully Loaded)
Bytes: 3,346KB/3,454KB (Document complete v Fully Loaded)

Quick Web Page Performace Score Card (Before Optimizations):

First Byte: F (I should have captured the subscores)
Keep Alive Enabled: F (I should have captured the subscores)
Compress Transfer: F (I should have captured the subscores)
Compress Images: A (I should have captured the subscores)
Cache Static Content: X (I should have captured the subscores)
Effective use of CDN: X (I should have captured the subscores)

My initial scores were bad across the range of tests (before optimisations). On the upside, I was manually compressing images with a tool on my desktop before uploading images and this showed an “A” but this scorecard overall was really bad.

Here are the results after Quick Optimizations (EWWW.io image compress, moved the server, reorganizing the site and Lazy Load Images)

Now I get these results after speeding up my site (after using the EWWW.io image resizing, reorganizing the site, minifying, lazy load images, moving servers etc.).

Web Page Performace Test Results (After Simple Optimizations):

Load Time: 8.823s (down 14.849s)
First Byte: 3.5533s (down 3.1897s)
Start Render: 5,4800s (down 6.35s)
Speed Index: 5594 (down 9430)
Requests: 73/76 (Document complete v Fully Loaded) (down 59/88)
Bytes: 848KB/855KB (Document complete v Fully Loaded) (down 2,498KB/2,599KB)

Quick Web Page Performace Score Card (After Simple Optimizations):

First Byte: F (I should have captured the subscores)
Keep Alive Enabled: A (I should have captured the subscores)
Compress Transfer: A (I should have captured the subscores)
Compress Images: A (I should have captured the subscores)
Cache Static Content: B (I should have captured the subscores)
Effective Use of CDN: X (I should have captured the subscores)

Even at 4 seconds web page “First Byte“, this is considered not good. My brain says I want sub 1 second, I doubt this is achievable with WordPress over thousands of miles away with SSL (read here about scalability).

I know https and non-geographically favourable servers add half a second to data. SSL will add processing overheads and latency period. If you only want speed don’t setup SSL but if you want SEO and security then setup SSL.

WebPageTest.org test reveals there is no effective use of Content Delivery Networks (CDN) on fearby.com (that’s why I am about to install EWWW.io ExactDN).

ExactDN (Content Delivery Network)

I did try and set up a number of caching and CDN plugins in the past (e.g Max CDN, W3 Total Cache, WP Fastest Cache, Cache Enable, WP Rocket, WP Super Cache, etc.) but they either made results worse or were impossible to set up.

Now that EWWW.io has a CDN let’s give that a go.

What is EWWW.io’s ExactDN?

You can read more about EWWW.io’s two-pronged approach to delivering one plugin to A) “compress images” and B) “add a Content Delivery Network (CDN)” here: https://ewww.io/resize/

Ensure you read up about EWWW.io’s ExactDN here: https://ewww.io/resize/ . If you have not used EWWW.io check out my review of the EWWW.io image compression plugin here first.

Purchasing and Installing ExactDN Exact DN

FYI: If you have used the Optimise images automatically in WordPress plugin from EWWW.io plugin before, then you will see past purchases here.

Now you can go back to the EWWW.io ExactDN product page (https://ewww.io/resize/) and purchase a subscription (Make sure you are logged in with an EWWW.io account before you purchase ExactDN).

I purchased an ExactDN monthly subscription for $9.00 monthly (with a $1 signup fee for the cloud compression service).

Purchase confirmation screen.

Post-purchase, I was advised to find my “Site Address (URL)” in WordPress and add it to EWWW.io Manage Sites screen.

I now noticed I had a CDN for my domain ( fearby-com.exactdn.com ). Nice.

EWWW.io said to tick the CDN option in the Image Resize area of the EWWW.io plugin. But before I do that I will update WordPress Core and WordPress Plugins before enabling the ExactDN as they are out of date.

TIP: I update WordPress Core and WordPress plugins via command line (my guide here).

Backup WordPress (just in case)

It is always a good idea to backup your website, I logged into my Ubuntu server and checked the size of my site before backing it up.

du -hs ./www
513M    ./www

1 2	du -hs ./www 513M ./www

I copied the website folder (from “/www/” to “/www-backup”)

cp -rTv /www/ /www-backup

1	cp -rTv /www/ /www-backup

I confirmed the copied folder size (same, good)

du -hs ./www-backup/
513M    ./www-backup/

1 2	du -hs ./www-backup/ 513M ./www-backup/

I dumped all databases with the MySQL dump command

mysqldump --all-databases > /my-sql-backup.sql -u root -p
> Enter password: ##############################

ls my-sql*.sql -al
-rw-r--r-- 1 removed removed 75605938 Dec  1 21:11 my-sql-backup.sql

mysqldump --all-databases > /my-sql-backup.sql -u root -p

> Enter password: ##############################

ls my-sql*.sql -al

-rw-r--r-- 1 removed removed 75605938 Dec 1 21:11 my-sql-backup.sql

While I was there I compress the SQL backup files (text files)

tar -czvf my-sql-backup.sql.tar.gz my-sql-backup.sql

1	tar -czvf my-sql-backup.sql.tar.gz my-sql-backup.sql

Checking the CDN Status

I checked the DNS replication for fearby-com.exactdn.com with this DNS checker. It’s setup and ready to go across the globe 🙂

A quick CNAME check reveals its upstream provider is fearbycom-8ba8.kxcdn.com. Keycdn.com have been around since 2012. It is great that EWWW.io has paired with keycdn and included their image compression magic in a single WordPress plugin.

Configuring the EWWW.io plugin in WordPress.

Before you enable the CDN below do check the https://www.whatsmydns.net/ and see if your CDN has replicated around the world (Australia can be a bit slow at DNS replication from time to time). Do wait at least 10 minutes before proceeding.

Click the settings in your EWWW Image Optimizer Plugin.

Now click the Resize Settings tab and click Enable CDN in the EWWW Image Optimizer plugin screen.

Don’t forget to click Save Changes

Testing the CDN

https://www.webpagetest.org NOW reports that my site is using a CDN 🙂

FYI: I have always used webpage test from Singapore and I have done the same here to compare apples to apples. Singapore is not a good test location in Australia (my server is in Australia) and it is 200ms away and the added layer of SSL adds latency to the connection (read here). This is a real-world test though (worst case).

CDN's Used:
fearby.com : 
fearby-com.exactdn.com : KeyCDN
fonts.googleapis.com : Google
fonts.gstatic.com : Google
static.addtoany.com : Cloudflare
www.googletagmanager.com : Google
pagead2.googlesyndication.com : Google
www.youtube.com : Google
adservice.google.com.sg : Google
adservice.google.com : Google
googleads.g.doubleclick.net : Google
www.google-analytics.com : Google
s.ytimg.com : Google
stats.g.doubleclick.net : Google
fearby-com.disqus.com : Fastly
ssl.google-analytics.com : Google

CDN's Used:

fearby.com :

fearby-com.exactdn.com : KeyCDN

fonts.googleapis.com : Google

fonts.gstatic.com : Google

static.addtoany.com : Cloudflare

www.googletagmanager.com : Google

pagead2.googlesyndication.com : Google

www.youtube.com : Google

adservice.google.com.sg : Google

adservice.google.com : Google

googleads.g.doubleclick.net : Google

www.google-analytics.com : Google

s.ytimg.com : Google

stats.g.doubleclick.net : Google

fearby-com.disqus.com : Fastly

ssl.google-analytics.com : Google

FYI: https://www.webpagetest.org does want other (all) static content to be on a CDN to give a higher score than 42 out 100, sorry I did not get a subscore before enabling the CDN.

I was expecting a green A here for CDN but Webpage Test has given me more items to investigate to ensure WordPress plugins are also fast (minify or move to CDN). It appears WordPress includes are my next target for optimizing.

Web Page Test indicates the following WordPress assets should also be in CDN.

FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/styles/ytprefs.min.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/svgxuse.js?ver=1.1.21
FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/scripts/ytprefs.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/add-to-any/addtoany.min.js?ver=1.0
FAILED - https://fearby.com/wp-content/plugins/crayon-syntax-highlighter/js/min/crayon.min.js?ver=_2.7.2_beta
FAILED - https://fearby.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1
FAILED - https://fearby.com/wp-includes/js/jquery/jquery.js?ver=1.12.4
FAILED - https://fearby.com/wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/menu/superfish.args.js?ver=2.5.3
FAILED - https://fearby.com/wp-content/plugins/crayon-syntax-highlighter/css/min/crayon.min.css?ver=_2.7.2_beta
FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/skip-links.js?ver=2.5.3
FAILED - https://fearby.com/wp-includes/js/comment-reply.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/js/hoverIntent.min.js?ver=1.8.1
FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/menu/superfish.js?ver=1.7.5
FAILED - https://fearby.com/wp-includes/js/jquery/ui/tabs.min.js?ver=1.11.4
FAILED - https://fearby.com/wp-content/themes/news-pro/js/global.js?ver=3.2.2
FAILED - https://fearby.com/wp-content/themes/news-pro/js/responsive-menus.min.js?ver=3.2.2
FAILED - https://fearby.com/wp-includes/js/jquery/ui/core.min.js?ver=1.11.4
FAILED - https://fearby.com/wp-content/themes/news-pro/style.css?ver=3.2.2
FAILED - https://fearby.com/wp-includes/js/jquery/ui/widget.min.js?ver=1.11.4
FAILED - https://fearby.com/wp-content/themes/news-pro/js/jquery.matchHeight.min.js?ver=3.2.2
FAILED - https://fearby.com/wp-content/plugins/contact-form-7/includes/css/styles.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/scripts/fitvids.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/js/wp-embed.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/bj-lazy-load/js/bj-lazy-load.min.js?ver=2
FAILED - https://fearby.com/wp-content/plugins/disqus-comment-system/media/js/count.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/js/wp-emoji-release.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/wp-seo-html-sitemap/style.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/symbol-defs.svg
FAILED - https://fearby.com/wp-includes/css/dashicons.min.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/themes/news-pro/images/favicon.ico
FAILED - https://fearby.com/wp-content/plugins/genesis-tabs/style.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/css/style.css?ver=2.0.1
FAILED - https://fearby.com/wp-content/plugins/add-to-any/addtoany.min.css?ver=1.14

FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/styles/ytprefs.min.css?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/svgxuse.js?ver=1.1.21

FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/scripts/ytprefs.min.js?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/plugins/add-to-any/addtoany.min.js?ver=1.0

FAILED - https://fearby.com/wp-content/plugins/crayon-syntax-highlighter/js/min/crayon.min.js?ver=_2.7.2_beta

FAILED - https://fearby.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1

FAILED - https://fearby.com/wp-includes/js/jquery/jquery.js?ver=1.12.4

FAILED - https://fearby.com/wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/menu/superfish.args.js?ver=2.5.3

FAILED - https://fearby.com/wp-content/plugins/crayon-syntax-highlighter/css/min/crayon.min.css?ver=_2.7.2_beta

FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/skip-links.js?ver=2.5.3

FAILED - https://fearby.com/wp-includes/js/comment-reply.min.js?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-includes/js/hoverIntent.min.js?ver=1.8.1

FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/menu/superfish.js?ver=1.7.5

FAILED - https://fearby.com/wp-includes/js/jquery/ui/tabs.min.js?ver=1.11.4

FAILED - https://fearby.com/wp-content/themes/news-pro/js/global.js?ver=3.2.2

FAILED - https://fearby.com/wp-content/themes/news-pro/js/responsive-menus.min.js?ver=3.2.2

FAILED - https://fearby.com/wp-includes/js/jquery/ui/core.min.js?ver=1.11.4

FAILED - https://fearby.com/wp-content/themes/news-pro/style.css?ver=3.2.2

FAILED - https://fearby.com/wp-includes/js/jquery/ui/widget.min.js?ver=1.11.4

FAILED - https://fearby.com/wp-content/themes/news-pro/js/jquery.matchHeight.min.js?ver=3.2.2

FAILED - https://fearby.com/wp-content/plugins/contact-form-7/includes/css/styles.css?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/scripts/fitvids.min.js?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-includes/js/wp-embed.min.js?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/plugins/bj-lazy-load/js/bj-lazy-load.min.js?ver=2

FAILED - https://fearby.com/wp-content/plugins/disqus-comment-system/media/js/count.js?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-includes/js/wp-emoji-release.min.js?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/plugins/wp-seo-html-sitemap/style.css?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/symbol-defs.svg

FAILED - https://fearby.com/wp-includes/css/dashicons.min.css?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/themes/news-pro/images/favicon.ico

FAILED - https://fearby.com/wp-content/plugins/genesis-tabs/style.css?ver=230d722825ddde2688088d563a906075

FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/css/style.css?ver=2.0.1

FAILED - https://fearby.com/wp-content/plugins/add-to-any/addtoany.min.css?ver=1.14

It would be a huge effort trying to read and keep static plugin and theme related files in a CDN. I’ll ask the EWWW.io developer to see if this is possible in a future version (that would be nice). The developer did promptly point me here to opt into using the CDN to deliver CSS, JS etc.

Is that it? It can’t be that simple!

I can load my site at https://fearby.com and my images load from https://fearby-com.exactdn.com 🙂 I am impressed, blog posts are now loading images from the CDN network and I did not have to edit posts. I did have to subscribe to ExactDN and tick a checkbox in WordPress though.

Here is a sample image (my largest) from this blog post.

https://fearby-com.exactdn.com/wp-content/uploads/2017/10/Infographic-So-you-have-an-idea-for-an-app-v1-3.jpg?strip=all&quality=60&ssl=1

1	https://fearby-com.exactdn.com/wp-content/uploads/2017/10/Infographic-So-you-have-an-idea-for-an-app-v1-3.jpg?strip=all&quality=60&ssl=1

fearby.com WebPAGE TEST Results – Singapore

Web Page Performace Test Results – Singapore (After Setting up the EWWW.io ExactDN):

Load Time: 4.177s (down 4.646s from previous optimizations ((My Original load times were 23.672s))
First Byte: 2.240s (down 1.3133s from previous optimizations (My Original First Byte: 6.743s))
Start Render: 2.800s (down 2.68s from previous optimizations (My Original Start Render: 11.8300s))
Speed Index: 3009 (down 2585 from previous optimizations (Speed Index: 15024))
Requests: 67/68 (Document complete v Fully Loaded)
Bytes: 535KB/538KB (Document complete v Fully Loaded)

Quick Web Page Performace Score Card – Singapore (After Setting up the EWWW.io ExactDN):

First Byte: F (I should have captured the subscores).
Keep Alive Enabled: A (I should have captured the subscores).
Compress Transfer: A (I should have captured the subscores).
Compress Images: A (I should have captured the subscores).
Cache Static Content: B (I should have captured the subscores).
Effective Use of CDN: X (I should have captured the subscores). plugins now need to be on a CDN.

The Web Page Test site does give detailed scores and recommendations if you scroll down or click the score car (A-F). Do read the recommendations and see what you may need to do next. I am happy that I now have a CDN via EWWW.io. Clicking the first byte and CDN buttons at Web Page Test reveal sub-scores to allow you to see if you have made improvements, regrettably, I did not know of and capture sub-scores until after installing the CDN (I suggest you do).

Full Dump.

Details
First Byte Time (back-end processing): 0/100
2240 ms First Byte Time
ms Target First Byte Time

Use persistent connections (keep alive): 100/100

Use gzip compression for transferring compressable responses: 100/100
404.6 KB total in compressible text, target size = 404.6 KB - potential savings = 0.0 KB

Compress Images: 100/100
98.7 KB total in images, target size = 98.7 KB - potential savings = 0.0 KB

Use Progressive JPEGs: 100/100
97.4 KB of a possible 97.4 KB (100%) were from progressive JPEG images

Leverage browser caching of static assets: 88/100
FAILED - (No max-age or expires) - https://fearby-com.disqus.com/count.js
FAILED - (8.2 minutes) - https://www.google-analytics.com/analytics.js
FAILED - (15.0 minutes) - https://www.googletagmanager.com/gtag/js?id=UA-93963-1
FAILED - (60.0 minutes) - https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js
WARNING - (1.1 hours) - https://www.google-analytics.com/ga.js
WARNING - (12.0 hours) - https://pagead2.googlesyndication.com/pub-config/r20160913/ca-pub-9241521190070921.js
WARNING - (24.0 hours) - https://fonts.googleapis.com/css?family=Raleway%3A400%2C700&ver=3.2.2
WARNING - (2.0 days) - https://static.addtoany.com/menu/page.js
WARNING - (4.5 days) - https://s.ytimg.com/yts/jsbin/www-widgetapi-vflUJbESo/www-widgetapi.js

Use a CDN for all static assets: 46/100
FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/css/style.css?ver=2.0.1
FAILED - https://fearby.com/wp-content/plugins/add-to-any/addtoany.min.css?ver=1.14
FAILED - https://fearby.com/wp-content/plugins/crayon-syntax-highlighter/css/min/crayon.min.css?ver=_2.7.2_beta
FAILED - https://fearby.com/wp-content/themes/news-pro/style.css?ver=3.2.2
FAILED - https://fearby.com/wp-content/plugins/contact-form-7/includes/css/styles.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/wp-seo-html-sitemap/style.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/css/dashicons.min.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/contact-form-7/includes/js/scripts.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/js/comment-reply.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/crayon-syntax-highlighter/js/min/crayon.min.js?ver=_2.7.2_beta
FAILED - https://fearby.com/wp-includes/js/hoverIntent.min.js?ver=1.8.1
FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/menu/superfish.js?ver=1.7.5
FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/menu/superfish.args.js?ver=2.5.3
FAILED - https://fearby.com/wp-content/themes/genesis/lib/js/skip-links.js?ver=2.5.3
FAILED - https://fearby.com/wp-content/themes/news-pro/js/jquery.matchHeight.min.js?ver=3.2.2
FAILED - https://fearby.com/wp-content/themes/news-pro/js/responsive-menus.min.js?ver=3.2.2
FAILED - https://fearby.com/wp-includes/js/jquery/ui/tabs.min.js?ver=1.11.4
FAILED - https://fearby.com/wp-content/plugins/add-to-any/addtoany.min.js?ver=1.0
FAILED - https://fearby.com/wp-content/themes/news-pro/js/global.js?ver=3.2.2
FAILED - https://fearby.com/wp-includes/js/jquery/ui/core.min.js?ver=1.11.4
FAILED - https://fearby.com/wp-includes/js/jquery/ui/widget.min.js?ver=1.11.4
FAILED - https://fearby.com/wp-content/plugins/bj-lazy-load/js/bj-lazy-load.min.js?ver=2
FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/scripts/fitvids.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/js/wp-embed.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/disqus-comment-system/media/js/count.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/styles/ytprefs.min.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/js/wp-emoji-release.min.js?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/symbol-defs.svg
FAILED - https://fearby.com/wp-content/plugins/genesis-tabs/style.css?ver=230d722825ddde2688088d563a906075
FAILED - https://fearby.com/wp-includes/js/jquery/jquery.js?ver=1.12.4
FAILED - https://fearby.com/wp-content/themes/news-pro/images/favicon.ico
FAILED - https://fearby.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1
FAILED - https://fearby.com/wp-content/plugins/simple-social-icons/svgxuse.js?ver=1.1.21
FAILED - https://fearby.com/wp-content/plugins/youtube-embed-plus/scripts/ytprefs.min.js?ver=230d722825ddde2688088d563a906075

CDN's Used:
fearby.com : 
fonts.googleapis.com : Google
fonts.gstatic.com : Google
www.googletagmanager.com : Google
static.addtoany.com : Cloudflare
fearby-com.exactdn.com : KeyCDN
pagead2.googlesyndication.com : Google
www.youtube.com : Google
www.google-analytics.com : Google
adservice.google.com : Google
adservice.google.com.sg : Google
googleads.g.doubleclick.net : Google
s.ytimg.com : Google
fearby-com.disqus.com : Fastly
stats.g.doubleclick.net : Google

Details

First Byte Time (back-end processing): 0/100

2240 ms First Byte Time

ms Target First Byte Time

Use persistent connections (keep alive): 100/100

Use gzip compression for transferring compressable responses: 100/100

404.6 KB total in compressible text, target size = 404.6 KB - potential savings = 0.0 KB

Compress Images: 100/100

98.7 KB total in images, target size = 98.7 KB - potential savings = 0.0 KB

Use Progressive JPEGs: 100/100

97.4 KB of a possible 97.4 KB (100%) were from progressive JPEG images

Leverage browser caching of static assets: 88/100

FAILED - (No max-age or expires) - https://fearby-com.disqus.com/count.js

FAILED - (8.2 minutes) - https://www.google-analytics.com/analytics.js

FAILED - (15.0 minutes) - https://www.googletagmanager.com/gtag/js?id=UA-93963-1

FAILED - (60.0 minutes) - https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js

WARNING - (1.1 hours) - https://www.google-analytics.com/ga.js

WARNING - (12.0 hours) - https://pagead2.googlesyndication.com/pub-config/r20160913/ca-pub-9241521190070921.js

WARNING - (24.0 hours) - https://fonts.googleapis.com/css?family=Raleway%3A400%2C700&ver=3.2.2

WARNING - (2.0 days) - https://static.addtoany.com/menu/page.js

WARNING - (4.5 days) - https://s.ytimg.com/yts/jsbin/www-widgetapi-vflUJbESo/www-widgetapi.js

Use a CDN for all static assets: 46/100