• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Create a VM ($25 Credit)
  • Buy a Domain
  • 1 Month free Back Blaze Backup
  • Other Deals
    • Domain Email
    • Nixstats Server Monitoring
    • ewww.io Auto WordPress Image Resizing and Acceleration
  • About
  • Links

IoT, Code, Security, Server Stuff etc

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

  • Cloud
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to buy a new domain and SSL cert from NameCheap, a Server from Digital Ocean and configure it.
    • Setting up a Vultr VM and configuring it
    • All Cloud Articles
  • Dev
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to setup pooled MySQL connections in Node JS that don’t disconnect
    • NodeJS code to handle App logins via API (using MySQL connection pools (1000 connections) and query parameters)
    • Infographic: So you have an idea for an app
    • All Development Articles
  • MySQL
    • Using the free Adminer GUI for MySQL on your website
    • All MySQL Articles
  • Perf
    • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap
    • All Performance Articles
  • Sec
    • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
    • Using OWASP ZAP GUI to scan your Applications for security issues
    • Setting up the Debian Kali Linux distro to perform penetration testing of your systems
    • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
    • PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API
    • Setting strong SSL cryptographic protocols and ciphers on Ubuntu and NGINX
    • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
    • All Security Articles
  • Server
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All Server Articles
  • Ubuntu
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Useful Linux Terminal Commands
    • All Ubuntu Articles
  • VM
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All VM Articles
  • WordPress
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
    • How to backup WordPress on a host that has CPanel
    • Moving WordPress to a new self managed server away from CPanel
    • Moving a CPanel domain with email to a self managed VPS and Gmail
    • All WordPress Articles
  • All

SEO

Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

June 5, 2018 by Simon

How can you measure VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 2 of 4

Read Part 1, Part 2, Part 3 or Part 4

Measure Disk Performance with Bonnie++

Installing Bonnie++ on Ubuntu

apt-get install bonnie++

Read this. post on using Bonnie++

Benchmark disk IO with DD and Bonnie++

Starting Bonnie++

bonnie++ -d /tmp -r 2048 -u username

Bonnie++ Readme.

Disk io with bonnie++ on Vultr/Sydney

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
servername 4G 656 99 308954 68 113706 33 1200 92 188671 30 10237 251
Latency 26067us 119ms 179ms 29139us 26069us 16118us
Version 1.97 ------Sequential Create------ --------Random Create--------
servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 1463us 703us 880us 263us 119us 593us
1.97,1.97,servername,1,1528177870,4G,,656,99,308954,68,113706,33,1200,92,188671,30,10237,251,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,26067us,119ms,179ms,29139us,26069us,16118us,1463us,703us,880us,263us,119us,593us

Disk io with bonnie++ on Digital Ocean/London

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
servername 4G 699 99 778636 74 610414 60 1556 99 1405337 59 +++++ +++
Latency 17678us 10099us 17014us 7027us 3067us 2366us
Version 1.97 ------Sequential Create------ --------Random Create--------
servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 1243us 376us 611us 108us 59us 181us
1.97,1.97,servername,1,1528186398,4G,,699,99,778636,74,610414,60,1556,99,1405337,59,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,17678us,10099us,17014us,7027us,3067us,2366us,1243us,376us,611us,108us,59us,181us

Disk io with bonnie++ on UpCloud/Singapore

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
servername 4G 1014 99 407179 24 366622 32 2137 99 451886 17 +++++ +++
Latency 11297us 54232us 16443us 4949us 44883us 1595us
Version 1.97 ------Sequential Create------ --------Random Create--------
servername -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 264us 340us 561us 138us 66us 327us
1.97,1.97,servername,1,1528226703,4G,,1014,99,407179,24,366622,32,2137,99,451886,17,+++++,+++,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,11297us,54232us,16443us,4949us,44883us,1595us,264us,340us,561us,138us,66us,327us

Now read this site on how to make sense of this data

< Previous – Next >

Read Part 1, Part 2, Part 3 or Part 4

Filed Under: CDN, Cloud, Cloudflare, Digital Ocean, disk, Domain, ExactDN, HTTPS, Performance, PHP, php72, Scalability, Scalable, SEO, Ubuntu, UI, UpCloud, VM, Vultr, Wordpress Tagged With: and, can, comparing, Concurrent Users etc, cpu, Digital Ocean and UpCloud - Part 2 of 4, Disk, How, Latency, measure, on, Performance, ubuntu, vm, vultr, you

Setting strong SSL cryptographic protocols and ciphers on Ubuntu and NGINX

May 15, 2018 by Simon

This guide will aim to inform you of strong cryptographic protocols and ciphers to use on a web server on Ubuntu 16.04 and NGINX.

Secure encryption protocols are used to secure communications between a server and client. Older SSL protocols like Netscape’s Secure Sockets Layer (SSL) are flagged as DO NOT USE use by the Internet Engineering Task Force (IETF). Newer protocols like Transport Layer Security (TLS) are the newer recommended SSL protocols to use.

Wikipedia Article on Cryptographic Protocol’s

A security protocol (cryptographic protocol or encryption protocol) is an abstract or concrete protocol that performs a security-related function and applies cryptographic methods, often as sequences of cryptographic primitives. A protocol describes how the algorithms should be used. A sufficiently detailed protocol includes details about data structures and representations, at which point it can be used to implement multiple, interoperable versions of a program.

Wikipedia on Ciphers

In cryptography, a cipher (or cypher) is an algorithm for performing encryption or decryption—a series of well-defined steps that can be followed as a procedure. An alternative, less common term is encipherment. To encipher or encode is to convert information into cipher or code. In common parlance, “cipher” is synonymous with “code,” as they are both a set of steps that encrypt a message; however, the concepts are distinct in cryptography, especially classical cryptography.

Wikipedia article on Elliptic-curve cryptography

Wikipedia article on Diffie–Hellman key exchange

Bad SSL Assumptions I have heard for not using HTTPS

  • I am not a bank so I don’t need HTTPS
  • SSL overhead is was too high on servers.
  • My site only has static content, I don’t need HTTPS
  • I don’t need SSL to secure my site I just need to be less of a target than others
  • I don’t hold confidential information (Wrong)

Don’t be Lazy and secure a site poorly

A local business that wanted me to buy their goods is not convincing me.

Bad SSL

(tested with SSL labs and asafaweb)

Why SSL

If you are unsure of why you need SSL visit https://doesmysiteneedhttps.com/, Avoiding the Not Secure Warning in Chrome, Why HTTPS matters and securing your site with HTTPS.

Google has an HTTPS usage graph for all communications to its services (hint it’s growing): https://transparencyreport.google.com/https/overview?hl=en

SSL Usage

SSL Future

SSL is here to stay, Non-SSL sites will soon be labelled insecure, Non-SSL sites will have Search Engine Optimization (SEO) adversely affected.

http insecure

Also, secure pages will be treated as normal (not flagged as secure)

In October, Chrome will remove the “secure” indicator on all HTTPS pages and mark pages that do no use the secure version of the HTTP protocol with a red “not secure” warning. This change will make the web safer to use by default. https://t.co/ar3lwB9aRt

— J-François Lavigne (@jflavigne) May 25, 2018

History of Protocol’s – Launch Dates

  • SSL 1.0 (never launched)
  • SSL 2.0 1995
  • SSL 3.0 1996
  • TLS 1.0 1999
  • TLS 1.1 2006
  • TLS 1.2 2008
  • TLS 1.3 2018

Sites like https://caniuse.com can show you if our browser can use new protocols like TLS (e.g TLS 1.0, TLS 1.1, TLS 1.2, TLS 1.3)

  • TLS 1.0 is supported by All Browsers
  • TLS 1.1 is supported on IE11+, Edge, Firefox 24+, Chrome 22+, Safari 7+, Opera 12.1+, iOS Safari 5.1+, Chrome 62 on Android 5+ etc
  • TLS 1.2 is supported on IE11+, Edge, Firefox 27, Chrome 30+, Safari 7+, Opera 17+, iOS Safari 5.1, Chrome 62 on Android 5+ etc
  • TLS 1.3 is not supported by IE, Edge, Safari, iOS Safari, Android but is supported by Firefox 52, Chrome 56, Opera 43.

TLS 1.3

I have a guide here on setting up TLS 1.3 on Ubuntu 16.05 and Chrome, I use the draft build of OpenSSL but Open SSL 1.1.1 will support TLS 1.3. I am still figuring our TLS 1.3 on Ubuntu 18.04.

At the time of writing, you need to opt into TLS 1.3 draft specification in Chrome.

Enable TLS in Chrome

Cypher or Cypher

Read this page to see the history of the word Cipher or Cypher?

Buying an SSL certificate

Opening your wallet may not buy you the best certificate either, this was an SSL Labs review of a $150 SSL certificate Ii purchased a few years ago from a CPanel web host.

Bad CPanel SSL Certificate

I don’t buy commercial certificates anymore, I prefer free SSL certificates from Lets Encrypt

SSL Strength

I prefer to set up my own (free) SSL certificate with Lest Encrypt and tets those certificated with https://dev.ssllabs.com/ssltest/

You can configure your web server to only use certain protocols.

ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.2 TLSv1.3;

And define preferred ciphers

ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256;

SSL Test 2018

Don’t forget to renew your SSL certificates ahead of time.

Also run a modern browser like Google Chrome Canary as some old browsers thnk expired SSL certificates are Secure

Ciphers

OpenSSL has implemented support for five TLS v1.3 cipher suites:

  • TLS13-AES-256-GCM-SHA384
  • TLS13-CHACHA20-POLY1305-SHA256
  • TLS13-AES-128-GCM-SHA256
  • TLS13-AES-128-CCM-8-SHA256
  • TLS13-AES-128-CCM-SHA256

Test OpenSSL Cipher Suites

openssl ciphers -s -v
TLS_AES_256_GCM_SHA384  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(256) Mac=AEAD
TLS_CHACHA20_POLY1305_SHA256 TLSv1.3 Kx=any      Au=any  Enc=CHACHA20/POLY1305(256) Mac=AEAD
TLS_AES_128_GCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=ECDH     Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
DHE-RSA-CHACHA20-POLY1305 TLSv1.2 Kx=DH       Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH       Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
DHE-RSA-AES256-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-RSA-AES128-SHA256   TLSv1.2 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256
ECDHE-ECDSA-AES256-SHA  TLSv1 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
ECDHE-RSA-AES256-SHA    TLSv1 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
ECDHE-ECDSA-AES128-SHA  TLSv1 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA1
ECDHE-RSA-AES128-SHA    TLSv1 Kx=ECDH     Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
AES256-GCM-SHA384       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD
AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
AES128-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA256
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1

A handy guide about using ciphers

SSL/TLS: How to choose your cipher suite

Testing a remote host’s ciphers and protocols with cipherscan

Clone this repository: https://github.com/mozilla/cipherscan

Scan a site

./cipherscan fearby.com

Result

Target: fearby.com:443

prio  ciphersuite                        protocols  pfs                 curves
1     ECDHE-ECDSA-CHACHA20-POLY1305-OLD  TLSv1.2    ECDH,P-256,256bits  prime256v1
2     ECDHE-ECDSA-AES128-GCM-SHA256      TLSv1.2    ECDH,P-256,256bits  prime256v1
3     ECDHE-ECDSA-AES128-SHA             TLSv1.2    ECDH,P-256,256bits  prime256v1
4     ECDHE-ECDSA-AES128-SHA256          TLSv1.2    ECDH,P-256,256bits  prime256v1
5     ECDHE-ECDSA-AES256-GCM-SHA384      TLSv1.2    ECDH,P-256,256bits  prime256v1
6     ECDHE-ECDSA-AES256-SHA             TLSv1.2    ECDH,P-256,256bits  prime256v1
7     ECDHE-ECDSA-AES256-SHA384          TLSv1.2    ECDH,P-256,256bits  prime256v1

Certificate: trusted, 256 bits, ecdsa-with-SHA256 signature
TLS ticket lifetime hint: 64800
NPN protocols: h2,http/1.1
OCSP stapling: supported
Cipher ordering: server
Curves ordering: server - fallback: no
Server supports secure renegotiation
Server supported compression methods: NONE
TLS Tolerance: yes

Intolerance to:
 SSL 3.254           : absent
 TLS 1.0             : PRESENT
 TLS 1.1             : PRESENT
 TLS 1.2             : absent
 TLS 1.3             : absent
 TLS 1.4             : absent

Cipher scan can also recommend settings to change to help you harden a server (based on https://wiki.mozilla.org/Security/Server_Side_TLS)

Analyze Command

./analyze.py -t fearby.com

Results

fearby.com:443 has bad ssl/tls

Things that are bad:
* remove cipher ECDHE-ECDSA-CHACHA20-POLY1305-OLD

Changes needed to match the old level:
* remove cipher ECDHE-ECDSA-CHACHA20-POLY1305-OLD
* enable TLSv1.1
* enable TLSv1
* enable SSLv3
* add cipher DES-CBC3-SHA
* use a certificate with sha1WithRSAEncryption signature
* use DHE of 1024bits and ECC of 160bits

Changes needed to match the intermediate level:
* remove cipher ECDHE-ECDSA-CHACHA20-POLY1305-OLD
* consider enabling TLSv1.1
* consider enabling TLSv1
* add cipher AES128-SHA
* use a certificate signed with sha256WithRSAEncryption

Changes needed to match the modern level:
* remove cipher ECDHE-ECDSA-CHACHA20-POLY1305-OLD
* remove cipher ECDHE-ECDSA-AES128-SHA
* remove cipher ECDHE-ECDSA-AES256-SHA

More info on hardening here.

TLS 1.3 Information

More Reading

SSLLabs Grading of certificates

Read about SSL Labs grading here

snip from here

  • A+ – exceptional configuration
  • A – strong commercial security
  • B – adequate security with modern clients, with older and potentially obsolete crypto used with older clients; potentially smaller configuration problems
  • C – obsolete configuration, uses obsolete crypto with modern clients; potentially bigger configuration problems
  • D – configuration with security issues that are typically difficult or unlikely to be exploited, but can and should be addressed
  • E – unused
  • F – exploitable and/or patchable problems, misconfigured server, insecure protocols, etc.

We wish to make clear that, while A+ is clearly the desired grade, both A and B grades are acceptable and result in adequate commercial security. The B grade, in particular, may be applied to configurations designed to support very wide audiences, many of whom use very old programs to connect. The C grade is generally used for configurations that don’t follow best practices. Grades D and F are used for servers with serious configuration and security issues.

REady to go SSL configuration: https://cipherli.st/

Download ready to go Diffie–Hellman primes. https://2ton.com.au/dhtool/

We have dedicated 48 CPU cores to the task of continuously generating 2048, 3072, 4096 and 8192 bit DH parameters, and the public service we present here allows access to the most-recent 128 of each.

Diffie–Hellman key exchange (DH) is a method of securely exchanging cryptographic keys over a public channel and was one of the first public-key protocols as originally conceptualized by Ralph Merkle and named after Whitfield Diffie and Martin Hellman. DH is one of the earliest practical examples of public key exchange implemented within the field of cryptography.

Traditionally, secure encrypted communication between two parties required that they first exchange keys by some secure physical channel, such as paper key lists transported by a trusted courier. The Diffie–Hellman key exchange method allows two parties that have no prior knowledge of each other to jointly establish a shared secret key over an insecure channel. This key can then be used to encrypt subsequent communications using a symmetric key cipher.

Diffie–Hellman is used to secure a variety of Internet services. However, research published in October 2015 suggests that the parameters in use for many DH Internet applications at that time are not strong enough to prevent compromise by very well-funded attackers, such as the security services of large governments.

More to come, I hope this guide helps someone.

fyi:

Windows Protocol/Cipher installer: https://www.nartac.com/

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

V1.2 expired and use a modern browser

v1.1 bad SSL

v1.0 Initial post

Filed Under: HTTPS, Security, SEO, TLS Tagged With: and ciphers, cryptographic, on Ubuntu and NGINX, protocols, Setting, ssl, strong

Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap

March 13, 2018 by Simon

This guide will show how you can set up a website to use Cloudflare on a VM hosted on Vultr and Namecheap

I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. This post will show how to let Cloudflare handle the DNS for the domain.

Update 2018: For the best performing VM host (UpCloud) read my guide on the awesome UpCloud VM hosts (get $25 free credit by signing up here).

Snip from here “Cloudflare’s enterprise-class web application firewall (WAF) protects your Internet property from common vulnerabilities like SQL injection attacks, cross-site scripting, and cross-site forgery requests with no changes to your existing infrastructure.”

Buy a Domain 

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Cloudflare Benefits (Free Plan)

  • DDoS Attack Protection (Huge network to absorb attacks DDoS attacks over 600Gbps are no problem for our 15 Tbps networks)
  • Global CDN
  • Shared SSL certificate (I disabled this and opted to use my own)
  • Access to audit logs
  • 3 page rules (maximum)

View paid plan options here.

Cloudflare CDN map

Cloudflare CDN says it can load assets up to 2x faster, 60% less bandwidth from your servers by delivering assets from 127 data centres.

Cloudflare Global Network

Setup

You will need to sign up at cloudflare.com

Cloudflare

After you create an account you will be prompted to add a siteAdd SiteCloudflare will pull your public DNS records to import.

Query DNS

You will be prompted to select a plan (I selected free)

Plan Select

Verify DNS settings to import.

DNS Import

You will now be asked to change your DNS nameservers with your domain reseller

DNS Nameservers

TIP: If you have an SSL cert (e.g Lets Encrypt) already setup head to the crypto section and select ” Full (Strict)” to prevent ERR_TOO_MANY_REDIRECTS errors.

Strict SSL

Cloudflare UI

I asked Twitter if they could kindly load my site so I could see if Cloudflare dashboard/stats were loading.

Could I kindly ask if you are reading this that you visit https://t.co/9x5TFARLCt, I am writing a @Cloudflare blog post and need to screenshot stats. Thanks in advance

— Simon Fearby (Developer) (@FearbySoftware) March 13, 2018

The Cloudflare CTO responded.  🙂

Sure thing 🙂

— John Graham-Cumming (@jgrahamc) March 13, 2018

Confirm Cloudflare link to a domain from the OSX Comand line

host -t NS fearby.com
fearby.com name server dane.ns.cloudflare.com.
fearby.com name server nora.ns.cloudflare.com.

Caching Rule

I set up the following caching rule to cache everything for 8 hours instead of WordPress pages

Page Rules

“fearby.com.com/wp-*” Cache level: Bypass

“fearby.com.com/wp-admin/post.php*” Cache level: Bypass

“fearby.com/*” Cache Everything, Edge Cache TTL: 8 Hours

Cache Results

Cache appears to be sitting at 50% after 12 hours.  having cache os dynamic pages out there is ok unless I need to fix a typo, then I need to login to Cloudflare and clear the cache manually (or wait 8 hours)

Performance after a few hours

DNS times in gtmetrix have now fallen to a sub 200ms (Y Slow is now a respectable A, it was a C before).  I just need to wait for caching and minification to kick in.

DNS Improved

webpagetest.org results are awesome

See here: https://www.webpagetest.org/result/180314_PB_7660dfbe65d56b94a60d7a604ca250b3/

  • Load Time: 1.80s
  • First Byte 0.176s
  • Start Render 1.200s

webpagetest

Google Page Speed Insights Report

Mobile: 78/100

Desktop: 87/100

Check with https://developers.google.com/speed/pagespeed/insights/

Update 24th March 2018 Attacked?

I noticed a spike in and traffic (incoming and threats) on the 24th of March 2018.

I logged into Cloudflare on my mobile device and turned on Under Attack Mode.

Under Attack Flow

Cloudflare was now adding a delay screen in the middle of my initial page load. Read more here.  A few hours after the Attach started it was over.

After the Attack

I looked at the bandwidth and found no increase in traffic from my initial host VM. Nice.

cloudflare-attack-001

Thanks, Cloudflare.

Cloudflare Pros

  • Enabling Attack mode was simple.
  • Soaked up an attack.
  • Free Tier
  • Many Reports
  • Option to force HTTPS over HTTP
  • Option to ban/challenge suspicious IP’s and set challenge timeframes.
  • Ability to setup IP firewall rules and Application Firewalls.
  • User-agent blocking
  • Lockdown URL’s to IP’s (pro feature)
  • Option to minify Javascript, CSS and HTML
  • Option to accelerate mobile links
  • Brotli compression on assets served.
  • Optio to enable BETA Rocket loader for Javascript performance tweaks.
  • Run Javascript service workers from the 120+ CDN’s
  • Page/URL rules o perform custom actions (redirects, skip cache, Encryption etc)
  • HTTP/2 on, IPV6 ON
  • Option to setup load balancing/failover
  • CTO of Cloudflare responded in Twitter 🙂
  • Option to enable rate limiting (charged at 10,000 hits for $0.05c)
  • Option to block countries (pro feature)
  • Option to install apps in Cloudflare like(Goole Analytics,

Cloudflare Cons

  • No more logging into NameCheap to perform DNS management (I now goto Cloudflare, Namecheap are awesome).
  • Cloudflare Support was slow/confusing (I ended up figuring out the redirect problem myself).
  • Some sort of verify Cloudflare Setup/DNS/CDN access would be nice. After I set this up my gtmetrix load times were the same and I was not sure if DNS needs to replicate? Changing minify settings in Cloudflare did not seem to happen.
  • WordPress draft posts are being cached even though page riles block wp-admin page caching.
  • Would be nice to have ad automatic Under Attack mode
  • Now all sub-domains were transferred in the setup ( id did not know for weeks)

Cloudflare status

Check out https://www.cloudflarestatus.com/ for status updates.

Don’t forget to install the CloudFlare Plugin for WordPress if you use WordPress.

More Reading

Check out my OWASP Zap and Kali Linux self-application Penetration testing posts.

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.8 host Command from the OSX CLI

v1.7 Subdomain error

v1.6 Cloudflare Attack

v1.5 WordPress Plugin

v1.4 More Reading

v1.3 added WAF snip

v1.2 Added Google Page Speed Insights and webpage rest results

v1.1 Added Y-Slow

v1.0 Initial post

Filed Under: Analytics, App, Cache, CDN, Cloud, Cloudflare, DNS, Domain, Hosting, LetsEncrypt, Marketing, Secure, Security, SEO, Server, VM, Vultr, Website, Wordpress Tagged With: a, and, Cloudflare, hosted, namecheap, on, Setting, to, up, use, vm, vultr, website

Manage Social Media posts with Buffer

October 10, 2017 by Simon

Here is a quick setup guide for Buffer.com where you can connect to and post (manually or scheduled) to multiple social media platforms.

You can view pricing here. You can signup for a  free Buffer Individual plan: https://buffer.com/signup. Signup to Buffer (Free, Limited)

Post Signup Setup

Connect Buffer to Social Media Platforms

Buffer SIgnup

Type post content

Schedule

Change the default image

Define an Image

Choose the platforms and images

Choose Platforms

Schedule the Post

Schedule

You can manually share to a platform at any time.

Share Now

TIP: If you share now you will need to manually share on each platform separately.

Results

Buffer Results

Buffer features I like

  • Good Free Plan
  • Post Scheduling
  • Image Creation Integration (Paid)
  • Reply integration (Paid)
  • Manage all your social accounts from one simple dashboard
  • Ability to set custom posting slots.

Buffer features I Don’t like

  • Manual Share to all feature missing.
  • Timezones earlier than US Timezones appear to be untouchable (my Timezone is set)
    Timezone

Buffer FAQ’s: https://faq.buffer.com/

Tip: Create custom posting slots

Custom Slots

More soon (reply automation and image creation).

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.1 Added Posting Slots Info

etc

Filed Under: Analitics, Analytics, Automation, Blog, Business, Marketing, Networking, SEO Tagged With: Buffer, Manage, Media, posts, Social

How to optimize your sites Search Engine Optimization (SEO) and grow customers without paying for Ads

September 9, 2017 by Simon

How to optimize your sites Search Engine Optimization (SEO) and grow customers without paying for Ads.

This guide is a shorter post around setting up SEO (Search Engine Optimization) and driving more traffic to your site without buying ADs.  In a nutshell, to have better SEO you need to jump some technical hurdles in order to drive more traffic to your site from search engines along with understanding your customer’s needs and making things easier for them.

I have blogged about these topics before but these posts are too long in reflection.

  • Setting up Google Analytics on your website
  • How to boost your site’s SEO
  • Improving the speed of WordPress
  • Digital marketing and user engagement 101
  • Add Google AdWords to your WordPress blog
  • etc

Buying Ad’s?

Facebook, Google, Bing and advertising agencies will recommend you set goals around growth and site traffic and pay for those goals to succeed (usually by advertisements).

Don’t get me wrong Advertising works but it is a competitive market, Online sites can easily setup the display of Ad’s on their site (my guide here Add Google AdWords to your WordPress blog, https://fearby.com/article/add-google-adwords-wordpress-blog/ ). You can buy physical billboard ad’s on the side of roads (e.g http://www.buythisspace.com.au/). I tried to enquire about the costs of a physical billboard but the agencies robot verification rejected my enquiry submission so I gave up.  Advertising is buying peoples times and people now how to avoid ad’s and not interact with them (7 Marketing Lessons from Eye-Tracking Studies https://blog.kissmetrics.com/eye-tracking-studies/)

Do more of what works

Spoiler: This guide will recommend you do more of what works over buying millions of ad’s and hoping for new and engaged customers and customer growth.

  • If you don’t already have Google Analytics setup on your site then do it, you cannot identify your customers or identify what is broken or in turn fix it (Setting up Google Analytics on your website, https://fearby.com/article/setting-up-google-analytics-on-your-website/ )
  • Monitor Data – Do review your logs and customer related data (review orders, customers and try and identify what works. Software like https://www.zoho.com/one/applications/web.html will help you connect the dots.
  • Adobe Audience Cloud: http://www.adobe.com/au/experience-cloud.html is a more expensive software suite for driving decisions based on data.
  • Benchmarks – Set goals and work toward them (e.g I want 10x more customers).

SEO Tip’s

This older article on  How to boost your site’s SEO  attempts to mention what you need to do it to get better SEO.

Do run a modern great site

I am a big fan of word of mouth over free/organic traffic over paid customers via advertising (Mostly because I am tight and realize advertising can be a bottomless pit). The single biggest thing you can do to have more organic traffic from search engines is run a modern and fast website, have valuable content and make it as easy for the customer as possible. This is why I moved my site and setup an SSL certificate (link to article).

Search engines like your site to be fast, updated frequently, have sitemaps to make their jobs easier and have an SSL certificate to keep the web safe etc.

Google, Bing and other search engines will not send traffic your way if you do not satisfy them that your site is liked or has valuable content.  Google makes money from Google Analytics by helping people understand their site’s visitors then recommend you pay for ad’s to use on sites that have AdWords on their site ( WordPress to a new self-managed server away from CPanel ).

  • How to boost your site’s SEO https://fearby.com/article/how-to-boost-your-sites-seo/
  • Your website needs to be fast, use sites like https://www.webpagetest.org to measure how fast your site is (Aim for all A’s). Read this page for information on the impact of slow websites https://www.searchenginejournal.com/mobile-page-speed-benchmarks/194511/
  • Mobile friendly – Ensure your site is mobile friendly (or risk being dropped from search engine results)
  • SSL – Do have a secure SSL certificate on your website (view mine here https://www.ssllabs.com/ssltest/analyze.html?d=www.fearby.com&s=45.63.29.217&latest).
  • Incoming links – Having incoming links to your site tell search engines that your site is popular. 

Traffic Source types

  • Organic – An organic visitor to your site is one who found your site by searching something that was relevant to their search term and not by clicking on an advertisement.
  • Paid – A paid user is someone who has clicked an ad to come to your site.
  • Social – A social visitor is one who is known to come from a social media site, using social media sites like Twitter, Facebook or Instagram is a must to driving organic traffic (go where the people are).

Engagement

How engaged are your customers?  Have you asked your customers recently what they value or appreciate about your business or product? Have you asked for feedback recently?

User Engagement Levels

  • None – Do you have landing pages that quickly inform customers of your products or services?
  • Low – What do they need to know about your product or service?
  • Medium – Aware (engaged)
  • High – Can this person be an advocate for your business?
  • Gone – Did you get exit Feedback?

Ways to engage already engaged customers.

  • Setup a free MailChimp Newsletter to allow willing people to be alerted of new communication https://login.mailchimp.com/signup/?source=website&pid=GAW
  • Web Browser popup Alerts can be a great way to engage with users when new content is added to your site (Read the guide here https://documentation.onesignal.com/docs/web-push-setup )
  • Mobile apps or mobile friendly website are a no brainer given 2 billion people use mobile phones ( http://www.smartinsights.com/mobile-marketing/mobile-marketing-analytics/mobile-marketing-statistics/ ).

What can you do to help understand your customer’s needs and make their purchase processes easier?

Why are your customers leaving?

Understand more about your customers reasons for leaving and act upon preventing others from leaving.

  • Trying something new (Does your website need to be simpler?)
  • Are your products too expensive?.
  • Your site (or ordering) is not convenient (Do you need to setup online ordering/subscriptions and delivery?)
  • etc

Who are your customers

  • Personas – Do setup customer personas in order to focus on your customer segments (get a free customer persona template here https://blog.hubspot.com/blog/tabid/6307/bid/33491/everything-marketers-need-to-research-create-detailed-buyer-personas-template.aspx )
  • Does your website match these personas?

Are your customers.

  • Engaged
  • Informed
  • Advocates

Feedback

  • Do you have feedback loops (A simple feedback form can solve this)?

What do you know about your customers?

  • Product Satisfaction
  • Product Loyalty
  • Product Awareness

Paid Traffic (Ad’s)

  • Google Ad’s – Signup Here http://www.google.com.au/adwords/get-started/
  • Bing – Advertise on Bing here https://advertise.bingads.microsoft.com/
  • Facebook – Advertise on Facebook here https://www.facebook.com/business/products/ads

Free Traffic (SEO + Organic Ad’s)

  • Blog Posts (Sharing value/passion)
  • Social Media Posts (use hashtags)
  • Instagram (Post value/passion)

Most importantly Do what works (Measure and replicate).

Focus on Business Value

Generate a  SWOT Analysis ( Free tool here https://xtensio.com/ )

  • What are your Strengths?
  • What are your Weaknesses?
  • What are your Opportunities?
  • What are your Threats?

Goals

Goals allow you to investigate, learn, act and measure I order to improve.

  • Investigate – Data.
  • Learn/Insight – Make Assumptions.
  • Act – Act and measure.

Read more about customer engagement here https://en.wikipedia.org/wiki/Customer_engagement

Bonus

 Do ensure your website is compliant with accessibility and technical standards

  • Test our sites Accessibility – https://achecker.ca/checker/index.php
  • Test your sites HTML5 Compliance – https://validator.w3.org
  • Test your Google PageSpeed Test – https://developers.google.com/speed/pagespeed/insights/
  • Do A B testing to determine the statistical significance of changes to your site.

Conclusion

The more you know the better you can connect, Do set goals and as a minimum setup Google Analytics, SSL certificate and submit your site to search engines, then focus on a fast site that makes things simple for your customers.

Donate and make this blog better



Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.0 Initial version

Filed Under: Ads, Analytics, Business, LetsEncrypt, SEO, ssl, Website Tagged With: analytics, seo, ssl

Securing Ubuntu in the cloud

August 9, 2017 by Simon

It is easy to deploy servers to the cloud within a few minutes, you can have a cloud-based server that you (or others can use). ubuntu has a great guide on setting up basic security issues but what do you need to do.

If you do not secure your server expects it to be hacked into. Below are tips on securing your cloud server.

First, read more on scanning your server with Lynis security scan.

Always use up to date software

Always use update software, malicious users can detect what software you use with sites like shodan.io (or use port scan tools) and then look for weaknesses from well-published lists (e.g WordPress, Windows, MySQL, node, LifeRay, Oracle etc). People can even use Google to search for login pages or sites with passwords in HTML (yes that simple).  Once a system is identified by a malicious user they can send automated bots to break into your site (trying millions of passwords a day) or use tools to bypass existing defences (Security researcher Troy Hunt found out it’s child’s play).

Portscan sites like https://mxtoolbox.com/SuperTool.aspx?action=scan are good for knowing what you have exposed.

You can also use local programs like nmap to view open ports

Instal nmap

sudo apt-get install nmap

Find open ports

nmap -v -sT localhost

Starting Nmap 7.01 ( https://nmap.org ) at 2017-08-08 23:57 AEST
Initiating Connect Scan at 23:57
Scanning localhost (127.0.0.1) [1000 ports]
Discovered open port 80/tcp on 127.0.0.1
Discovered open port 3306/tcp on 127.0.0.1
Discovered open port 22/tcp on 127.0.0.1
Discovered open port 9101/tcp on 127.0.0.1
Discovered open port 9102/tcp on 127.0.0.1
Discovered open port 9103/tcp on 127.0.0.1
Completed Connect Scan at 23:57, 0.05s elapsed (1000 total ports)
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00020s latency).
Not shown: 994 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
3306/tcp open  mysql
9101/tcp open  jetdirect
9102/tcp open  jetdirect
9103/tcp open  jetdirect

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds
           Raw packets sent: 0 (0B) | Rcvd: 0 (0B)

Limit ssh connections

Read more here.

Use ufw to set limits on login attempts

sudo ufw limit ssh comment 'Rate limit hit for openssh server'

Only allow known IP’s access to your valuable ports

sudo ufw allow from 123.123.123.123/32 to any port 22

Delete unwanted firewall rules

sudo ufw status numbered
sudo ufw delete 8

Only allow known IP’s to certain ports

sudo ufw allow from 123.123.123.123 to any port 80/tcp

Also, set outgoing traffic to known active servers and ports

sudo ufw allow out from 123.123.123.123 to any port 22

Don’t use weak/common Diffie-Hellman key for SSL certificates, more information here.

openssl req -new -newkey rsa:4096 -nodes -keyout server.key -out server.csr
 
Generating a 4096 bit RSA private key
...

More info on generating SSL certs here and setting here and setting up Public Key Pinning here.

Intrusion Prevention Software

Do run fail2ban: Guide here https://www.linode.com/docs/security/using-fail2ban-for-security

I use iThemes Security to secure my WordPress and block repeat failed logins from certain IP addresses.

iThemes Security can even lock down your WordPress.

You can set iThemes to auto lock out users on x failed logins

Remember to use allowed whitelists though (it is so easy to lock yourself out of servers).

Passwords

Do have strong passwords and change the root password provided by the hosts. https://howsecureismypassword.net/ is a good site to see how strong your password is from brute force password attempts. https://www.grc.com/passwords.htm is a good site to obtain a strong password.  Do follow Troy Hunt’s blog and twitter account to keep up to date with security issues.

Configure a Firewall Basics

You should install a firewall on your Ubuntu and configure it and also configure a firewall with your hosts (e.g AWS, Vultr, Digital Ocean).

Configure a Firewall on AWS

My AWS server setup guide here. AWS allow you to configure the firewall here in the Amazon Console.

Type Protocol Port Range Source Comment
HTTP TCP 80 0.0.0.0/0 Opens a web server port for later
All ICMP ALL N/A 0.0.0.0/0 Allows you to ping
All traffic ALL All 0.0.0.0/0 Not advisable long term but OK for testing today.
SSH TCP 22 0.0.0.0/0 Not advisable, try and limit this to known IP’s only.
HTTPS TCP 443 0.0.0.0/0 Opens a secure web server port for later

Configure a Firewall on Digital Ocean

Configuring a firewall on Digital Ocean (create a $5/m server here).  You can configure your Digital Ocean droplet firewall by clicking Droplet, Networking then Manage Firewall after logging into Digital Ocean.

Configure a Firewall on Vultr

Configuring a firewall on Vultr (create a $2.5/m server here).

Don’t forget to set IP rules for IPV4 and IPV6, Only set the post you need to allow and ensure applications have strong passwords.

Ubuntu has a firewall built in (documentation).

sudo ufw status

Enable the firewall

sudo ufw enable

Adding common ports

sudo ufw allow ssh/tcp
sudo ufw logging on
sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 53
sudo ufw allow 443
sudo ufw allow 873
sudo ufw enable
sudo ufw status
sudo ufw allow http
sudo ufw allow https

Add a whitelist for your IP (use http://icanhazip.com/ to get your IP) to ensure you won’t get kicked out of your server.

sudo ufw allow from 123.123.123.123/24 to any port 22

More help here.  Here is a  good guide on ufw commands. Info on port numbers here.

https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

If you don’t have a  Digital Ocean server for $5 a month click here and if a $2.5 a month Vultr server here.

Backups

rsync is a good way to copy files to another server or use Bacula

sudo apt install bacula

Basics

Initial server setup guide (Digital Ocean).

Sudo (admin user)

Read this guide on the Linux sudo command (the equivalent if run as administrator on Windows).

Users

List users on an Ubuntu OS (or compgen -u)

cut -d: -f1 /etc/passwd

Common output

cut -d: -f1 /etc/passwd
root
daemon
bin
sys
sync
games
man
lp
mail
news
uucp
proxy
www-data
backup
list
irc
gnats
nobody
systemd-timesync
systemd-network
systemd-resolve
systemd-bus-proxy
syslog
_apt
lxd
messagebus
uuidd
dnsmasq
sshd
pollinate
ntp
mysql
clamav

Add User

sudo adduser new_username

e.g

sudo adduser bob
Adding user `bob' ...
Adding new group `bob' (1000) ...
Adding new user `bob' (1000) with group `bob' ...
Creating home directory `/home/bob' ...
etc..

Add user to a group

sudo usermod -a -G MyGroup bob

Show users in a group

getent group MyGroup | awk -F: '{print $4}'

This will show users in a group

Remove a user

sudo userdel username
sudo rm -r /home/username

Rename user

usermod -l new_username old_username

Change user password

sudo passwd username

Groups

Show all groups

compgen -ug

Common output

compgen -g
root
daemon
bin
sys
adm
tty
disk
lp
mail
proxy
sudo
www-data
backup
irc
etc

You can create your own groups but first, you must be aware of group ids

cat /etc/group

Then you can see your systems groups and ids.

Create a group

groupadd -g 999 MyGroup

Permissions

Read this https://help.ubuntu.com/community/FilePermissions

How to list users on Ubuntu.

Read more on setting permissions here.

Chmod help can be found here.

Install Fail2Ban

I used this guide on installing Fail2Ban.

apt-get install fail2ban

Check Fail2Ban often and add blocks to the firewall of known bad IPs

fail2ban-client status

Best practices

Ubuntu has a guide on basic security setup here.

Startup Processes

It is a good idea to review startup processes from time to time.

sudo apt-get install rcconf
sudo rcconf

Accounts

  • Read up on the concept of least privilege access for apps and services here.
  • Read up on chmod permissions.

Updates

Do update your operating system often.

sudo apt-get update
sudo apt-get upgrade

Minimal software

Only install what software you need

Exploits and Keeping up to date

Do keep up to date with exploits and vulnerabilities

  • Follow 0xDUDE on twitter.
  • Read the GDI.Foundation page.
  • Visit the Exploit Database
  • Vulnerability & Exploit Database
  • Subscribe to the Security Now podcast.

Secure your applications

  • NodeJS: Enable logging in applications you install or develop.

Ban repeat Login attempts with FailBan

Fail2Ban config

sudo nano /etc/fail2ban/jail.conf
[sshd]

enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 3

Hosts File Hardening

sudo nano /etc/host.conf

Add

order bind,hosts
nospoof on

Add a whitelist with your ip on /etc/fail2ban/jail.conf (see this)

[DEFAULT]
# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not                          
# ban a host which matches an address in this list. Several addresses can be                             
# defined using space separator.
                                                                         
ignoreip = 127.0.0.1 192.168.1.0/24 8.8.8.8

Restart the service

sudo service fail2ban restart
sudo service fail2ban status

Intrusion detection (logging) systems

Tripwire will not block or prevent intrusions but it will log and give you a heads up with risks and things of concern

Install Tripwire.

sudo apt-get install tiger tripwire

Running Tripwire

sudo tiger

This will scan your system for issues of note

sudo tiger
Tiger UN*X security checking system
   Developed by Texas A&M University, 1994
   Updated by the Advanced Research Corporation, 1999-2002
   Further updated by Javier Fernandez-Sanguino, 2001-2015
   Contributions by Francisco Manuel Garcia Claramonte, 2009-2010
   Covered by the GNU General Public License (GPL)

Configuring...

Will try to check using config for 'x86_64' running Linux 4.4.0-89-generic...
--CONFIG-- [con005c] Using configuration files for Linux 4.4.0-89-generic. Using
           configuration files for generic Linux 4.
Tiger security scripts *** 3.2.3, 2008.09.10.09.30 ***
20:42> Beginning security report for simon.
20:42> Starting file systems scans in background...
20:42> Checking password files...
20:42> Checking group files...
20:42> Checking user accounts...
20:42> Checking .rhosts files...
20:42> Checking .netrc files...
20:42> Checking ttytab, securetty, and login configuration files...
20:42> Checking PATH settings...
20:42> Checking anonymous ftp setup...
20:42> Checking mail aliases...
20:42> Checking cron entries...
20:42> Checking 'services' configuration...
20:42> Checking NFS export entries...
20:42> Checking permissions and ownership of system files...
--CONFIG-- [con010c] Filesystem 'fuse.lxcfs' used by 'lxcfs' is not recognised as a valid filesystem
20:42> Checking for indications of break-in...
--CONFIG-- [con010c] Filesystem 'fuse.lxcfs' used by 'lxcfs' is not recognised as a valid filesystem
20:42> Performing rootkit checks...
20:42> Performing system specific checks...
20:46> Performing root directory checks...
20:46> Checking for secure backup devices...
20:46> Checking for the presence of log files...
20:46> Checking for the setting of user's umask...
20:46> Checking for listening processes...
20:46> Checking SSHD's configuration...
20:46> Checking the printers control file...
20:46> Checking ftpusers configuration...
20:46> Checking NTP configuration...
20:46> Waiting for filesystems scans to complete...
20:46> Filesystems scans completed...
20:46> Performing check of embedded pathnames...
20:47> Security report completed for simon.
Security report is in `/var/log/tiger/security.report.simon.170809-20:42'.

My Output.

sudo nano /var/log/tiger/security.report.username.170809-18:42

Security scripts *** 3.2.3, 2008.09.10.09.30 ***
Wed Aug  9 18:42:24 AEST 2017
20:42> Beginning security report for username (x86_64 Linux 4.4.0-89-generic).

# Performing check of passwd files...
# Checking entries from /etc/passwd.
--WARN-- [pass014w] Login (bob) is disabled, but has a valid shell.
--WARN-- [pass014w] Login (root) is disabled, but has a valid shell.
--WARN-- [pass015w] Login ID sync does not have a valid shell (/bin/sync).
--WARN-- [pass012w] Home directory /nonexistent exists multiple times (3) in
         /etc/passwd.
--WARN-- [pass012w] Home directory /run/systemd exists multiple times (2) in
         /etc/passwd.
--WARN-- [pass006w] Integrity of password files questionable (/usr/sbin/pwck
         -r).

# Performing check of group files...

# Performing check of user accounts...
# Checking accounts from /etc/passwd.
--WARN-- [acc021w] Login ID dnsmasq appears to be a dormant account.
--WARN-- [acc022w] Login ID nobody home directory (/nonexistent) is not
         accessible.

# Performing check of /etc/hosts.equiv and .rhosts files...

# Checking accounts from /etc/passwd...

# Performing check of .netrc files...

# Checking accounts from /etc/passwd...

# Performing common access checks for root (in /etc/default/login, /securetty, and /etc/ttytab...
--WARN-- [root001w] Remote root login allowed in /etc/ssh/sshd_config

# Performing check of PATH components...
--WARN-- [path009w] /etc/profile does not export an initial setting for PATH.
# Only checking user 'root'

# Performing check of anonymous FTP...

# Performing checks of mail aliases...
# Checking aliases from /etc/aliases.

# Performing check of `cron' entries...
--WARN-- [cron005w] Use of cron is not restricted

# Performing check of 'services' ...
# Checking services from /etc/services.
--WARN-- [inet003w] The port for service ssmtp is also assigned to service
         urd.
--WARN-- [inet003w] The port for service pipe-server is also assigned to
         service search.

# Performing NFS exports check...

# Performing check of system file permissions...
--ALERT-- [perm023a] /bin/su is setuid to `root'.
--ALERT-- [perm023a] /usr/bin/at is setuid to `daemon'.
--ALERT-- [perm024a] /usr/bin/at is setgid to `daemon'.
--WARN-- [perm001w] The owner of /usr/bin/at should be root (owned by daemon).
--WARN-- [perm002w] The group owner of /usr/bin/at should be root.
--ALERT-- [perm023a] /usr/bin/passwd is setuid to `root'.
--ALERT-- [perm024a] /usr/bin/wall is setgid to `tty'.

# Checking for known intrusion signs...
# Testing for promiscuous interfaces with /bin/ip
# Testing for backdoors in inetd.conf

# Performing check of files in system mail spool...

# Performing check for rookits...
# Running chkrootkit (/usr/sbin/chkrootkit) to perform further checks...
--WARN-- [rootkit004w] Chkrootkit has detected a possible rootkit installation
Possible Linux/Ebury - Operation Windigo installetd

# Performing system specific checks...
# Performing checks for Linux/4...

# Checking boot loader file permissions...
--WARN-- [boot02] The configuration file /boot/grub/menu.lst has group
         permissions. Should be 0600
--FAIL-- [boot02] The configuration file /boot/grub/menu.lst has world
         permissions. Should be 0600
--WARN-- [boot06] The Grub bootloader does not have a password configured.

# Checking for vulnerabilities in inittab configuration...

# Checking for correct umask settings for init scripts...
--WARN-- [misc021w] There are no umask entries in /etc/init.d/rcS

# Checking Logins not used on the system ...

# Checking network configuration
--FAIL-- [lin013f] The system is not protected against Syn flooding attacks
--WARN-- [lin017w] The system is not configured to log suspicious (martian)
         packets

# Verifying system specific password checks...

# Checking OS release...
--WARN-- [osv004w] Unreleased Debian GNU/Linux version `stretch/sid'

# Checking installed packages vs Debian Security Advisories...

# Checking md5sums of installed files

# Checking installed files against packages...
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/udev/hwdb.bin' does not belong to any package.

# Performing check of root directory...

# Checking device permissions...
--WARN-- [dev003w] The directory /dev/block resides in a device directory.
--WARN-- [dev003w] The directory /dev/char resides in a device directory.
--WARN-- [dev003w] The directory /dev/cpu resides in a device directory.
--FAIL-- [dev002f] /dev/fuse has world permissions
--WARN-- [dev003w] The directory /dev/hugepages resides in a device directory.
--FAIL-- [dev002f] /dev/kmsg has world permissions
--WARN-- [dev003w] The directory /dev/lightnvm resides in a device directory.
--WARN-- [dev003w] The directory /dev/mqueue resides in a device directory.
--FAIL-- [dev002f] /dev/rfkill has world permissions
--WARN-- [dev003w] The directory /dev/vfio resides in a device directory.

# Checking for existence of log files...
--FAIL-- [logf005f] Log file /var/log/btmp permission should be 660
--FAIL-- [logf007f] Log file /var/log/messages does not exist

# Checking for correct umask settings for user login shells...
--WARN-- [misc021w] There is no umask definition for the dash shell
--WARN-- [misc021w] There is no umask definition for the bash shell

# Checking symbolic links...

# Performing check of embedded pathnames...
20:47> Security report completed for username.

More on Tripwire here.

Hardening PHP

Hardening PHP config (and backing the PHP config it up), first create an info.php file in your website root folder with this info

<?php
phpinfo()
?>

Now look for what PHP file is loadingPHP Config

Back that your PHP config file

TIP: Delete the file with phpinfo() in it as it is a security risk to leave it there.

TIP: Read the OWASP cheat sheet on using PHP securely here and securing php.ini here.

Some common security changes

file_uploads = On
expose_php = Off
error_reporting = E_ALL
display_errors          = Off
display_startup_errors  = Off
log_errors              = On
error_log = /php_errors.log
ignore_repeated_errors  = Off

Don’t forget to review logs, more config changes here.

Antivirus

Yes, it is a good idea to run antivirus in Ubuntu, here is a good list of antivirus software

I am installing ClamAV as it can be installed on the command line and is open source.

sudo apt-get install clamav

ClamAV help here.

Scan a folder

sudo clamscan --max-filesize=3999M --max-scansize=3999M --exclude-dir=/www/* -i -r /

Setup auto-update antivirus definitions

sudo dpkg-reconfigure clamav-freshclam

I set auto updates 24 times a day (every hour) via daemon updates.

tip: Download manual antivirus update definitions. If you only have a 512MB server your update may fail and you may want to stop fresh claim/php/nginx and mysql before you update to ensure the antivirus definitions update. You can move this to a con job and set this to update at set times over daemon to ensure updates happen.

sudo /etc/init.d/clamav-freshclam stop

sudo service php7.0-fpm stop
sudo /etc/init.d/nginx stop
sudo /etc/init.d/mysql stop

sudo freshclam -v
Current working dir is /var/lib/clamav
Max retries == 5
ClamAV update process started at Tue Aug  8 22:22:02 2017
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 1152
Software version from DNS: 0.99.2
Retrieving http://db.au.clamav.net/main.cvd
Trying to download http://db.au.clamav.net/main.cvd (IP: 193.1.193.64)
Downloading main.cvd [100%]
Loading signatures from main.cvd
Properly loaded 4566249 signatures from new main.cvd
main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Querying main.58.82.1.0.C101C140.ping.clamav.net
Retrieving http://db.au.clamav.net/daily.cvd
Trying to download http://db.au.clamav.net/daily.cvd (IP: 193.1.193.64)
Downloading daily.cvd [100%]
Loading signatures from daily.cvd
Properly loaded 1742284 signatures from new daily.cvd
daily.cvd updated (version: 23644, sigs: 1742284, f-level: 63, builder: neo)
Querying daily.23644.82.1.0.C101C140.ping.clamav.net
Retrieving http://db.au.clamav.net/bytecode.cvd
Trying to download http://db.au.clamav.net/bytecode.cvd (IP: 193.1.193.64)
Downloading bytecode.cvd [100%]
Loading signatures from bytecode.cvd
Properly loaded 66 signatures from new bytecode.cvd
bytecode.cvd updated (version: 308, sigs: 66, f-level: 63, builder: anvilleg)
Querying bytecode.308.82.1.0.C101C140.ping.clamav.net
Database updated (6308599 signatures) from db.au.clamav.net (IP: 193.1.193.64)

sudo service php7.0-fpm restart
sudo /etc/init.d/nginx restart
sudo /etc/init.d/mysql restart 

sudo /etc/init.d/clamav-freshclam start

Manual scan with a bash script

Create a bash script

mkdir /script
sudo nano /scripts/updateandscanav.sh

# Include contents below.
# Save and quit

chmod +X /scripts/updateandscanav.sh

Bash script contents to update antivirus definitions.

sudo /etc/init.d/clamav-freshclam stop

sudo service php7.0-fpm stop
sudo /etc/init.d/nginx stop
sudo /etc/init.d/mysql stop

sudo freshclam -v

sudo service php7.0-fpm restart
sudo /etc/init.d/nginx restart
sudo /etc/init.d/mysql restart

sudo /etc/init.d/clamav-freshclam start

sudo clamscan --max-filesize=3999M --max-scansize=3999M -v -r /

Edit the crontab to run the script every hour

crontab -e
1 * * * * /bin/bash /scripts/updateandscanav.sh > /dev/null 2>&1

Uninstalling Clam AV

You may need to uninstall Clamav if you don’t have a lot of memory or find updates are too big.

sudo apt-get remove --auto-remove clamav
sudo apt-get purge --auto-remove clamav

Setup Unattended Ubuntu Security updates

sudo apt-get install unattended-upgrades
sudo unattended-upgrades -d

At login, you should receive

0 updates are security updates.

Other

  • Read this awesome guide.
  • install Fail2Ban
  • Do check your log files if you suspect suspicious activity.

Check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.92 added hardening a linux server link

Filed Under: Ads, Advice, Analitics, Analytics, Android, API, App, Apple, Atlassian, AWS, Backup, BitBucket, Blog, Business, Cache, Cloud, Community, Computer, CoronaLabs, Cost, CPI, DB, Development, Digital Ocean, DNS, Domain, Email, Feedback, Firewall, Free, Git, GitHub, GUI, Hosting, Investor, IoT, JIRA, LetsEncrypt, Linux, Malware, Marketing, mobile app, Monatization, Monetization, MongoDB, MySQL, Networking, NGINX, NodeJS, NoSQL, OS, Planning, Project, Project Management, Psychology, push notifications, Raspberry Pi, Redis, Route53, Ruby, Scalability, Scalable, Security, SEO, Server, Share, Software, ssl, Status, Strength, Tech Advice, Terminal, Transfer, Trello, Twitter, Ubuntu, Uncategorized, Video Editing, VLOG, VM, Vultr, Weakness, Web Design, Website, Wordpress Tagged With: antivirus, brute force, Firewall

How to boost your site’s SEO

July 21, 2017 by Simon

In these olden days of the Internet, you just needed to submit your site to search engines and wait a few days and everything was fine and dandy. Making your site appear high in search engines these days is next to impossible due to changing rules and formulas defined by search engines.

It is advisable to avoid so-called experts that spam your inbox saying they can improve your site’s SEO for $99 (but strangely do not have a website themselves).

There is no golden rule or trade secret to getting pages ranked high on search engines. You will need to do a number of things to get higher in search results or even appear on search results at all.

Google is not the only search engine and you will need to do a number of things to appear on all of them. Other search engines include DuckDuckGo, Bing, Yahoo, Ask and 14 more search engines here.

Site maps

Google and other top search engines expect your website to declare pages on your site via a sitemap.xml file (specifications here). There is a number of ways you can generate a site map file. If you use the WordPress CMS there are a number of plugins that generate sitemap.xml files for you. Search Engines have automatic bots that will read known sitemap files in order to update their search results.

Q1) Does your web site have a sitemap file?

Warnings

Most search engines will ignore your website if it is not trustworthy. You will be classified trustworthy if:

  • Your website is linked to from many other trustworthy sites.
  • Your website is not in an untrustworthy country or on an untrustworthy web site host.
  • Your website does not have banned words or linked banned media.
  • You don’t serve malicious files (or your Web servers IP address has never served malicious files).
  • Your website is updated frequently. Sites like http://web.archive.org/web/*/www.microsoft.com will let you know how frequently a site has been updated.
  • Your website needs to be secure (must have a secure SSL certificate installed and enabled). Google tells you HTTPS sites get better ranking here.
  • Your website needs to be fast, a search engine will not point people to slow sites period.
  • Your website also needs to have a lot of past and current traffic.

Content

Your site will also need to have good content in order to be seen as a valuable site. A blog is a good way to create new content and get new organic traffic to your site. You will need a desirable product or service to be high in the search engines result (e.g Chart generator or sell a  good product or service ). Your site also needs to contain the matching keywords to be referred to when people search for a particular thing.

It is easy to appear high in the search results if you use feed google enough words that match your article but it is hard enough to get to the top of the rankings for the term “SEO” with “144,00,000” other matching pages in google coming up for “SEO”

If you are going to pay someone to improve your SEO ask for their track record on SEO ranks, ask for examples, adding the more words to search results to get higher is lazy. I think being high in google rankings with 2 words is near impossible unless you use a location in the search term.  Google search is a trying to match the right results based on what you feed it and if you fail in any way your site is ranked down.

Analytics

Using search engines analytics like Google Analytics is a good way to ensure search engines know you exist (Google at least) and let you know what works and allows you to generate more content of what works.

Inbound Links

Most search engines prioritize your site over others if you have a high number of links to your site (inbound links), you can have the best site but with no inbound links to your site, you are in trouble. Creating fake links from non-important sites won’t cheat the system.

Site Speed 

Your web site need needs to be fast in order to be deemed trustworthy from search engines. A slow site will fall down the search engine page listing. Do everything you can do to speed up your site (start with ensuring the server is in the right location on a  non-shared server with a fast website utilizing cached content).

I have updated my post on Speeding up WordPress, my site used to load in 28 seconds, it now loads less than 9 seconds.

Secure sites

Google and other search engines will soon require your site to have a strong SSL certificate and high security (read my guide on SSL here and content security policy and public key pinning here).

You can check your site’s SSL certificate here. Check your site with shodan.io often for security flaws.

Blacklists

You will need to ensure your site is hosted on non-malicious serves and is using modern clean web technology.  Search engines prefer to redirect people to sites that use modern web technologies like HTML5 over transitional HTML websites.

Metadata (Dublin Core)

Metadata is important to allow search engines to know what is on a web page without remembering everything on each page. Read more on the latest metadata standards here.

Logs

Do check your logs (error and activity logs) for website crawlers activity (active and missed).

Search your website logs for known crawler activity. Are you missing search engines?

Your logs will tell you how many hits are coming from search engines.  I have had 1300+ people visit my site from Google this month alone.

Set goals around search engine targets and work toward them (percentage or total traffic).

Campaigns

Although campaigns are not directly related to SEO they do allow you to combine stats around page loads and advertising campaigns. I prefer to track campaigns with a custom central redirect page (e.g /go/index.php) that logs the inbound clicks.

// Save this to a file called index.php (maybe under a folder called "/go/" at the top of your server (e.g "https://www.fearby.com/go/index.php")
header("Cache-Control: no-cache, must-revalidate");
header("Expires: Sat, 26 Jul 1997 05:00:00 GMT");
header('Content-Type: text/html; charset=utf-8');

// Insert campaign logging code here
// todo

/* Sample Usage
https://www.fearby.com/go/?seo
https://www.fearby.com/go/?aws
*/
switch ($_SERVER['QUERY_STRING']) { 

	case "seo":
		header( 'Location: https://fearby.com/article/how-to-boost-your-sites-seo/' ) ;
		break;

	case ("beyondssl"):
		header( 'Location: https://fearby.com/article/beyond-ssl-with-content-security-policy-public-key-pinning-etc/' ) ;
		break;

	case ("mysql"):
		header( 'Location: https://fearby.com/article/setting-up-a-fast-distributed-mysql-environment-with-ssl/' ) ;
		break;
	
	case ("aws"):
		header( 'Location: https://fearby.com/article/creating-an-aws-ec2-ubuntu-14-04-server-with-nginx-node-and-mysql-and-phpmyadmin/' ) ;
		break;

	case ("do"):
		header( 'Location: https://fearby.com/article/how-to-buy-a-new-domain-and-ssl-cert-from-namecheap-a-server-from-digital-ocean-and-configure-it/' ) ;
		break;

	default: 
		echo "I could not find a page for the URL you specified.";
		break;
}

Long Tail Titles.

I use long and descriptive titles to mention what my blog post is about. I find that this helps with matching what and Google search loves it.

Some of my descriptive post titles

  • How to develop software ideas
  • Setting up a fast distributed MySQL environment with SSL
  • Adding secure credit card payments to your website
  • How to setup pooled MySQL connections in Node JS that don’t disconnect

Using long page titles can ensure people click on your links with confidence ( and they will know what your site is about ). Pageviews are not everything, engagement is.

My most recent popular posts.

I think I need to create some more technical guides as they seem popular.

Stats

Viewing statistics and reports can help you determine the popularity of your site.

Analytics

Reviewing Logs, viewing campaign data, viewing misc reports can help you identify incoming traffic to verify working acquisition sources.

I like looking at Google Analytics Behaviour reports for viewing inbound visits.  I can see over 19,000 incoming visitors from google over the last 2.5 years with no sightmap.xml (this proves that content and long titles work).

Adding your site to bing searches

Go to https://www.bing.com/toolbox/submit-site-url and submit your site.

Bing

I tried to submit my XML files to Bing but the form stalled (my bad I did not see the organization size mandatory field).

Bing

I changed the domain above to just the domain (no sitemap).

Bing

I added my 6 sitemaps to the Bing Webmaster tools.

Bing

Submit your sitemap files in the Bing Webmaster tools.

Adding Your Site to the Google Search Console

I am trying the WordPress SEO plugin by Yoest but before I do I am adding my site to the Google Search Console (linked to my Gmail account). You will need to add your site to Google Search Console to have Google index your site.

Before Google trusts your domain you need to verify ownership of the domain  ( for me I uploaded the verification files with the free FileZilla FTP program then loaded the file, then clicked verify at the Google Search Console page ).

Now my domain is verified and the Google Search console allows me to view the search status of my domain.

Installing a WordPress SEO Plugin – Yoast

Install the Yoast Plugin in WordPress (follow the install steps after you activate from step 1-8). You can add Yoast to your WordPress via the Add Plugins screen or download then upload it.

Don’t forget to activate then run the advanced Yoast configuration wizard.

After you get an authorization code it is important that you select your website from the profile list (if you don’t see this STOP and add your site to the Google Search Console (you cannot skip this step)).

Continue through steps 9-12 and finish installing the Yoast plugin.

Don’t forget to enable sitemaps (and review your settings). Now you can open the SEO plugin advanced options (/wp-admin/admin.php?page=wpseo_dashboard) and review the plugin options.

I installed the plugin but no sitemap.xml was created???

Twitter user https://www.twitter.com/atimmer10 said I may need to save the/wp-admin/options-permalink.php settings (no luck, this did not fix it and no sitemap.xml was created ).

I edited an existing published post and changed the short link and re-published a post ( no luck, this did not fix it and no sitemap.xml was created ).

I read this Yoast FAQ about the sitemap.xml not being created. I added the following to my .htaccess file ( no luck, this did not fix it and no sitemap.xml was created ).

# Yoast SEO - XML Sitemap Rewrite Fix
RewriteEngine On
RewriteBase /
RewriteRule ^sitemap_index.xml$ /index.php?sitemap=1 [L]
RewriteRule ^locations.kml$ /index.php?sitemap=wpseo_local_kml [L]
RewriteRule ^geo_sitemap.xml$ /index.php?sitemap=geo [L]
RewriteRule ^([^/]+?)-sitemap([0-9]+)?.xml$ /index.php?sitemap=$1&sitemap_n=$2 [L]
RewriteRule ^([a-z]+)?-?sitemap.xsl$ /index.php?xsl=$1 [L]
# END Yoast SEO - XML Sitemap Rewrite Fix

I still cannot get Yoast to create a sitemap.xml???

It turns out Yoast was creating a hidden /sitemap_index.xml

I then added the 6 individual sitemaps to the Google Search Console and Google started indexing my site.

Now Google will index my site and Yoast will auto-update my sitemaps when I Add/Edit or Delete a piece of content.

Do monitor your sitemap syncs from time to time.

Add your sitemaps(s) to Yahoo, Bing and Ask search engines

More on adding your site to search engines on the Yoast website here. Also, read this article.

robots.txt

robots.txt is also an older sitemap.xml file format, read more here.

I made a backwards-compatible robots.txt file

User-agent: *
Disallow: 
Disallow: /cgi-bin/
Disallow: /wp-admin/
Sitemap: https://fearby.com/post-sitemap.xml
Sitemap: https://fearby.com/page-sitemap.xml
Sitemap: https://fearby.com/news-sitemap.xml
Sitemap: https://fearby.com/category-sitemap.xml
Sitemap: https://fearby.com/post_tag-sitemap.xml
Sitemap: https://fearby.com/news-category-sitemap.xml

Generate a robots.txt here and read more here.

Other Plugins

Generate sitemaps from Yoast sitemaps with WP SEO HTML Sitemaps.

Other Tools

  • Google has a Structured Data Testing Tool to allow you to diagnose various search issues.
  • Webpage accessibility checker.
  • Webpage speed checker

Top WordPress Plugins

Do More of What Works

Review pages and see what works, you may find old pages or posts are more popular than new ones.

What Works

Set Goals and Measure Them

Goals like page views are popular targets for SEO results but focusing on lowering bounce rates and increasing customer session time is just as important.

Through changes in the last 30 days, I have lowered my bounce rate from 94% to 80%. I have made some changes to my site to pick up chicks at the end of posts and hope to get a 50% or lower bounce rate.

Bounce

Where are your visitors coming from?

Google Analytics will reveal where visitors come from. Twitter, Facebook, etc. Knowing where your readers are located allows you to target more users.

Where

Time of Day

Knowing the time of day (from Google Analytics) will show you when it is best to post and promote.

Visitor Frequency

Google Analytics will tell you if your visitors are one-time visitors or regular visitors.

Visitors

Conclusion

Ensure your sitemaps is up to date and keep adding new content.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

V1.7 added info 1 month later

Filed Under: SEO, Website, Wordpress Tagged With: seo

Primary Sidebar

Poll

What would you like to see more posts about?
Results

Support this Blog

Create your own server today (support me by using these links

Create your own server on UpCloud here ($25 free credit).

Create your own server on Vultr here.

Create your own server on Digital Ocean here ($10 free credit).

Remember you can install the Runcloud server management dashboard here if you need DevOps help.

Advertisement:

Tags

2FA (9) Advice (17) Analytics (9) App (9) Apple (10) AWS (9) Backup (21) Business (8) CDN (8) Cloud (49) Cloudflare (8) Code (8) Development (26) Digital Ocean (13) DNS (11) Domain (27) Firewall (12) Git (7) Hosting (18) HTTPS (6) IoT (9) LetsEncrypt (7) Linux (20) Marketing (11) MySQL (24) NGINX (11) NodeJS (11) OS (10) PHP (13) Scalability (12) Scalable (14) Security (44) SEO (7) Server (26) Software (7) SSH (7) ssl (17) Tech Advice (9) Ubuntu (39) Uncategorized (23) UpCloud (12) VM (44) Vultr (24) Website (14) Wordpress (25)

Disclaimer

Terms And Conditions Of Use All content provided on this "www.fearby.com" blog is for informational purposes only. Views are his own and not his employers. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. Never make changes to a live site without backing it up first.

Advertisement:

Footer

Popular

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Add Google AdWords to your WordPress blog

Security

  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • Setting up DNSSEC on a Namecheap domain hosted on UpCloud using CloudFlare
  • Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx
  • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
  • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
  • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
  • Beyond SSL with Content Security Policy, Public Key Pinning etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Run an Ubuntu VM system audit with Lynis
  • Securing Ubuntu in the cloud
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Code

  • How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains
  • Useful Java FX Code I use in a project using IntelliJ IDEA and jdk1.8.0_161.jdk
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider
  • How to setup PHP FPM on demand child workers in PHP 7.x to increase website traffic
  • Installing Android Studio 3 and creating your first Kotlin Android App
  • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
  • How to use Sublime Text editor locally to edit code files on a remote server via SSH
  • Creating your first Java FX app and using the Gluon Scene Builder in the IntelliJ IDEA IDE
  • Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

Tech

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Is OSX Mojave on a 2014 MacBook Pro slower or faster than High Sierra
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • The case of the overheating Mac Book Pro and Occam’s Razor
  • Useful Linux Terminal Commands
  • Useful OSX Terminal Commands
  • Useful Linux Terminal Commands
  • What is the difference between 2D, 3D, 360 Video, AR, AR2D, AR3D, MR, VR and HR?
  • Application scalability on a budget (my journey)
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

Wordpress

  • Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution
  • Setting web push notifications in WordPress with OneSignal
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Wordfence Security Plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
  • Moving WordPress to a new self managed server away from CPanel
  • Moving WordPress to a new self managed server away from CPanel

General

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Using the WinSCP Client on Windows to transfer files to and from a Linux server over SFTP
  • Connecting to a server via SSH with Putty
  • Setting web push notifications in WordPress with OneSignal
  • Infographic: So you have an idea for an app
  • Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl
  • Building faster web apps with google tools and exceed user expectations
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in

Some ads on this site use cookies. You can opt-out if of local analytics tracking by scrolling to the bottom of the front page or any article and clicking "You are not opted out. Click here to opt out.". Accept Reject Read More
GDPR, Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT