Code, InfoSec and Server Stuff

Views are personal and not my employers.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

"If you're not still learning, you're already dying."
- Ryan Holiday - Ego is the Enemy

Follow me on Twitter: @FearbySoftware

View All Posts.

server

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

by

Before UpCloud, when I had 512MB ram on Vultr I had multiple NGINX crashed a day so I used a bash script and scheduled a cron job to clear memory cache when memory fell below 100MB (view the script here).  To further increase the speed of the WordPress I have configured the OS to use memory over the disk.  About once a day free memory falls below 100MB (this is not a problem as my script clears cached items automatically).

Read the full article here: https://fearby.com/article/upgrading-an-ubuntu-server-on-upcloud-to-add-more-cpu-memory-and-disk-space/

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

by

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

Advertisement:



If you have not read my previous posts I have now moved my blog from Vultr to the awesome UpCloud host (signup using this link to get $25 free credit).

Recently I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Spoiler: UpCloud performance is great.

Upcloud Site Speed in GTMetrix

Why Upgrade

I have 1 CPU, 1 GB memory and 50GB storage and it is running well?  I have PHP child workers set up and have set up the preferred use of memory over swap file usage.

View of htop querying processes on Ubuntu

Before UpCloud, when I had 512MB ram on Vultr I had multiple NGINX crashed a day so I used a bash script and scheduled a cron job to clear memory cache when memory fell below 100MB (view the script here).  To further increase the speed of the WordPress I have configured the OS to use memory over the disk.  About once a day free memory falls below 100MB (this is not a problem as my script clears cached items automatically).

Graph of memory falling below 100MB every day

I’d like to add more memory as I am working on some things (watch this space) and I will use the extra memory. I’d prefer the server is set up now for the expected workload.

How to Upgrade

This is how I upgraded from 1xCPU/2GB Memory/50GB Storage/2TB Transferred Data to a 2 CPU/4GB Mmeory/80GB Storage/4TB Transferred Data server.

UpCloud Pricing: https://www.upcloud.com/pricing/

Pricing table form https://www.upcloud.com/pricing/

Upgrade an UpCloud VM

I shut down my existing VM. Read this guide to setup a VM.

Login to the UpCloud dashboard, select your server (confirm the server has shut down) and click General Settings, choose the upgrade and click Update.

Upgrade the Server, shut it down the server and choose upgrade

I confirmed the upgrade options (2x CPU, 4096 MB Memory).

Confirm Upgrade Options

Click Update

Upgrade Applied

After 10 seconds you can start your server from the UpCloud server.

I confirmed the CPU upgrade was visible in the VM

Software Tweaks Post Upgrade.

I added these settings to the top of /etc/nginx/nginx.conf to ensure the extra CPU was used.

I increased PHP FPM ( /etc/php/7.2/fpm/php.ini ) to increase the memory usage and child workers. I doubled child workers and max memeory limit.

I restarted NGINX and PHP

I tweaked WordPress max memory limits

MySQL Tweaks: I logged into MySQL

I ran “SHOW GLOBAL STATUS” to view stats

Read more on SHOW GLOBAL STATUS here. Read more on the values here.

I can see NO major errors here (possibly due to UpClouds awesome disk IO) so I won’t be making memory tweaks in MySQL. Sign Up using this link and get $25 credit free on UpCloud and see for yourself how fast they are.

Configure Ubuntu System Memory Usage

Edit /etc/sysctl.conf

Add the following to allow things to sit in ram longer

Snip from: https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Read these pages here and here regarding setting MySQL memory.

Reboot

Resize the disk

The 2x CPU, 4GB memory plan comes with 80GB storage allowance.  My disk at present is 50GB and I will update the size soon following this guide soon.

Upgrade disk from 50gb to 80GB soon

Quick Benchmark

I used loader.io to load 500 users to access my site in 1 minute.

HTOP showing 2x busy CPU's running at 60%

The benchmark worked with no errors.

Loader.io Success with 500 concurrent users

This benchmark was performed with no Cloudflare caching. I should get Cloudflare caching working again to lower the average response time. I loaded my website manually in Google Chrome while loader.io threw 500 users at my site and it loaded very fast.

Conclusion

After a few days, I checked my memory logs and there were no low memory triggers (just normal internal memory management triggers). Ubuntu was happier.

No Low memory low triggers

This graph was taken before I set “vm.vfs_cache_pressure” so I will update this graph in a few days.

I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

1.0 Initial Post

Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare

by

This guide will show you how to enable the latest Transport Layer Security (TLS) 1.3 protocol with it’s predecessor Secure Sockets Layer (SSL) with NGINX and OpenSSL for better website security on an Ubuntu 16.04 server

Advertisement:



I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Making sure your server is up to date and running the latest SSL software is important. I have updated Open SSL before and blogged about this here.  Do backup your server before changing settings and if you use  Cloudflare (if you don’t do it now) enable Development Mode (and disable caching until changes are made).

For the best performing VM host (UpCloud) read my guide on the awesome UpCloud VM hosts (get $25 free credit by signing up here).

TLS 1.3 is the latest SSL security protocol that can be used between clients and servers to encrypt connections on the web.

TLS 1.3 uptake is only 60% according to https://caniuse.com/#search=TLS%201.3

TLS 1.3

Read why TLS 1.3 is important and news on TLS 1.3 can be found here: https://www.openssl.org/blog/blog/2018/02/08/tlsv1.3/

The Good and Bad

Done be like this commercial site with very poor security (tested with SSL labs and asafaweb)

Bad SSL

Here is what the top 1 million sites do

 

Installing Open SSL on Ubuntu

Connect to your Ubuntu 16.04 server via SSH (I connected to my Vultr server)

Check what version of OpenSSL you have? My OpenSSL is out of date.

Tip: What Ciphers does your Open SSL Support?

Time to update Open SSL

OpenSSL 1.1.1 beta is available and supports TLS 1.3  but it is n BETA form.  OpenSSL code is available here.

I did the following to download and build the latest version of OpenSSL.

I tried to check the open SSL version but had an error?

A quick GitHub ticket revealed I needed to set a path variable.

Open SSL now reports it’s version.

What version NGINX do you have (1.13 supports TLS 1.3) read here

Backup your NGINX

Do backup your server files and take a snapshot if need be.  I am not responsible;e for a broken server,

Edit NGINX Configuration

Update NGINX configuration: /etc/nginx/sites-available/default

tip: Review other NGINX hardening settings here.  Also remove TLSv1.0

I tested my NGINX config loaded them and restarted NGINX

Check the status of NGINX

If you have configured Cloudflare then log in and enable TLS support.

Cloudflare TLS Settings

Enable TLS 1.3 in Chrome by visiting chrome://flags/#tls13-variant This should be automatic in later versions of Chrome and other browsers.

Enable TLS in Chrome

Verify TLS

I used the developer tools in Chrome to confirm the page was verified in TLS 1.3.

Verify TLS

 

Updated to 1.1.1-pre6-dev

Don’t forget to test your SSL strength with https://dev.ssllabs.com/ssltest/

SSL Test 2018

I hope this guide helps someone.

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.3 added bad ssl cert.

v1.2 ssl test v1.1 updated to 1.1.1-pre6-dev

v1.0 Initial post

How to purchase your own domain name and set up a web server from $5 a month

by

This guide will show technically minded people how you can purchase your own domain name, set up a web server on Vultr with an online store using WordPress/WooCommerce from $5 a month. Warning this post is technical (if you have never used SSH, Ubuntu, Linux Command Line, hate risk or are not patient then this is NOT the guide you are after).

Advertisement:


Draft Post (released early to share)

I personally recommend (not a paid endorsement) the free WooCommerce plugin for the free WordPress.org CMS on the free Ubuntu Operating system with the free NGINX web server and the free MYSQL database engine and free SSL certificates from Lets Encrypt.

Update 2018: For the best performing VM host (UpCloud) read my guide on the awesome UpCloud VM hosts (get $25 free credit by signing up here).

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Sorry for using the word free a lot but I like free things.  One of the benefits of a using a self-managed server is you get the option to install free software and configure the server how you want and secure it how you want. Truth be told managed ho (e.g CPanel, etc) are in the business of making money via monthly feed, expensive SSL certificates, taxing your transactions or pushing you to higher priced tiers.

Legend:

  • Self Managed Server = A server that you create, you configure patch and support (all the reward and risk is owned by you and costs are low).
  • Hosted Server = A server you have partial control of and the hosts manage the server and support (You hand away all risk and most of the control and pay for support/features).

I moved to a self-managed server after I was paying $25/m for a poorly performing website and $150/y for a poor quality  SSL certificate and a slice of a server that seemed to always say “Usage Limit Exceeded”. Why pay for an insecure website that my visitors could not view because the usage limit was exceeded.

Bad CPanel SSL Certificate

fyi: Fearby.com costs me $10 a month for a server and $5/m for CDN abilities.

CPanel hosts are an option when you don’t want to self-manage a service and take on the hassle but be prepared for server limitations (The image below was taken on an older CPanel based hosts before I moved to a self-managed Vultr server)

cpenal_usage_exceeded

I recently discovered a well known and established website hosting service (that I used to use) and a friend is still using is insecure. My friend’s site has a static website on it but the server underneath was very old and insecure. Having a secure web server should be at the top of your list with any self-managed or hosted website (this will help search engine optimization and prevent risks to your website visitors).

Static Website

Sites like Virus Total, SSL Labs and Alexa Site Info , Qualys are good ways to review a site’s credibility.

fyi: The awesome https://seositecheckup.com/ is awesome for evaluating our sites SEO score.

Before we set up a server with WordPress on your own server let’s quickly look at the alternative commercial ready to go website builders.

 

Alternative (paid) DIY Website Builders

The following leading commercial sites will allow you to build a site online.

In my opinion, five things matter with setting up site online website.

  • Setup Costs, Monthly Cost and Commissions (what are the hidden charges)
  • Security (having a food SSL Certificate is key to having a good organic traffic from search engines)
  • Site Speed (Having a slow site will impact search engine optimization and drive visitors away)
  • Accessibility (if your site is not WCAG accessible it will not rank high on search engines).
  • Control (will you be able to do everything you want too, nothing worse than going so far and being limited)

Ok, let’s see how much it will cost to set up a simple business site on the sites above.

 

Wix 

Setup: Goto https://www.wix.com/, Login to Wix, click Create Site, click Business, Click Choose a Template, Edit the page, Click Save, Click “Connect your own customized domain“, Click “Connect a domain you already own“.

I was redirected to a Wix plan pricing page where I need to choose a plan to continue. From what I researched you cant control HTML on Wix so can’t add a MailChimp newsletter signup form so you would have to go with the $24.5/m option to enable Email Campaigns.

Wix Plans Chooser

I could not see information about included SSL certificates, SEO or other chargers.  SSL is free after you pay right?

The Wix editor appears OK (it may take a bit of learning though).

Wix Editoer

I clicked publish and the site was live

Wix site published

A quick check of the SSL, Accessibility and SEO and no obvious deal breakers here apart from the price and platform lock-in.

Wix Checkup

I performed a security check on the site with https://freescan.qualys.com (passed)

Conclusion: I hear Wix templates are hard to change so choose your template wisely, A large collection of apps are available that you can add to the site.

Although Wix was nice and it does include a full-featured look at the engine it is not for me ($24/m USD is too expensive).

 

Squarespace 

Squarespace basic websites cost $16/$25 a month or $34/52 for online stores: https://www.squarespace.com/pricing/

SquareSpace Pricing

Setup a Squarespace website: Goto https://www.squarespace.com/, Click Start a Free Trial, Choose a Template, Create an Account (a quick read of the terms of service and privacy policy, #scary), SpareSpace sites are pre-published?

Square Space Build

Loading the webpage on a non-logged-in (with SquareSpace login) browser displays a trial warning.  Trial pages are essentially restricted (unlike Wix).

Login Challenge

The mobile view does not match the template?  I guess the chosen template is more of a vibe and not a template.

Mobile view

Setting up a Squarespace website may take some time. Squarespace does have some nifty advance options in a slide-out menu though.

Squarespace Settings

Because the public view of the page is restricted I cannot scan it with WCAG accessibility tools. Scanning the site performance speeds with gtmetrix also fails.

Performance Rejected

Squarespace is well known to be difficult to set up a website when compared to other drag and drop editors (but Squarespace sites do look nice).

I am not paying $54/m for a website so let’s move on.

 

Shopify

Shopify Setup: Goto https://shopify.com and click Create, Sign up and enter your store name. Complete the wizard. 

Shopify

Choose a Shopify Plan

Shopify Plan

Scalping transactions, no thanks. let’s move on.

 

Weebly

Weebly Setup: Goto https://www.weebly.com/au and click Get Started under Create Store. Enter your account details and click Create Your Site, enter the name of the store, Click I’m just trying Weebly, click the type of product you will be selling.

Weebly Site Setup

Weebly Setup

Theme Selection

Theme Select

Choose a Domain

Domain Select

Publish the site

Publish

Clicking publish appears to be a dead end.

Verify Weebly

“Please contact Weebly Support to verify your account”, No Thanks, let’s move on.

One candidate remains and that is WordPress hosted (wordpress.com not wordpress.org).

 

WordPress.com

WordPress.com offer hosted plans for WordPress in the cloud.

Setup a WordPress site, the only one that removes WordPress branding and allows third-party plugins to be installed it the Business plans for $33 a month.

WordPress Plans

Setup Basics

Wordpress

Choose a WordPress theme.

Choose Theme

Assign a Domain

WordPress Domain

In order to buy a domain, you need to log in (top right) with an account

My working WordPress account (is no longer working), it was in my password manager.

wordpresscomerror

I seem to be stuck in a signup loop

Wordpress

Time to move on. Time to set up my own server on Vultr and setup WordPress and  WooCommerce,

But, before we do, let’s ensure our name is secure online.

Search for your Name/Brand

Do search for your website (or thing) in search engines to see if your name is already taken, don’t buy a domain that is owned or has IP or trademark presence. It is a  good idea to use sites like https://namechk.com/ to see if your site or social media is already taken.

https://namechk.com/

namechk.com will allow you to search for name availability online.  The name “mything” is not fully available online.

https://namechk.com/ 2

You will want to see all green squares (name available) below before buying a domain name. This looks better.

Namechk ok

I would recommend you create your social media accounts before or right after buying your domain. Sites like Twitter will insist on short usernames names so get your social media sites first.

 

Trademark and Brand Search

Also, perform a trademark and IP search.

Australian Trademark Search: https://search.ipaustralia.gov.au/trademarks/search/quick

United States Trademark Database: https://www.uspto.gov/trademarks-application-process/search-trademark-database

Global brand Search: http://www.wipo.int/branddb/en/

etc

Self Managed Warning

I tend to go the “self-managed server route” and install the free WordPress CMS because:

There are many reasons why you would not want to “self-manage” your own server

  • Technical Requirements (and time to support).
  • Higher Risk.
  • Applying Updates and Patches.
  • etc.

Being technically minded and choosing a “self-managed web servers” can take away time from the fun stuff like SEO, Site Design, customer needs, branding etc.

Self Managed Costs

For $5 a month you can buy a server with enough memory to install WordPress (cheaper if you don’t need WordPress)

Vultr is great. Vultr does have ready to go servers that you can deploy that have WordPress all set up.

wordpress-template

The Vultr template above does use the Centos OS (read my guide setting up Centos on a different service provider here) but I prefer to manually setup a server with Ubuntu 16.04 OS on Vultr.

Wih a $5 server you can do what you want with it.  I have blogged before about setting up your own Server. e.g Installing Centos and Ubuntu server on Digital Ocean.  Digital Ocean does not have data centres in Australia and this kills scalability. AWS is good but 4x the price of Vultr. I have blogged about setting up and AWS server here (and upgrading an AWS instance). I tried to check out Alibaba Cloud but the verification process was broken so I decided to check our Vultr.

Manual Setup of Vultr on an Ubuntu 16.04 server

  • Deploy a Vultr Server – Guide here  ($2.5/m to New Jersey or Florida or $5/m to Sydney,  I would recommend you opt in for the auto backup for $0.50c/m and $1/m respectively).
  • Setup NGINX.
  • Setup PHP and PHP-FPM (see guide above), consider adding PHP child workers.
  • Setup and secure MySQL (see guide above), create a database for WordPress to use.
  • Instal Adminder MySQL GUI (guide here).
  • Setup a free Lets Encrypt SSL certificate (guide here).
  • Install WordPress (and Jetpack plugin).
  • Install WordPress CLI.
  • Instal the WooCommerce Storefront WordPress Theme.
  • Install WooCommerce Plugin.
  • Secure Ubuntu.
  • Also consider linking your domain to Cloudflare to boost performance, scanning your site with Qualys Freescan and OWASP ZAP).
  • Consider setting up a WordPress image compressor and CDN plugin. like EWWW.io

Manual WooCommerce Plugin Setup

Once you setup Woocommerce you can setup the store defaults. Goto the WordPress dashboard and click WooCommerce Settings

Woo Commerce Settings

Settings – General

General

  • Set Address, City and State and Postcode
  • Set allowed countries to sell in (e.g Australia)
  • Set allowed countries to ship items to (e.g Australia)
  • Set Enable Taxes
  • Set Currency
  • etc

Settings – Products

Products

  • Set Weight
  • Set Dimensions
  • Enable Product Reviews
  • Enable Star Ratings on Reviews
  • etc

Settings – Shipping

Shipping

  • Enable Shipping Calculator
  • Add Shipping Classes
  • Shipping Zones
  • etc.

Settings – Checkout

Settings checkout

  • Force Secure Checkup
  • Create a Terms and Conditions page (and set).
  • etc

Settings – Account

Accounts

  • Set Account Options
  • etc

Settings – Emails

Emails

  • Set Email Preferences
  • Set Email Header Image
  • Set Email Colour
  • Set Footer Text
  • etc

Settings – API

API

  • API can be disabled if you don’t need it.

Optional Actions

  • Setup Yoast Plugin
  • Setup other plugins

 

Instaling a Woo Commerce Child Theme

Goto https://woocommerce.com/product-category/themes/storefront-child-theme-themes/ and choose a theme.

Themes

Purchase and Install the desired child theme (I uploaded it to my /wp-content/themes/ folder with forklift). I chose a free deli theme.

Goto your WordPress then themes folder and activate your new child theme.

Activate Theme

 

 

Post Site Setup

Just because your site is live does not mean you can rest.

SEO Optimization

Do use sites like https://seositecheckup.com/ and follow recommended actions to improve your SEO like updating meta tags.

More Reading

Attaching an email to your domain

You can pay $5 a month and link a G Suite email to your domain.

  • Dedicated professional Google G Suite email account for $5 a month with 30GB storage (If you don’t want ot to buy a G Suite email and link it to your domain then you don’t need this).

Once you have a G Suite account you can link other domains (and domain emails) to it. You can login to your G Suite emails via G Mail and send emails from apps or the command line.

Why Vultr

I use the server host Vultr as they have data centres all around the world and the support of great, Digital Ocean is good too but they don’t have data centres in my country (Australia). Vultr allows you to deploy all over the world upgrade servers, move servers, add storage and restore servers.

Alternatively, you can buy a $2.5/m server and generate  a static website

I use the Platforma Web HTML generator to build mobile and WCAG compliant websites.

Buying a domain,  I buy my domains from https://www.namecheap.com/  it is a good idea to look for coupons first at https://www.namecheap.com/promos/coupons.aspx before buying a domain.

Once you buy a domain you can point it to a Vultr server and upload your website.

I hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v.1.2 WordPress WooCommerce

v1.1 SEO

v1.0 Initial Draft

Backing up OSX or an Ubuntu server with Backblaze B2 Cloud Storage from the Command Line

by

This computer will show you can back up computer or server with Backblaze B2 Cloud Storage from the Command Line n OSX and Ubuntu.

Advertisement:



This post is still being written. I have a number of guides on moving hasting away form CPanel, Setting up VM’s on AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line. Also, I have blogged about how you can add block storage to a Vultr server, backup and restore snapshots , syncing files with rsync along with using GitHub and Bitbucket but what do you do if you need to backup large amounts of data?

Backblaze has a Cloud storage solution that costs as low as $0.005c a GB (a month), The first 10G is free. Backblaze say “From bytes to petabytes Backblaze B2 is the lowest cost high-performance cloud storage in the world. ”

Back Blaze have open sourced internal drive enclosure designs and drive failure stats and it’s time I gave them a try.

Goto https://www.backblaze.com

Backblaze

Create or sign in.

Backblaze Login

After you login got the dashboard.

B2 Cloud

Click Backblaze B2 Cloud Storage

Activate B2 Cloud

Signup

Create a Bucket

Create Bucket

Name the bucket (long names with a GUID are good).

Name the Bucket

 

You can rename the bucket here and change public/private and or upload/download files manually.

Manage Bucket

The first thing I did was limit the versions of files under the lifecycle settings for the bucket.

Version Settings

Now I created a series of subfolders to store files from different servers (I could have used many buckets but one bucket will do).

Folders

I can upload files via the Backblaze bucket GUI if I needed to.

Upload and Download

Back Blaze has a command line tool for uploading: https://www.backblaze.com/b2/docs/quick_command_line.html

Install Steps

Backblaze state “The B2 command-line tool is available from the Python Package Index (PyPI) using the standard pip installation tool. Your first step is to make sure that you have either Python 2 (2.6 or later) or Python 3 (3.2 or later) installed.”

I have Python 2.7 installed

Install PIP

I ran into issues updating b2 CLI

I tried installing via the alternative method (with no luck)

I tried running the setup script (with no luck)

Upgrading setup tools also failed

Backing up a Mac via command line with B2

More to come when I can get B2 CLI Installed.

Backing up a an Ubuntu machine via command line with B2

More to come when I can get B2 CLI Installed.

Update

My ticket with Backblaze was automatically closed with this note “If the issue is persisting, it may be easiest to map the installation to the user folder, rather than the system level.”

No ideas how but something to research.

 

Ask a question or the recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

V1.1 ticket closed

v1.0 Initial post

Self Managed Server – Frequent Manual Tasks

by

Below are the manual Task’s I perform frequently on all self-managed Ubuntu serves I own (along with automated backups and update tasks).

Advertisement:



Here is a post on automatically backing up a server to another via crontab entries. I have a number of Ubuntu Servers on Vultr and Digital Ocean (online cloud server for as low as $2.5 a month).

I also perform manual backups to ensure files are backed up.

Manual Backup Files.

First I create the following folder structure on my OSX desktop (for each server I need to backup).

My Server 01

  • ~\Desktop\www.myserver01.com\
  • ~\Desktop\www.myserver01.com\www
  • ~\Desktop\www.myserver01.com\db
  • ~\Desktop\www.myserver01.com\nginx
  • ~\Desktop\www.myserver01.com\misc

My Server 02

  • ~\Desktop\www.myserver02.com\
  • ~\Desktop\www.myserver02.com\www
  • ~\Desktop\www.myserver02.com\db
  • ~\Desktop\www.myserver02.com\nginx
  • ~\Desktop\www.myserver02.com\misc

etc (for each server)

I then use Forklift 3 (not a paid endorsement, I just love it) to manually backup files on the server. I manually copy files that are available via existing SFTP connections in Forklift (e.g WWW, MySQL, NGINX, MongoDB etc).

manual_backup_001

I simply drag and drop important file system files in Forklift (from the remote SFTP instance to a local folder).

SFTP copy progress can be viewed in Forklift.

manual_backup_002

fyi: SFTP is not the fastest transfer protocol. It appears only 39MB (4055 items) has been downloaded in 8 hours over ADSL (on a server that is over 400ms away).

Slow SFTP

FYI: Slow servers (ping) do not like like SFTP.

TIP: Consider zipping smaller files first (into one larger file) or using RSync instead.

Pre Zipping up files before backing up.

I used this command to pre-compress entire folders before downloading them over SFTP.

I created a bash script to manually prepare files to backup on each server.

Don’t forget to make the script file executable

Now I call the script to compress files before backup

TIP: Add other things to back up to the script (e.g MongoDB and MySQL).

Below is my script to backup WWW, NGINX, MySQL and MongoDB and zip them before copying over SFTP.

Thanks to this thread for MySQL export to separate file help.

Note: The warning “Using a password on the command line interface can be insecure.” will be shown when exporting a database from the CLI.

Now I can download the 4 outputted files manually from each server.

Snapshot Tasks

Taking snapshots in Vultr is a great way to backup too.

Snapshot

Manually Updating the Server

I also connect to the server via SSH and check for package updates

Now I can now manually update packages

Quick Reboot to ensure packages are updated.

More to come (on viewing logs (errors, firewall, stats), backing up php etc).

Here is a post on automatically backing up a server to another via crontab entries. I have a number of Ubuntu Servers on Vultr and Digital Ocean (online cloud server for as low as $2.5 a month).

Hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.0 Initial Version

How to upload and download files to a server with sftp (over the SSH protocol)

by

I have blogged about http://c9.io before and how it makes managing a remote Ubuntu server easier. Recently AWS acquired C9 and integrated it into AWS. This has triggered me to find a more open/free way to connect to my servers. I like AWS but I can tell that C9 will someday block you from talking to non-AWS servers.

Advertisement:



C9 Aquisition

C9 on AWS: https://aws.amazon.com/cloud9/?origin=c9io

C9 Aquisition: https://c9.io/announcement

I have blogged about connecting to an AWS EC2 instance with C9 (before AWS acquired C9), The quickest way to set up a scalable development IDE and web server when I was using Digital Ocean Servers, How to buy a new domain (dedicated server from digital ocean) and add an SSL certificate from NameCheap but when I moved to Vultr serves I decided to ditch C9 and save $9 a month.  I got used to setting up an SSH connection, using rmate to edit remote files locally with Sublime and SSH’ing into the box with vSSH on OSX but nothing replaced the C9 file management experience.

C9 was a good IDE (but recent price hikes and AWS purchase have made me cautious).

C9

Uploading files without C9 to Ubuntu

Uploading files from an OSX laptop to a remote Ubuntu server is possible with the SCP command

Server Actions (to upload the old way, with scp)

There has to be a better way

I have used the Forklift 3 program before (my review here) and in recent google searches, it was suggested that Forklift best SFTP program on OSX. Yay. This unix.stackexchange.com thread mentions the difference between SFTP and SSH.

Below are the steps to use SFTP in Foftlift 3 on OSX to connect to a Vultr server (you could use Digital Ocean). Ensure you have a working SSH connection setup from your server to your local OSX.

Setting up an SFTP Connection in Forklift 3

Read my Forklift 3 guide here first.

Update your Forklift to the latest version.

Add a Favorite in Forklift 3

Add Favorite

Name the connection

Name the connection

In Forklift edit the connection

Edit

Specify SFTP and your servers working SSH username and password, and “/” folders

Settings

Save.

Double Click the entry to connect to your server.

Connected

To upload or download files with Forklift over SFTP, simply drag and drop.

Upload

Simple

Upload

You can now use Forklift to navigate folders, delete files, set remote file permissions all from a GUI. You can still use SSH CLI and or Sublime.

Quik Edit works too.

Quick Edit

Nice, this gives me a convenient GUI way to upload, download and edit files instead of via CSP/SSH/CLI.

Now is there an SFTP plugin for Sublime?

More to come soon.

 

Donate and make this blog better

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.0 Initial Draft

 

Connect to a remote server with ssh keys generated on OSX

by

Below is the way I connect to a remote server via SSH keys generated on OSX.

Advertisement:


Setting up a server

When you set up an Ubuntu server on Vultr (read my guide on setting up a Vultr server for as low as $2.5 a month) or Digital Ocean (use this link to get two months free when you setup an Ubuntu server on digital ocean) you can specify an SSH key to use for remote connections during the server create stage (old guide here).

How to create an SSH key on OSX to use to connect to a remote server

Run the following command (“sudo ssh-keygen -t rsa“) to generate an ssh key paid in “~/.ssh/” on OSX.

TIP: It is a good idea to also generate a passphrase to use with the key (double protection). You will be prompted to enter this password to use the RSA key.

You can now see the generated keys in ~/.ssh/

You can view the contents of the public file (you can use this when generating Digital Ocean, Vultr, AWS or Azure or other cloud servers).

fyi: Replace 123.123.123.123 with your remote serves ip.

How to connect to a server (the old way).

As long as your host added the desired public ssh key file contents to the server (adding the public ssh key contents to “~/.ssh/authorized_keys” you will be able to connect to the server.

Run the following command on OSX command line to connect to the server via SSH.

You should see..

Congratulations, you should now be able to connect to your server via SSH.

Securing your ubuntu Server

Read my guides here, here and here.

Don’t forget to add a firewall and set up an SSL certificate.

How to connect to a server (faster way).

todo: ~/.ssh/config method

Now we can connect to your remote server with the shorter method.

todo: ~/.ssh/config method

Donate and make this blog better

Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Revision History

v1.0 Initial Post

etc

Security checklist for securing a self-managed Ubuntu server in the cloud

by

Below is a (perpetually updated) security checklist for securing a self-managed Ubuntu server. Recently WordPress released patch v4.8.3  that fixed some SQL injection issues.  Is your OS, Database, Web Server, OS and software up to date?

Advertisement:



Although I have recently blogged about securing Ubuntu in the cloud, and running a server Audit with Lynus,  this new post is really about obtaining a mindset change and allocating time (each week) to ensure your self-managed servers and software is kept up to date. You can easily list down the actions you need to follow but keeping a system up to date is hard work. Sites like www.shodan.io will reveal what servers or services are vulnerable, let software updates lapse long enough and an open exploit may open a hole to your system.  It only takes minutes to set up a $2.5  a month Ubuntu server with Vultr, $5 a month Digital Ocean Server or AWS server but you need to maintain it.

I highly recommend that you watch the following video that highlights the need for even minor vulnerabilities to be patched asap. If you leave one minor vulnerability open you will give hackers a foothold into your system.

Follow @jawache on twitter.

Troy Hunt has a great post about the simplicity of hacking. Hacking is child’s play.

General Security Checklist

Find log files on your system:

Output (handy logs to review):

  • Enable brute force detection and banning (fail2ban etc) Read more here.
  • Secure folders with service accounts.
  • Do secure software (e.g WordPress Wordfence)
  • Do use SSL Certificates (and use modern cyphers and test with https://www.ssllabs.com/ssltest/)
  • Monitor SSL vulnerabilities.
  • Do a Lynis security report.
  • Install a Virus scanner (read here).
  • Secure MySQL/Databases.

First, find the version of MySQL

Read the official MySQL manual here and security guidelines here.

Read this Digital Ocean guide on securing MySQL.

  • Other: _______

Application (coding) checklist

Retain and protect information.

Infrastructure

Plan for the worst, hope for the best.

  • Use the latest Long Term Support (LTS) version or Ubuntu
  • Update packages

View app packages (Ubuntu 16.04) with updates

View app packages (Ubuntu 16.04) with updates

To update packages type (remember to backup data and config files first)

Among other things, you will see the following information

Show available updates

  • Only work on code checked into GitHub or BitBucket (You will thank me when data or servers disappear).
  • Backup configuration files or backup to remote servers (my rsync guide here)
  • Use snapshots of VM’s.
  • Use Green/Blue server deployments (toggle one server a Prod and the other and Dev/Test and have one ready for a hot spare). Digital Ocean has a good guide here.
  • Consider forcing Content Security Polic and Public Key Pinning or at least using LetsEncrypt SSL certificates.
  • Take Snapshots of VM’s (automate)
  • Backup MySQL databases:

Other Useful Linus Terminal Commands.

Mindset/Culture

Dedicate time to securing your site.

  1. Spend one day a week (or automate) the updating of the OS/Software (no excuses).
  2. Follow people on twitter and subscribe to newsletters of those that are security conscious

Don’t forget to read securing Ubuntu in the cloud blog post here.

And check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

More to come..

Donate and make this blog better


Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

v1.2 added link to Hardening Linux Server link

v1.1 added @jawache link

Short (Article):

Debug an offline Vultr Ubuntu server

by

Having a self-managed Ubuntu server from Vultr for as low as $2.5 a month (or Digital Ocean for $5 a month (setup guides here or here respectively)) can certainly be cheap but you take on all the support and risk in keeping the services up.

Advertisement:



Vultr has a good monitoring information but this does not show you services performance.

Monitor

Node JS monitoring service like http://docs.keymetrics.io/ is great at monitoring Node applications where webmin and PHPServerMonitor are good options for monitoring basic servers and services.

The more software you install the more complex the setup and find potential errors.

Service Interruption Notifications

How will you be notified when things are down?  Having automated monitoring scripts (or Self Service Status Pages) or external monitoring services is a good idea, the last thing you wants is to see your server or service is down based on a twitter reply.

Down

Much thanks goes to Michael Boelen on Twitter for reporting that my server was down:  Follow: @mboelen

Founder of @cisofy_is, author of rkhunter and Lynis, blogger at linux-audit.com, public speaker. #linux #security

Read more Linux troubleshooting tips at https://linux-audit.com/

You can have many different types of errors and it is going to be hard to suggest where your problem is going to be.

Provider/Networking Errors

Always assume provider errors are an issue, more often than not my servers have gone offline due to provider problems outside of my control.

I have had networking errors on Vultr with the default Dynamic DHCP IP’s (prompting changes to Static IP’s). I have also had issues with Static IP’s on Vultr and had to log a Support Tickets again when my server went offline, In this case, Vultr were able to restore my server but I am unsure of what happened?

Example Problem (DHCP IP): Dynamic IP, VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).

I logged a ticket with Vultr after I rebooted the server.

DHCP

Vultr support quickly identified the problem was my server was not picking up an IP address and suggested I set static IP and reboot.

DHCP Error

Example Problem (Static IP): VM is totally offline (not accessible via the web or Telnet, but is accessible via Vultr Web Console).

Again I logged a support ticket and Vultr indicated the issues was network related.

Networking

In both cases, there was nothing I could do (a reboot did not fix each error) and a support ticket had to be logged.

Broken WordPress

Example Problem (Website Down): Error 502 bad gateway

Today I had a server stay-up buy NGINX reported error 502 bad gateway. IN this case I rebooted teh server

Know your NGINX log file location

Show the last 22 lines of the identifies NGINX log file

Error  Hints

PHP-FPM is reporting errors.

Wordpress too is reporting errors, I could have restarted NGINX and PHP but it was just as easy to reboot the server as I forgot the right commands to restart PHP/NGINX.

I rebooted the server in 30 seconds.

Reboot

How I Rebooted in Ubuntu

How to prevent this error in future?

  • Extend PHP file execution time (No)
  • Daily reboots of the server (No)?
  • Daily restarts of NGINX/PHP (Yes, Short Term)
  • Check the server for errors and automatically reboot (Yes, Long Term).

How to check system uptime (verifying the server rebooted)

I will set a cronjob entry to perform diagnostics (via a Bash Script) and restart NGINX/PHP or the server. I might set up remote monitoring of the web server content too.

I should have checked memory/CPU usage too but forgot (sometimes it is best to do a deep investigation and get more information).

Troubleshooting Network Errors

As mentioned above the providers can have network issues (with a static or Dynamic IP) so don’t feel bad if you cannot fix every networking error.

Perform External Ping

Can you ping a remote server from your server?

Has your domain expired?

Check the expiry of your domain here