IoT, Code, Security and Server Stuff

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

on

Adding HTTPS to Apache on OSX High Sierra

by

This guide will help you create and install a self-signed SSL certificate in Apache on OSX (HIgh Sierra) to aid local SSL/HTTPS development.

Advertisement:



UPDATE: Nov 21st 2018 – This works on OSX Mojave too.

I usually force HTTPS traffic on everything I develop (see code below in PHP).  This PHP code will direct all HTTP requests to HTTPS.

Also, you can deny non-https traffic in NGINX with online servers by editing your/etc/nginx/sites-available/default file

Add this to your Nginx sites available file (above) to force SSL at the web server.

You can also deny port 80 connections in your firewall and NGINX if you don’t trust the directive(s) above.

Apache Configuration (sor https)

Edit httpd.conf

Uncomment lines with these text strings in httpd.conf

"socache_shmcb_module" (or "LoadModule socache_shmcb_module libexec/apache2/mod_socache_shmcb.so")
"ssl_module" (or "LoadModule ssl_module libexec/apache2/mod_ssl.so")

I had to find my and “httpd.conf” file

Find your “httpd-ssl.conf” file

Copy the config file (just in case a future update overwrites it).

Now go back and edit the “httpd.conf” file

Add the following line (to the end of the file, on a new line) so the new SSL config file loads.

You can have a look at the file “httpd-ssl-localhost.conf”. Take note of paths in the VirtualHost node.

Generating a Key and Certificate

Generate a Key (I prefer filenames start with “localhost” so future updates to not overwrite it).

Check that file files ” localhost.crt” and ” localhost.key” were outputted to “/private/etc/apache2/”

pwd
/private/etc/apache2
192-168-1-200:apache2 simon$ ls -al
total 208
drwxr-xr-x   13 root  wheel    416 18 Dec 00:33 .
drwxr-xr-x  128 root  wheel   4096 16 Dec 19:41 ..
...
-rw-r--r--    1 root  wheel   1318 18 Dec 00:33 localhost.crt
-rw-r--r--    1 root  wheel   1708 18 Dec 00:33 localhost.key
...

Open “/private/etc/apache2/original/extra/httpd-ssl-localhost.conf” and replace the following text.

Replace the following text:

  • “server.crt” with “localhost.crt”
  • “server.key” with “localhost.key”

Changes shoud have been made to the following values in “<VirtualHost _default_:443>”

  • SSLCertificateFile “/private/etc/apache2/localhost.crt”

  • SSLCertificateKeyFile “/private/etc/apache2/localhost.key”

Check your apache conf

I had the following error

OSX SSL

I removed the invalid first line (I must have inserted something while searching using nano (I must have presses Option+W instead of Control+W to find in Nano)).

I ran the Apache config check again and received the following error.

I added my domain but with a 2 on the end for later use via local DNS.

<VirtualHost _default_:443>
...
ServerName https://www.mydomainname2.com:443
...

TIP: This will not by default allow you to load this address locally without, not without DNS changes (to be added soon).

Restart Apache (you may still receive a warning about the ServerName (I Ignored it)).

Restart Apache

Loading Your Site

You can now load the website localhost in your browser (it will show as insecure).

Insecure localhost

Click “Proceed to localhost (unsafe)”

The reason for no trust is the self-signed cert is not trusted by OSX or browsers.

Cert Details

OSX Certificate Trust (Key Chain Access)

Let’s tell OSX we trust this certificate (by adding it to the keychain)

Restart Apache

TIP: Check your certificate date in your browser, and clear your cache if older certificates are loading from cache.

Results

Great, Safari uses the SSL cert (and obeys Keychain’s Trust)

Safari

Google Chrome reports the Cert has issues (even though it is trusted locally). I will investigate and update this post soon.

Chrome

I suspect that Chrome will need to trust this cert fully to allow AJAX and API calls to be made.It’ss weird that Google Chrome has detected it is trusted by all users but does not trust it.

This is my next link to research a solution.

More

Read Useful OSX Terminal Commands or Useful OSX Linux Commands and Securing an Ubuntu VM with a free LetsEncrypt SSL certificate in 1 Minute first.

Hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.4 Nov 208 – Works on Mojave

v1.3 Small Edits, Added link to investigate Chrome issue (Current Version)

v1.2 Added More Explanations

v1.1 Reworded

v1.0 Initial Version

Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl

by

Below is my quick guide to showing how I recover lost files on a removable (or internal drive) with software on Windows. This is not a paid advert but this is what works for me (and has worked with everything I have thrown at it). Tell me what works for you (especially on OSX).

Advertisement:


“Helping one person might not change the world but it could change the world for one person” – Buddha.

I hope this guide helps someone, I have never seen someone happier than when I have been able to restore lost files or photos for someone. Not everyone backs up data (read my guide here).

Before you start

Don’t touch (format, access or install software) onto drives and USB drive that you want to restore files from or you may overwrite files on that drive. Doing this will potentially block the ability to restore files. Electronics die, software gets corrupted, this can be complex so If in doubt seek professional help and advice before proceeding. This is advice only and it’s your call with your data, Restoring files on damaged magnetic or USB drives can damage the data storage platter or storage chips, so proceed at your own risk. (legal disclaimer over, good luck).

More on how file systems and storage work

How File Systems Work

How the File Allocation Table Works

Installing DIY Data Recovery

The DIY Data Recovery software can restore images (photos) from a UBS, SD card or files (images or non-images) from data or operating system drives. If you have a system or data drive from a PC you can plug it into another PC running DIY Dat Recovery and scan and restore from the possibly bad drive to a good host drive, never restore files to a bad drive from a bad drive. I am an Apple Mac user now but only know of this Windows method for restoring files, please let me know of an Apple Mac way to restore files on FAT, NTFS and other file systems please.

I usually connect USB drives directly or connect SATA drives via a small caddy

On a working Windows computer install DIY Data Recovery software from here.

Install

Plug in your dead USB or another drive to restore files from (do not interact with the drive once Windows can see it).

Insert Drive

Do not format or modify the drive if prompted (cancel this screen).

Dont Format

Sometimes Windows will offer to fix it (do not interact with this dialog).

From memory, you can follow the steps below on a free trial version but you will need a paid version (serial number) to actually restore files. The free trial will show files that it can restore though. I paid about $60 for the software a few years ago, it is currently 59 EUROS. Enter your serial number if you have one.

Register iRecovery

After you register it (or continue with the free trial) select the drive to scan for files and click Next.

Select Drive

Here is another unreadable SATA drive

DIY Data Recovery will scan each sector and try and find files. On a smaller healthy drive, the scan won’t take long, on a larger drive (e.g a 2TB NTFS drive it may take 18+ hours).

In this case, we can see the USB drive has two bad sectors (red squares) at the front of the USB stick in the file system sectors (green squares). This possibly happened when the USB was pulled out when the operating system was writing to the USB stick. If the USB was dying the bad sectors would be at random positions.

Data fragments are listed as blue squares.

Let it scan

Here is another sample scan results

When the scan is done click ‘Save” to save the found sectors state (saving the state can save you time later).

Disclaimer: Clicking save does not save lost files it just saves the current scan state.

choose Files to recover

You can click the legend button to see the state of each file (Not processed, Presumably valid, Invalid, bad sectors etc)

Tick the files you want to be restored.

Legend

FYI: You will see some files you know, some files you forgot and some weird files. Restoring files is hit and miss and you may never see files again (do backup). Benjamin Franklin once said “Failing to plan is planning to fail”, You can also say “Failing to backup is guaranteeing lost data”.

Restoring Files

Do

  • Create a folder on your working Desktop to restore files to.
  • Do increase Read and Write cache
  • Do set longer Timeouts (e.g 1000, 5000, 10000, 60000)

Restore

Click “Start copying the selected files” to restore.

restore

Hopefully, you will have files restored, if not seek professional help. I’d recommend keeping the corrupt or source drive in a drawer, you may be able to use other tools to restore more files at a later date.

Restored Files

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

V1.1 Added more screenshot from a second DATA drive scan

v1.0 Initial Post

Building your first app on Heroku

by

Below is my blog post on setting up a hello world app on Heroku that is powered by a database (e.g Client requests a page, the page reads a database and return hello world).

Advertisement:


I found IBM/Cloudant would cost multiple thousands a month to run a good NoSQL database backend, I moved to a self-managed server on Digital Ocean in Singapore but the distance of the server killed latency and maximum throughput in Australia.  I moved to a dedicated AWS EC2 instance with a  Mongo DB cluster on AWS in Australia and it was great but pricey. A self-managed server (on Digital Ocean or Vultr) allows you to install whatever you want (e.g PHP, MySQL, MongoDB, Redis, SSL Certificate, Mail or WordPress) and configure what you want (but at what cost?).

Heroku say: “Get straight to building apps – Setting up, operating and maintaining your own platform is not where the race is won. Avoid the risk and complexity, and dedicate your energy to what really matters: building great apps.”.

heroku

Having setup serves on IBM Cloudant, Digital Ocean, AWS and Vultr (and even managing with RunCoud and Webmin) and not focusing on a self-managed server-issues like security, backup, security audits, upgrading PHP, updating ssl, setting up distributed mysql, updating https certificates etc sounds like the holy grail.

Let’s put that to the tets and build a Hello World App.

About Heroku

Heroku have a page here about what they offer.

Heroku

Sign up for Free: https://signup.heroku.com

Signup

Programming Languages (Ruby, PHP, Python, NodeJS, Java, Cloujure, Scala, Go etc). I so wanted to select “I’m not a developer” but I won’t.  I selected NodeJS.

Programming Language

Next: Verify your email, enter a password and you’ll find yourself here.

Setup Done

Now we can create an app?

App Help

Help

Ok, let’s skim the NodeJS help before we start: https://devcenter.heroku.com/articles/getting-started-with-nodejs#set-up

Let’s download the Command Line tools

Heroku CLI

Installing Heroku CLI on OSX was uneventful

Heroku CLI

 

I clicked the “I have installed the Heroku CLI” button.

Heroku wanted me to clone a repo but I logged in to Heroku the Heroku CLI and created a local Heroku folder first (in my local Terminal).

I checked the Heroku Version

I checked Heroku Help

heroku --help
Usage: heroku COMMAND

Help topics, type heroku help TOPIC for more details:

 access          manage user access to apps
 addons          tools and services for developing, extending, and operating your app
 apps            manage apps
 auth            heroku authentication
 authorizations  OAuth authorizations
 buildpacks      manage the buildpacks for an app
 certs           a topic for the ssl plugin
 ci              run an application test suite on Heroku
 clients         OAuth clients on the platform
 config          manage app config vars
 container       Use containers to build and deploy Heroku apps
 domains         manage the domains for an app
 drains          list all log drains
 features        manage optional features
 git             manage local git repository for app
 keys            manage ssh keys
 labs            experimental features
 local           run heroku app locally
 logs            display recent log output
 maintenance     manage maintenance mode for an app
 members         manage organization members
 notifications   display notifications
 orgs            manage organizations
 pg              manage postgresql databases
 pipelines       manage collections of apps in pipelines
 plugins         manage plugins
 ps              manage dynos (dynos, workers)
 redis           manage heroku redis instances
 regions         list available regions
 releases        manage app releases
 run             run a one-off process inside a Heroku dyno
 sessions        OAuth sessions
 spaces          manage heroku private spaces
 status          status of the Heroku platform
 teams           manage teams
 update          update CLI
 webhooks        setup HTTP notifications of app activity

I ran the advised commands

At this point, I followed the steps at https://devcenter.heroku.com/articles/getting-started-with-nodejs#deploy-the-app

Ran

The website is up

App Up

Where is this app located

Traceroute times out but appears to be in the eastern United States (AWS).

Q1) Can I change to Sydney Australia?

Twitter

Waiting for an answer from Twitter.

Virginia is a long way away.

Let’s test the SSL

It’s all good apart from on weak cipher (“TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa) WEAK (112)“).

SSL

A shodan scan of the IP reveals an open port 80 and 443 (as expected, nothing out of the ordinary)

Pushing Changes

I ran this to push the app.

I ran this to ensure the app was running

I ran this to open the app

This opened the apps URL in a browser.

A nice help page was pushed to my app.

I changed the contents of “node-js-getting-started/views/pages/index.ejs” and repushed.

It reported “Everything up-to-date“???

Repush

I tried to force a push (just in-case something on the server was changed)

???

Ok, let’s pull and see what’s new?

I edited “./index.js” and made the entry point “pages/index2.ejs” and tried to push and no luck.

I tried running push under sudo too (no luck)

Googling revealed I can go to https://dashboard.heroku.com/apps/YOUR-APP/deploy/heroku-gi and see what the issue is.

I logged into the app and could not find how to delete or edit it?

How do I edit files

At this point, I would have just ssh’ed in and restarted the node process in pm2 or node on my own self-managed server and edited remote files locally in sublime on my local machine.

Personally, I like to know where files live and not delve into another designers management interface.

I’ll come back to this guide later until then I’ll stick with Digital Ocean or Vultr

Linking to your own Domain on Heroku

It looks like I can use custom domains via Namecheap to link a domain to a heroku app. 

Or you can setup a domain sub domain to point to a heroku app with a DNS C Name record. 

Database Setup

Todo.. 

Node talking to the database. 

Todo.. 

App Cost on Heroku

todo: After Hello World is deployed. Digital Ocean (is as low as $5 a month) or Vultr (is as low as $2.5 a month)

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.4 added custom domain, database, CNAME info

etc

Connect to a remote server with ssh keys generated on OSX

by

Below is the way I connect to a remote server via SSH keys generated on OSX.

Advertisement:


Setting up a server

When you set up an Ubuntu server on Vultr (read my guide on setting up a Vultr server for as low as $2.5 a month) or Digital Ocean (use this link to get two months free when you setup an Ubuntu server on digital ocean) you can specify an SSH key to use for remote connections during the server create stage (old guide here).

How to create an SSH key on OSX to use to connect to a remote server

Run the following command (“sudo ssh-keygen -t rsa“) to generate an ssh key paid in “~/.ssh/” on OSX.

TIP: It is a good idea to also generate a passphrase to use with the key (double protection). You will be prompted to enter this password to use the RSA key.

You can now see the generated keys in ~/.ssh/

You can view the contents of the public file (you can use this when generating Digital Ocean, Vultr, AWS or Azure or other cloud servers).

fyi: Replace 123.123.123.123 with your remote serves ip.

How to connect to a server (the old way).

As long as your host added the desired public ssh key file contents to the server (adding the public ssh key contents to “~/.ssh/authorized_keys” you will be able to connect to the server.

Run the following command on OSX command line to connect to the server via SSH.

You should see..

Congratulations, you should now be able to connect to your server via SSH.

Securing your ubuntu Server

Read my guides here, here and here.

Don’t forget to add a firewall and set up an SSL certificate.

How to connect to a server (faster way).

todo: ~/.ssh/config method

Now we can connect to your remote server with the shorter method.

todo: ~/.ssh/config method

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.0 Initial Post

etc

Updating PHP 7.0 to 7.1 on an Ubuntu 16.04 Vultr VM

by

Here is how you can quickly update PHP 7.0 to 7.1 on a Vultr Ubuntu domain.

Advertisement:



I have configured a number of Vultr domains with NGINX and PHP 7.1 FPM and today I realised I need to update PHP 7.0 to 7.1 to fix a  few security exploits (read more here and here on securing Ubuntu in the cloud). PHP has a good page where you can keep up to date with PHP news here https://secure.php.net/. You can also view the PHP bug tracker to view bugs here. PHP aggregation user @php_net on twitter is good to follow, the official PHP twitter account is @official_php.

I have not noticed in daily Ubuntu package updates no option to update PHP 7.0 to 7.1, I must have to update manually.

WARNING: Backup your site and test this on a non production server before doing it on a live server.  I had an issue with PHP 7.1 breaking WordPress 3.9 (mysql issues with some plugins) and I had to roll back to 7.0 (see rollback tips in troubleshooting below). WordPress says it is PHP 7.1 compatible but issues exist. WordPress 3.9 ditches “mysql” and used “mysqli” and when instead PHP 7.1 WordPress could not find “mysqli”?

List packages with updates

You can run the following to view upgradable packages (TIP: Backup NGINX and other configuration files before any upgrades).

Update your server packages

Reboot

You should now see this on startup

You can view your installed PHP configuration file and installed version by typing to following in your servers command line.

Now let’s install a package viewer

Search installed packages (or non-installed) PHP packages.

Uninstall all local PHP related packages

Confirm packages are uninstalled

Install PHP 7.1 and common packages

Verify PHP 7.1 installation

Reboot

See if the PHP 7.1 FPM service has started

sudo systemctl | grep php
> php7.1-fpm.service

Restart PHP 7.1 FPM Service

Edit your /etc/nginx/sites-enabled/default and change the fastcgi_pass from “7.0” to “7.1”

Edits:

Reload NGINX configuration and restart NGINX

Your website should now be back up and running PHP 7.1

PHP 7.1

Post Install Tasks

View this blog post on other useful linux commands.

Run a Lynis security scan.

Edit your PHP.ini file and add required changes (e.g upload sizes).

sudo nano /etc/php/7.1/fpm/php.ini
# upload_max_filesize = 2M
+ upload_max_filesize = 8M

Troubleshooting

View PHP configuration values (add this to a debug.php and load in in a browser)

<?php

// Show all information, defaults to INFO_ALL
phpinfo();

// Show just the module information.
// phpinfo(8) yields identical results.
phpinfo(INFO_MODULES);

?>

I broke my WordPress 3.9 when I tried to update to PHP 7.1 so I rolled back to 7.0.

Google help had me stuck for a while when I had issues purging php 7.1.

Purge Error

Because my blog (with install steps) was down I used this site to help be find the commands to run.

Conclusion

Sometimes going with cutting edge tech you will go out on a limb, ensure you know and can restore a working site if need be.

Always have a backup and restore plan.

Hope this guide helps.

Donate and make this blog better


Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

v1.35 WordPress 3.9 error with PHP 7.1

Setting up additional server storage on cloud servers (block storage on Vultr)

by

Vultr has a generous disk quota with the cloud servers you can set up. But what do you do when you want more space than the default allocation (for backup or application data)?

Advertisement:



I have blogged before about setting up an Ubuntu server on the cloud on Vultr and configuring it if you do not already have a cloud server.

Vultr allows you to set up a server in minutes.

Server

A Vultr $2.5 a month server comes with 20GB storage, a $20 a month server comes with 60GB of SSD storage.

Vultr does offer more storage for about 0.10c per GB. At this time or writing Vultr allows you to add more storage to serves in NY/NJ (only). Read my guide on moving data between servers with RSync. And cond forget yo secure your server with a free SSL certificate and secure it (read more here and here).

An additional 10GB of storage would cost $1/m.

10GB

An additional 50GB of storage would cost $5/m.

50GB

An additional 100GB of storage would cost $10/m.

100GB

An additional 250 GB of storage would cost $25/m

250GB

View the Vultr pricing calculator here. Vultr does say that you can resize your block storage volume but there are manual actions and risks involved so get the space you need early on and prevent resizing later.

Read the Vultr Block Storage FAQ here: https://www.vultr.com/docs/block-storage

Vultr did offer early customers in (limited location’s) a free 50GB storage (read more on these limits here).

I am going to spin up a Block storage and attach to my server in Sydney.

fyi: Read the official guide on Attacking Block Storage to a Vultr server.

1. Login to your Vultr admin panel ( https://my.vultr.com/ ) and click Block Storage (  https://my.vultr.com/blockstorage/ ).

2. Click Add Block Storage

Add Block Storage

3. Choose the size of your block storage volume.

New Block Storage

Darn, I can’t choose Syndey yet as a location to create a block storage volume (I have asked Vultr when we can) so I’ll continue this guide with my existing (free) 50GB volume in New Jersey) and mount it in a server in NY/NJ (and also Syndey).

It appears I can’t connect to a  Block Storage volume outside the block storages location (data centre).

Manage Block Storage

 

You will need to attach the block storage volume to the server at that data centre location or you will get this error when you try and connect to it later.

Error

 

In my case, the server did not automatically restart so I manually restarted it.

 

Connecting the Block Storage to your VM

From the Vultr admin panel ( https://my.vultr.com ), Block Storage ( https://my.vultr.com/blockstorage/ ) you can manage individual Block Storage volumes and see the mounting information.

e.g

FYI: You can only connect to a  block storage from the same location (one server at a time I’d imagine).

4. From he Vultr Admin panel SSH into the server (in the same location).  See my guide here on setting up a Vultr server and configuring it.

Vultr say’s “Block storage is connected to your server as /dev/vdb. We do not create any filesystems on it by default.” Official Block storage documentation is located here.

5. Run the commands listed in the Block Storage screen (above)

Mount

Error: In my case, the echo command failed  add to configuration to the /etc/fstab file (even with sudo) and the mount command failed?

I checked the /etc/fstab file contents

I manually edited the /etc/fstab file and added the mount point configuration as suggested by Vultr.

Contents

6. I re-ran the mount command

sudo mount /mnt/blockstorage
#

Success

I can now directory list in the block storage volume.

ls /dev/vdb1 -al
brw-rw---- 1 username disk 253, 17 Nov  7 21:18 /dev/vdb1

 

Now let’s attach it to another folder in the root folder (e.g /data)

First, unmount the volume

Edit the /etc/fstab file with sudo nano

Change the mount point somewhere else (e.g /data)

Make a folder in the new path (/data), If you don’t do this the mount will fail.

Remount the volume (but use the new path)

sudo mount /data

You can now use the path and new storage.

cd /data
mkdir /data/test
cd /data/test
pwd
# /data/test/
sudo nano /data/test/test.txt

Nice

 

Disposing of Block Storage

TIP: Move or backup any data before you destroy or detach the volume.

First, you will need to unmount the volume (SSH session with your server).

Then remove the entry from the /etc/fstab file

Then you can navigate to the https://my.vultr.com/blockstorage/ and edit the said block storage volume and detach the volume (this will cause the server to reboot).

Detatch

After a few minutes you can delete the volume from the edited Block Storage Volume page  (click the Trashcan up the upper right).

Detatch

Done, You can now add and remove Block Storage volumes on Vultr.

 

How to check the disk usage of the block storage volume

You may need to remind yourself of the block storage volume (cat the /etc/fstab file and view the drive information on the mount line).

cat /etc/fstab
# .. /dev/vdb1 ..

How much space is used/free

df -h /dev/vdb1
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb1        50G   52M   47G   1% /data

You can also show the usage information in that mounted folder

du -xsch /data
10G     /folder1
10G     /folder2
20G     total

Use the pydf tool to view mounted partitions

Install pydf

Use pydf

 

Troubleshooting

  • You need to attach the block storage volume and reboot in the Vultr admin panel before mounting.
  • The echo command (as documented by Vultr) may not add information to the /etc/fstab file (a manual edit will work).

 

 

How to Resize a Block Storage Volume on Vultr.

Coming soon (if requested below).

 

Donate and make this blog better


Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

v1.2 added disk usage information