IoT, Code, Security and Server Stuff

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

MySQL

I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.

by

I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance. Here is what I did to set up a complete Ubuntu 18.04 system (NGINX, PHP, MySQL, WordPress etc). This is not a paid review (just me documenting my steps over 2 days).

Background (CPanel hosts)

In 1999 I hosted my first domain (www.fearby.com) on a host in Seattle (for $10 USD a month), the host used CPanel and all was good.  After a decade I was using the domain more for online development and the website was now too slow (I think I was on dial-up or ADSL 1 at the time). I moved my domain to an Australian host (for $25 a month).

After 8 years the domain host was sold and performance remained mediocre. After another year the new host was sold again and performance was terrible.

I started receiving Resource Limit Is Reached warnings (basically this was a plot by the new CPanel host to say “Pay us more and this message will go away”).

Page load times were near 30 seconds.

cpenal_usage_exceeded

The straw that broke the camel’s back was their demand of $150/year for a dodgy SSL certificate.

I needed to move to a self-managed server where I was in control.

Buying a Domain Name

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Self Managed Server

I found a good web IDE ( http://www.c9.io/ ) that allowed me to connect to a cloud VM.  C9 allowed me to open many files and terminal windows and reconnect to them later. Don’t get excited, though, as AWS have purchased C9 and it’s not the same.

C9 IDE

C9 IDE

I spun up a Digital Ocean Server at the closest data centre in Singapore. Here was my setup guide creating a Digital Ocean VM, connecting to it with C9 and configuring it. I moved my email to G Suite and moved my WordPress to Digital Ocean (other guides here and here).

I was happy since I could now send emails via CLI/code, set up free SSL certs, add second domain email to G Suite and Secure G Suite. No more usage limit errors either.

Self-managing servers requires more work but it is more rewarding (flexible, faster and cheaper).  Page load times were now near 20 seconds (10-second improvement).

Latency Issue

Over 6 months, performance on Digital Ocean (in Singapore) from Australia started to drop (mentioned here).  I tried upgrading the memory but that did not help (latency was king).

Moved the website to Australia

I moved my domain to Vultr in Australia (guide here and here). All was good for a year until traffic growth started to increase.

Blog Growth

I tried upgrading the memory on Vultr and I setup PHP child workers, setup Cloudflare.

GT Metrix scores were about a “B” and Google Page Speed Scores were in the lower 40’s. Page loads were about 14 seconds (5-second improvement).

Tweaking WordPress

I set up an image compression plugin in WordPress then set up a cloud image compression and CDN Plugin from the same vendor.  Page Speed info here.

GT Metrix scores were now occasionally an “A” and Page Speed Scores were in the lower 20’s. Page loads were about 3-5 seconds (10-second improvement).

Mixed bag from Vultr (more optimisation and performance improvements were needed).

This screenshot is showing poor www.gtmetrix.com scores , pool google page speed index scores and upgrading from 1GB to 2GB memory on my server.

Google Chrome Developer Console Audit Results on Vultr hosted website were not very good (I stopped checking as nothing helped).

This is a screenshot showing poor site performance (screenshot taken in Google Dev tools audit feature)

The problem was the Vultr server (400km away in Sydney) was offline (my issue) and everything above (adding more memory, adding 2x CDN’s (EWWW and Cloudflare), adding PHP Child workers etc) did not seem to help???

Enter UpCloud…

Recently, a friend sent a link to a blog article about a host called “UpCloud” who promised “Faster than SSD” performance.  This can’t be right: “Faster than SSD”? I was intrigued. I wanted to check it out as I thought nothing was faster than SSD (well, maybe RAM).

I signed up for a trial and ran a disk IO test (read the review here) and I was shocked. It’s fast. Very fast.

Summary: UpCloud was twice as fast (Disk IO and CPU) as Vultr (+ an optional $4/m firewall and $3/m for 1x backup).

This is a screenshot showing Vultr.com servers getting half the read and write disk io performance compared to upcloud.com.

fyi: Labels above are K Bytes per second. iozone loops through all file sizes from 4 KB to 16,348 KB and measures through the reads per second. To be honest, the meaning of the numbers doesn’t interest me, I just want to compare apples to apples.

This is am image showing iozone results breakdown chart (kbytes per sec on vertical axis, file size in horizontal axis and transfer size on third access)

(image snip from http://www.iozone.org/ which explains the numbers)

I might have to copy my website on UpCloud and see how fast it is.

Where to Deploy and Pricing

UpCloud Pricing: https://www.upcloud.com/pricing/

UpCloud Pricing

UpCloud does not have a data centre in Australia yet so why choose UpCloud?

Most of my site’s visitors are based in the US and UpCloud have disk IO twice as fast as Vultr (win-win?).  I could deploy to Chicago?

This image sows most of my visitors are in the US

My site’s traffic is growing and I need to ensure the site is fast enough in the future.

This image shows that most of my sites visitors are hitting my site on week days.

Advertisement:



Creating an UpCloud VM

I used a friend’s referral code and signed up to create my first VM.

FYI: use my Referral code and get $25 free credit.  Sign up only takes 2 minutes.

https://www.upcloud.com/register/?promo=D84793

When you click the link above you will receive 25$ to try out serves for 3 days. You can exit his trail and deposit $10 into UpCloud.

Trial Limitations

The trial mode restrictions are as following:

* Cloud servers can only be accessed using SSH, RDP, HTTP or HTTPS protocols
* Cloud servers are not allowed to send outgoing e-mails or to create outbound SSH/RDP connections
* The internet connection is restricted to 100 Mbps (compared to 500 Mbps for non-trial accounts)
* After your 72 hours free trial, your services will be deleted unless you make a one-time deposit of $10

UpCloud Links

The UpCloud support page is located here: https://www.upcloud.com/support/

More UpCloud links to read:

Signing up to UpCloud

Navigate to https://upcloud.com/signup and add your username, password and email address and click signup.

New UpCloud Signup Page

Add your address and payment details and click proceed (you don’t need to pay anything ($1 may be charged and instantly refunded to verify the card)

Add address and payment details

That’s it, check yout email.

Signup Done

Look for the UpCloud email and click https://my.upcloud.com/

Check Email

Now login

Login to UpCloud

Now I can see a dashboard 🙂

UpCloud Dashboard

I was happy to see 24/7 support is available.

This image shows the www.upcloud.com live chat

I opted in for the new dashboard

UpCloud new new dashboard

Deploy My First UpCloud Server

This is how I deployed a server.

Note: If you are going to deploy a server consider using my referral code and get $25 credit for free.

Under the “deploy a server” widget I named the server and chose a location (I think I was supposed to use an FQDN name -e.g., “fearby.com”). The deployment worked though. I clicked continue, then more options were made available:

  1. Enter a short server description.
  2. Choose a location (Frankfurt, Helsinki, Amsterdam, Singapore, London and Chicago)
  3. Choose the number of CPU’s and amount of memory
  4. Specify disk number/names and type (MaxIOPS or HDD).
  5. Choose an Operating System
  6. Select a Timezone
  7. Define SSH Keys for access
  8. Allowed login methods
  9. Choose hardware adapter types
  10. Where the send the login password

Deploy Server

FYI: How to generate a new SSH Key (on OSX or Ubuntu)

Output

Did the key export? (yes)

> /temp# ls /temp/ -al
> drwxr-xr-x 2 root root 4096 Jun 9 15:33 .
> drwxr-xr-x 27 root root 4096 Jun 8 14:25 ..
> -rw——- 1 user user 1766 Jun 9 15:33 example_rsa
> -rw-r–r– 1 user user 396 Jun 9 15:33 example_rsa.pub

“example_rsa” is the private key and “example_rsa.pub “is the public key.

  • The public key needs to be added to the server to allow access.
  • The private key needs to be added to any local ssh program used for remote access.

Initialisation script (after deployment)

I was pleased to see an initialization script section that calls actions after the server is deployed. I configured the initialisation script to pull down a few GB of backups from my Vultr website in Sydney (files now removed).

This was my Initialisation script:

Confirm and Deploy

I clicked “Confirm and deploy” but I had an alert that said trial mode can only deploy servers up to 1024MB of memory.

This image shows I cant deploy servers with 2/GB in trial modeExiting UpCloud Trial Mode

I opened the dashboard and clicked My Account then Billing, I could see the $25 referral credit but I guess I can’t use that in Trial.

I exited trial mode by depositing $10 (USD).

View Billing Details

Make a manual 1-time deposit of $10 to exit trial mode.

Deposit $10 to exit the trial

FYI: Server prices are listed below (or view prices here).

UpCloud Pricing

Now I can go back and deploy the server with the same settings above (1x CPU, 2GB Memory, Ubuntu 18.04, MaxIOPS Storage etc)

Advertisement:



Deployment takes a few minutes and depending on how you specified a password may be emailed to you.

UpCloud Server Deployed

The server is now deployed; now I can connect to it with my SSH program (vSSH).  Simply add the server’s IP, username, password and the SSH private key (generated above) to your ssh program of choice.

fyi: The public key contents start with “ssh-rsa”.

This image shows me connecting to my sever via ssh

I noticed that the initialisation script downloaded my 2+GB of files already. Nice.

UpCloud Billing Breakdown

I can now see on the UpCloud billing page in my dashboard that credit is deducted daily (68c); at this rate, I have 49 days credit left?

Billing Breakdown

I can manually deposit funds or set up automatic payments at any time 🙂

UpCloud Backup Options

You do not need to setup backups but in case you want to roll back (if things stuff up), it is a good idea. Backups are an additional charge.

I have set up automatic daily backups with an auto deletion after 2 days

To view backup scheduled click on your deployed server then click backup

List of UpCloud Backups

Note: Backups are charged at $0.056 for every GB stored – so $5.60 for every 100GB per month (half that for 50GB etc)

You can take manual backups at any time (and only be charged for the hour)

UpCloud Firewall Options

I set up a firewall at UpCloud to only allow the minimum number of ports (UpCloud DNS, HTTP, HTTPS and My IP to port 22).  The firewall feature is charged at $0.0056 an hour ($4.03 a month)

I love the ability to set firewall rules on incoming, destination and outgoing ports.

To view your firewall click on your deployed server then click firewall

UpCloud firewall

Update: I modified my firewall to allow inbound ICMP (IPv4/IPv6) and UDP (IPv4/IPv6) packets.

(Note: Old firewall screenshot)

Firewall Rules Allow port 80, 443 and DNS

Because my internet provider has a dynamic IP, I set up a VPN with a static IP and whitelisted it for backdoor access.

Local Ubuntu ufw Firewall

I duplicated the rules in my local ufw (2nd level) firewall (and blocked mail)

UpCloud Download Speeds

I pulled down a 1.8GB Ubuntu 18.08 Desktop ISO 3 times from gigenet.com and the file downloaded in 32 seconds (57MB/sec). Nice.

Install Common Ubuntu Packages

I installed common Ubuntu packages.

Timezone

I checked the server’s time (I thought this was auto set before I deployed)?

I reset the time to Australia/Sydney.

Now the timezone is set 🙂

Shell History

I increased the shell history.

SSH Login

I created a ~/.ssh/authorized_keys file and added my SSH public key to allow password-less logins.

I added my pubic ssh key, then exited the ssh session and logged back in. I can now log in without a password.

Install NGINX

nginx/1.14.0 is now installed.

A quick GT Metrix test.

This image shows awesome static nginx performance ratings of of 99%

Install MySQL

Run these commands to install and secure MySQL.

Securing the MySQL server deployment.
> Would you like to setup VALIDATE PASSWORD plugin?: n
> New password: **********************************************
> Re-enter new password: **********************************************
> Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
> Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
> Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
> Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
> Success.

I disabled the validate password plugin because I hate it.

MySQL Ver 14.14 Distrib 5.7.22 is now installed.

Set MySQL root login password type

Set MySQL root user to authenticate via “mysql_native_password”. Run the “mysql” command.

Now let’s set the root password authentication method to “mysql_native_password”

Check authentication method.

Now we need to flush permissions.

Done.

Advertisement:



Install PHP

Install PHP 7.2

PHP 7.2.5, Zend Engine v3.2.0 with Zend OPcache v7.2.5-1 is now installed. Do update PHP frequently.

I made the following changes in /etc/php/7.2/fpm/php.ini

> cgi.fix_pathinfo=0
> max_input_vars = 1000
> memory_limit = 1024M
> max_file_uploads = 20M
> post_max_size = 20M

Install PHP Modules

sudo apt-get install php-pear php7.2-curl php7.2-dev php7.2-mbstring php7.2-zip php7.2-mysql php7.2-xml

Install PHP FPM

Configure PHP FPM config.

Edit /etc/php/7.2/fpm/php.ini

> cgi.fix_pathinfo=0
> max_input_vars = 1000
> memory_limit = 1024M
> max_file_uploads = 20M
> post_max_size = 20M

Reload php sudo service.

Install PHP Modules

Configuring NGINX

If you are not comfortable editing NGINX config files read here, here and here.

I made a new “www root” folder, set permissions and created a default html file.

I edited the “root” key in “/etc/nginx/sites-enabled/default” file and set the root a new location (e.g., “/www-root”)

I added these performance tweaks to /etc/nginx/nginx.conf

> worker_cpu_affinity auto;
> worker_rlimit_nofile 100000

I add the following lines to “http {” section in /etc/nginx/nginx.conf

Check NGINX Status

Install Open SSL that supports TLS 1.3

This is a work in progress. The steps work just fine for me on Ubuntu 16.04. but not Ubuntu 18.04.?

Installing Adminer MySQL GUI

I will use the PHP based Adminer MySQL GUI to export and import my blog from one server to another. All I needed to do is install it on both servers (simple 1 file download)

Use Adminer to Export My Blog (on Vultr)

On the original server open Adminer (http) and..

  1. Login with the MySQL root account
  2. Open your database
  3. Choose “Save” as the output
  4. Click on Export

This image shows the export of the wordpress adminer page

Save the “.sql” file.

I used Adminer on the UpCloud server to Import My Blog

FYI: Depending on the size of your database backup you may need to temporarily increase your upload and post sizes limits in PHP and NGINX before you can import your database.

Edit /etc/php/7.2/fpm/php.ini
> max_file_uploads = 100M
> post_max_size =100M

And Edit: /etc/nginx/nginx.conf
> client_max_body_size 100M;

Don’t forget to reload NGINX config and restart NGINX and PHP. Take note of the maximum allowed file size in the screenshot below. I temporarily increased my upload limits to 100MB in order to restore my 87MB blog.

Now I could open Adminer on my UpCloud server.

  1. Create a new database
  2. Click on the database and click Import
  3. Choose the SQL file
  4. Click Execute to import it

Import MuSQL backup with Adminer

Don’t forget to create a user and assign permissions (as required – check your wp-config.php file).

Import MySQL Database

Tip: Don’t forget to lower the maximum upload file size and max post size after you import your database,

Cloudflare DNS

I use Cloudflare to manage DNS, so I need to tell it about my new server.

You can get your server’s IP details from the UpCloud dashboard.

Find IP

At Cloudflare update your DNS details to point to the server’s new IPv4 (“A Record”) and IPv6 (“AAAA Record”).

Cloudflare DNS

Advertisement:



Domain Error

I waited an hour and my website was suddenly unavailable.  At first, I thought this was Cloudflare forcing the redirection of my domain to HTTP (that was not yet set up).

DNS Not Replicated Yet

I chatted with UpCloud chat on their webpage and they kindly assisted me to diagnose all the common issues like DNS values, DNS replication, Cloudflare settings and the error was pinpointed to my NGINX installation.  All NGINX config settings were ok from what we could see?  I uninstalled NGINX and reinstalled it (and that fixed it). Thanks UpCloud Support 🙂

Reinstalled NGINX

I reinstalled NGINX and reconfigured /etc/nginx/nginx.conf (I downloaded my SSL cert from my old server just in case).

Here is my /etc/nginx/nginx.conf file.

Here is my /etc/nginx/sites-available/default file (fyi, I have not fully re-setup TLS 1.3 yet so I commented out the settings)

SSL Labs SSL Certificate Check

All good thanks to the config above.

SSL Labs

Install WP-CLI

I don’t like setting up FTP to auto-update WordPress plugins. I use the WP-CLI tool to manage WordPress installations by the command line. Read my blog here on using WP-CLI.

Download WP-CLI

Move WP-CLI to the bin folder as “wp”

Test wp

Update WordPress Plugins

Now I can run “wp plugin update” to update all WordPress plugins

Update WordPress Core

WordPress core file can be updated with “wp core update“

Troubleshooting: Use the flag “–allow-root “if wp needs higher access (unsafe action though).

Install PHP Child Workers

I edited the following file to setup PHP child workers /etc/php/7.2/fpm/pool.d/www.conf

Changes

> pm = dynamic
> pm.max_children = 40
> pm.start_servers = 15
> pm.min_spare_servers = 5
> pm.max_spare_servers = 15
> pm.process_idle_timeout = 30s;
> pm.max_requests = 500;
> php_admin_value[error_log] = /var/log/www-fpm-php.www.log
> php_admin_value[memory_limit] = 512M

Advertisement:



Restart PHP

Test NGINX config, reload NGINX config and restart NGINX

Output (14 workers are ready)

Memory Tweak (set at your own risk)

vm.swappiness = 1

Setting swappiness to a value of 1 all but disables the swap file and tells the Operating System to aggressively use ram, a value of 10 is safer. Only set this if you have enough memory available (and free).

Possible swappiness settings:

> vm.swappiness = 0 Swap is disabled. In earlier versions, this meant that the kernel would swap only to avoid an out of memory condition when free memory will be below vm.min_free_kbytes limit, but in later versions, this is achieved by setting to 1.[2]> vm.swappiness = 1 Kernel version 3.5 and over, as well as Red Hat kernel version 2.6.32-303 and over: Minimum amount of swapping without disabling it entirely.
> vm.swappiness = 10 This value is sometimes recommended to improve performance when sufficient memory exists in a system.[3]
> vm.swappiness = 60 The default value.
> vm.swappiness = 100 The kernel will swap aggressively.

The “htop” tool is a handy memory monitoring tool to “top”

Also you can use good old “watch” command to show near-live memory usage (auto-refreshes every 2 seconds)

Script to auto-clear the memory/cache

As a habit, I am setting up a cronjob to check when free memory falls below 100MB, then cache is automatically cleared (freeing memory).

Script Contents: clearcache.sh

I set the cronjob to run every 15 mins, I added this to my cronjob.

Sample log output

I will check the log (/scripts/clearcache.log) in a few days and view the memory trends.

After 1/2 a day Ubuntu 18.04 is handling memory just fine, no externally triggered cache clears have happened 🙂

Free memory over time

I used https://crontab.guru/every-hour to set the right schedule in crontab.

I rebooted the VM.

Update: I now use Nixstats monitoring

Swap File

FYI: Here is a handy guide on viewing swap file usage here. I’m not using swap files so it is only an aside.

After the system rebooted I checked if the swappiness setting was active.

Yes, swappiness is set.

File System Tweaks – Write Back Cache (set at your own risk)

First, check your disk name and file system

Take note of your disk name (e.g vda1)

I used TuneFS to enable writing data to the disk before writing to the journal. tunefs is a great tool for setting file system parameters.

Warning (snip from here): “I set the mode to journal_data_writeback. This basically means that data may be written to the disk before the journal. The data consistency guarantees are the same as the ext3 file system. The downside is that if your system crashes before the journal gets written then you may loose new data — the old data may magically reappear.“

Warning this can corrupt your data. More information here.

I ran this command.

I edited my fstab to append the “writeback,noatime,nodiratime” flags for my volume after a reboot.

Edit FS Tab:

I added “writeback,noatime,nodiratime” flags to my disk options.

Updating Ubuntu Packages

Show updatable packages.

Update Packages.

Unattended Security Updates

Read more on Ubuntu 18.04 Unattended upgrades here, here and here.

Install Unattended Upgrades

Enable Unattended Upgrades.

Now I configure what packages not to auto update.

Edit /etc/apt/apt.conf.d/50unattended-upgrades

Find “Unattended-Upgrade::Package-Blacklist” and add packages that you don’t want automatically updated, you may want to manually update these (and monitor updates).

I prefer not to auto-update critical system apps (I will do this myself).

FYI: You can find installed packages by running this command:

Enable automatic updates by editing /etc/apt/apt.conf.d/20auto-upgrades

Edit the number at the end (the number is how many days to wait before updating) of each line.

> APT::Periodic::Update-Package-Lists “1”;
> APT::Periodic::Download-Upgradeable-Packages “1”;
> APT::Periodic::AutocleanInterval “7”;
> APT::Periodic::Unattended-Upgrade “1”;

Set to “0” to disable automatic updates.

The results of unattended-upgrades will be logged to /var/log/unattended-upgrades

Update packages now.

Almost done.

I Rebooted

Advertisement:



GT Metrix Score

I almost fell off my chair. It’s an amazing feeling hitting refresh in GT Metrix and getting sub-2-second score consistently (and that is with 17 assets loading and 361KB of HTML content)

0.9sec load times

WebPageTest.org Test Score

Nice. I am not sure why the effective use of CDN has an X rating as I have the EWWW CDN and Cloudflare. First Byte time is now a respectable “B”, This was always bad.

Update: I found out the longer you set cache delays in Cloudflare the higher the score.

Web Page Test

GT Metrix has a nice historical breakdown of load times (night and day).

Upcloud Site Speed in GTMetrix

Google Page Speed Insight Desktop Score

I benchmarked with https://developers.google.com/speed/pagespeed/insights/

This will help with future SEO rankings. It is well known that Google is pushing fast servers.

100% Desktop page speed score

Google Chrome 70 Dev Console Audit (Desktop)

100% Chrome Audit Score

This is amazing, I never expected to get this high score.  I know Google like (and are pushing) sub-1-second scores.

My site is loading so well it is time I restored some old features that were too slow on other servers

  • I disabled Lazy loading of images (this was not working on some Android devices)
  • I re-added the News Widget and news images.

GTMetrix and WebpageTest sores are still good (even after adding bloat)

Benchmarks are still good

My WordPress site is not really that small either

Large website

FYI: WordPress Plugins I use.

These are the plugins I use.

How I use these plugins to speed up my site.

  • I use EWWW Image Optimizer plugin to auto-compress my images and to provide a CDN for media asset deliver (pre-Cloudflare). Learn more about ExactDN and EWWW.io here.
  • I use Autoptimize plugin to optimise HTML/CSS/JS and ensure select assets are on my EWWW CDN. This plugin also removes WordPress Emojis, removed the use of Google Fonts, allows you to define pre-configured domains, Async Javascript-files etc.
  • I use BJ Lazy Load to prevent all images in a post from loading on load (and only as the user scrolls down the page).
  • GTmetrix for WordPress and Cloudflare plugins are for information only?
  • I use WP-Optimize to ensure my database is healthy and to disable comments/trackbacks and pingbacks.

Let’s Test UpCloud’s Disk IO in Chicago

Looks good to me, Read IO is a little bit lower than UpCloud’s Singapore data centre but still, it’s faster than Vultr.  I can’t wait for more data centres to become available around the world.

Why is UpCloud Disk IO so good?

I asked UpCloud on Twitter why the Disk IO was so good.

  • “MaxIOPS is UpCloud’s proprietary block-storage technology. MaxIOPS is physically redundant storage technology where all customer’s data is located in two separate physical devices at all times. UpCloud uses InfiniBand (!) network to connect storage backends to compute nodes, where customers’ cloud servers are running. All disks are enterprise-grade SSD’s. And using separate storage backends, it allows us to live migrate our customers’ cloud servers freely inside our infrastructure between compute nodes – whether it be due to hardware malfunction (compute node) or backend software updates (example CPU vulnerability and immediate patching).“

My Answers to Questions to support

Q1) What’s the difference between backups and snapshots (a Twitter user said Snapshots were a thing)

A1) Backups and snapshots are the same things with our infrastructure.

Q2) What are charges for backup of a 50GB drive?

A2) We charge $0.06 / GB of the disk being captured. But capture the whole disk, not just what was used. So for a 50GB drive, we charge $0.06 * 50 = $3/month. Even if 1GB were only used.

  • Support confirmed that each backup is charged (so 5 times manual backups are charged 5 times). Setting up a daily auto backup schedule for 2 weeks would create 14 billable backup charges.
  • I guess a 25GB server will be $1.50 a month

Q3) What are data charges if I go over my 2TB quota?

A3) Outgoing data charges are $0.056/GB after the pre-configured allowance.

Q4) What happens if my balance hits $0?

A4) You will get notification of low account balance 2 weeks in advance based on your current daily spend. When your balance reaches zero, your servers will be shut down. But they will still be charged for. You can automatically top-up if you want to assign a payment type from your Control Panel. You deposit into your balance when you want. We use a prepay model of payment, so you need to top up before using, not billing you after usage. We give you lots of chances to top-up.

Support Tips

  • One thing to note, when deleting servers (CPU, RAM) instances, you get the option to delete the storages separately via a pop-up window. Choose to delete permanently to delete the disk, to save credit. Any disk storage lying around even unattached to servers will be billed.
  • Charges are in USD.

I think it’s time to delete my domain from Vultr in Sydney.

Deleted my Vultr domain

I deleted my Vultr domain.

Delete Vultr Server

Done.

More Reading on UpCloud

https://www.upcloud.com/documentation/faq/

UpCloud Server Status

http://status.upcloud.com

Check out my new guide on Nixstats for awesome monitoring

What I would like

  1. Ability to name individual manual backups (tag with why I backed up).
  2. Ability to push user defined data from my VM to the dashboard
  3. Cheaper scheduled backups
  4. Sydney data centres (one day)

Advertisement:



Update: Post UpCloud Launch Tweaks (Awesome)

I had a look at https://www.webpagetest.org/ results to see where else I can optimise webpage delivery.

Optimisation Options

Disable dasjhicons.min.css (for unauthenticated WordPress users).

Find functions.php in the www root

Edit functions.php

Add the following

// Remove dashicons in frontend for unauthenticated users
add_action( 'wp_enqueue_scripts', 'bs_dequeue_dashicons' );
function bs_dequeue_dashicons() {
    if ( ! is_user_logged_in() ) {
        wp_deregister_style( 'dashicons' );
    }
}

HTTP2 Push

I added http2 to my listening servers

I tested a http2 push page by defining this in /etc/nginx/sites-available/default 

Once I tested that push (demo here) was working I then defined two files to push that were being sent from my server

I used the WordPress Plugin Autoptimize to remove Google font usage (this removed a number of files being loaded when my page loads).

I used the WordPress Plugin WP-Optimize plugin into to remove comments and disable pingbacks and trackbacks.

WordPress wp-config.php tweaks

Add 2FA Authentication to server logins.

I recently checked out YubiCo YubiKeys and I have secured my Linux servers with 2FA prompts at login. Read the guide here. I secured my WordPress too.

Tweaks Todo

  • Compress placeholder BJ Lazy Load Image (plugin is broken)
  • Solve 2x Google Analytics tracker redirects (done, switched to Matomo)

Conclusion

I love UpCloud’s fast servers, give them a go (use my link and get $25 free credit).

I love Cloudflare for providing a fast CDN.

I love ewww.io’s automatic Image Compression and Resizing plugin that automatically handles image optimisations and pre Cloudflare/first hit CDN caching.

Read my post about server monitoring with Nixstats here.

Let the results speak for themselves (sub <1 second load times).

Results

I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

 

2020 Update. I have stopped using Putty and WinSCP. I now use MobaXterm (a tabbed SSH client for Windows) as it is way faster than WinSCP and better than Putty. Read my review post of MobaXTerm here.

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v2.1 Newer GTMetrix scores

v2.0 New UpCloud UI Update and links to new guides.

v1.9 Spelling and grammar

v1.8 Trial mode gotcha (deposit money ASAP)

v1.7 Added RSA Private key info

v1.7 – Added new firewall rules info.

v1.6 – Added more bloat to the site, still good.

v1.5 Improving Accessibility

v1.4 Added Firewall Price

v1.3 Added wp-config and plugin usage descriptions.

v1.2 Added GTMetrix historical chart.

v1.1 Fixed free typos and added final conclusion images.

v1.0 Added final results

v0.9 added more tweaks (http2 push, removing unwanted files etc)

v0.81 Draft  – Added memory usage chart and added MaxIOPS info from UpCloud.

v0.8 Draft post.

I set the cronjob to run every 15 mins, I added this to my cronjob.

 

Sample log output

 

I will check the log (/scripts/clearcache.log) in a few days and view the memory trends.

After 1/2 a day Ubuntu 18.04 is handling memory just fine, no externally triggered cache clears have happened 🙂

Free memory over time

I used https://crontab.guru/every-hour to set the right schedule in crontab.

I rebooted the VM.

Update: I now use Nixstats monitoring

Swap File

FYI: Here is a handy guide on viewing swap file usage here. I’m not using swap files so it is only an aside.

After the system rebooted I checked if the swappiness setting was active.

 

Yes, swappiness is set.

File System Tweaks – Write Back Cache (set at your own risk)

First, check your disk name and file system

 

Take note of your disk name (e.g vda1)

I used TuneFS to enable writing data to the disk before writing to the journal. tunefs is a great tool for setting file system parameters.

Warning (snip from here): “I set the mode to journal_data_writeback. This basically means that data may be written to the disk before the journal. The data consistency guarantees are the same as the ext3 file system. The downside is that if your system crashes before the journal gets written then you may loose new data — the old data may magically reappear.“

Warning this can corrupt your data. More information here.

I ran this command.

 

I edited my fstab to append the “writeback,noatime,nodiratime” flags for my volume after a reboot.

Edit FS Tab:

 

I added “writeback,noatime,nodiratime” flags to my disk options.

 

Updating Ubuntu Packages

Show updatable packages.

 

Update Packages.

 

Unattended Security Updates

Read more on Ubuntu 18.04 Unattended upgrades here, here and here.

Install Unattended Upgrades

 

Enable Unattended Upgrades.

 

Now I configure what packages not to auto update.

Edit /etc/apt/apt.conf.d/50unattended-upgrades

Find “Unattended-Upgrade::Package-Blacklist” and add packages that you don’t want automatically updated, you may want to manually update these (and monitor updates).

I prefer not to auto-update critical system apps (I will do this myself).

 

FYI: You can find installed packages by running this command:

 

Enable automatic updates by editing /etc/apt/apt.conf.d/20auto-upgrades

Edit the number at the end (the number is how many days to wait before updating) of each line.

> APT::Periodic::Update-Package-Lists “1”;
> APT::Periodic::Download-Upgradeable-Packages “1”;
> APT::Periodic::AutocleanInterval “7”;
> APT::Periodic::Unattended-Upgrade “1”;

Set to “0” to disable automatic updates.

The results of unattended-upgrades will be logged to /var/log/unattended-upgrades

Update packages now.

 

Almost done.

I Rebooted

Advertisement:



GT Metrix Score

I almost fell off my chair. It’s an amazing feeling hitting refresh in GT Metrix and getting sub-2-second score consistently (and that is with 17 assets loading and 361KB of HTML content)

0.9sec load times

WebPageTest.org Test Score

Nice. I am not sure why the effective use of CDN has an X rating as I have the EWWW CDN and Cloudflare. First Byte time is now a respectable “B”, This was always bad.

Update: I found out the longer you set cache delays in Cloudflare the higher the score.

Web Page Test

GT Metrix has a nice historical breakdown of load times (night and day).

Upcloud Site Speed in GTMetrix

Google Page Speed Insight Desktop Score

I benchmarked with https://developers.google.com/speed/pagespeed/insights/

This will help with future SEO rankings. It is well known that Google is pushing fast servers.

100% Desktop page speed score

Google Chrome 70 Dev Console Audit (Desktop)

100% Chrome Audit Score

This is amazing, I never expected to get this high score.  I know Google like (and are pushing) sub-1-second scores.

My site is loading so well it is time I restored some old features that were too slow on other servers

  • I disabled Lazy loading of images (this was not working on some Android devices)
  • I re-added the News Widget and news images.

GTMetrix and WebpageTest sores are still good (even after adding bloat)

Benchmarks are still good

My WordPress site is not really that small either

Large website

FYI: WordPress Plugins I use.

These are the plugins I use.

How I use these plugins to speed up my site.

  • I use EWWW Image Optimizer plugin to auto-compress my images and to provide a CDN for media asset deliver (pre-Cloudflare). Learn more about ExactDN and EWWW.io here.
  • I use Autoptimize plugin to optimise HTML/CSS/JS and ensure select assets are on my EWWW CDN. This plugin also removes WordPress Emojis, removed the use of Google Fonts, allows you to define pre-configured domains, Async Javascript-files etc.
  • I use BJ Lazy Load to prevent all images in a post from loading on load (and only as the user scrolls down the page).
  • GTmetrix for WordPress and Cloudflare plugins are for information only?
  • I use WP-Optimize to ensure my database is healthy and to disable comments/trackbacks and pingbacks.

Let’s Test UpCloud’s Disk IO in Chicago

Looks good to me, Read IO is a little bit lower than UpCloud’s Singapore data centre but still, it’s faster than Vultr.  I can’t wait for more data centres to become available around the world.

Why is UpCloud Disk IO so good?

I asked UpCloud on Twitter why the Disk IO was so good.

  • “MaxIOPS is UpCloud’s proprietary block-storage technology. MaxIOPS is physically redundant storage technology where all customer’s data is located in two separate physical devices at all times. UpCloud uses InfiniBand (!) network to connect storage backends to compute nodes, where customers’ cloud servers are running. All disks are enterprise-grade SSD’s. And using separate storage backends, it allows us to live migrate our customers’ cloud servers freely inside our infrastructure between compute nodes – whether it be due to hardware malfunction (compute node) or backend software updates (example CPU vulnerability and immediate patching).“

My Answers to Questions to support

Q1) What’s the difference between backups and snapshots (a Twitter user said Snapshots were a thing)

A1) Backups and snapshots are the same things with our infrastructure.

Q2) What are charges for backup of a 50GB drive?

A2) We charge $0.06 / GB of the disk being captured. But capture the whole disk, not just what was used. So for a 50GB drive, we charge $0.06 * 50 = $3/month. Even if 1GB were only used.

  • Support confirmed that each backup is charged (so 5 times manual backups are charged 5 times). Setting up a daily auto backup schedule for 2 weeks would create 14 billable backup charges.
  • I guess a 25GB server will be $1.50 a month

Q3) What are data charges if I go over my 2TB quota?

A3) Outgoing data charges are $0.056/GB after the pre-configured allowance.

Q4) What happens if my balance hits $0?

A4) You will get notification of low account balance 2 weeks in advance based on your current daily spend. When your balance reaches zero, your servers will be shut down. But they will still be charged for. You can automatically top-up if you want to assign a payment type from your Control Panel. You deposit into your balance when you want. We use a prepay model of payment, so you need to top up before using, not billing you after usage. We give you lots of chances to top-up.

Support Tips

  • One thing to note, when deleting servers (CPU, RAM) instances, you get the option to delete the storages separately via a pop-up window. Choose to delete permanently to delete the disk, to save credit. Any disk storage lying around even unattached to servers will be billed.
  • Charges are in USD.

I think it’s time to delete my domain from Vultr in Sydney.

Deleted my Vultr domain

I deleted my Vultr domain.

Delete Vultr Server

Done.

 

Check out my new guide on Nixstats for awesome monitoring

What I would like

  1. Ability to name individual manual backups (tag with why I backed up).
  2. Ability to push user defined data from my VM to the dashboard
  3. Cheaper scheduled backups
  4. Sydney data centres (one day)

 

Update: Post UpCloud Launch Tweaks (Awesome)

I had a look at https://www.webpagetest.org/ results to see where else I can optimise webpage delivery.

Optimisation Options

HTTP2 Push

I added http2 to my listening servers I tested a http2 push page by defining this in /etc/nginx/sites-available/default 

Once I tested that push (demo here) was working I then defined two files to push that were being sent from my server

2FA Authentication at login

I recently checked out YubiCo YubiKeys and I have secured my Linux servers with 2FA prompts at login. Read the guide here. I secured my WordPress aswel.

Performance

I used the WordPress Plugin Autoptimize to remove Google font usage (this removed a number of files being loaded when my page loads).

I used the WordPress Plugin WP-Optimize plugin into to remove comments and disable pingbacks and trackbacks.

Results

Conclusion

I love UpCloud’s fast servers, give them a go (use my link and get $25 free credit).

I love Cloudflare for providing a fast CDN.

I love ewww.io’s automatic Image Compression and Resizing plugin that automatically handles image optimisations and pre Cloudflare/first hit CDN caching.

Read my post about server monitoring with Nixstats here.

Let the results speak for themselves (sub <1 second load times).

More Reading on UpCloud

https://www.upcloud.com/documentation/faq/

UpCloud Server Status

http://status.upcloud.com

I hope this guide helps someone.

Free Credit

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

2020 Update. I have stopped using Putty and WinSCP. I now use MobaXterm (a tabbed SSH client for Windows) as it is way faster than WinSCP and better than Putty. Read my review post of MobaXTerm here.

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v2.2 Converting to Blocks

v2.1 Newer GTMetrix scores

v2.0 New UpCloud UI Update and links to new guides.

v1.9 Spelling and grammar

v1.8 Trial mode gotcha (deposit money ASAP)

v1.7 Added RSA Private key info

v1.7 – Added new firewall rules info.

v1.6 – Added more bloat to the site, still good.

v1.5 Improving Accessibility

v1.4 Added Firewall Price

v1.3 Added wp-config and plugin usage descriptions.

v1.2 Added GTMetrix historical chart.

v1.1 Fixed free typos and added final conclusion images.

v1.0 Added final results

v0.9 added more tweaks (http2 push, removing unwanted files etc)

v0.81 Draft  – Added memory usage chart and added MaxIOPS info from UpCloud.

v0.8 Draft post.

How to backup and restore a MySQL database on Windows and Linux

by

Why backup and restore

This is a quick guide demonstrating how you can backup and restore a MySQL database on Windows and Linux using Adminer.

You may need to know how to backup a restore a database for a number of reasons..

e.g

  • Send the database to someone to debug or give feedback while learning.
  • Move the database from a local machine to the cloud
  • Move the database from cloud vendor A to cloud vendor B
  • etc.

Advertisement:



Having a backup of the VM is good but having a backup of the database too is better. I use UpCloud for hosting my VM’s and setting backups is easy. But I cannot download those backups.

UpCloud Backup Screen

Murphy’s Law

“If anything can go wrong, it will”

The most important reason for taking a backup and knowing how to restore it is for disaster recovery reasons.

Backup (the easiest way) with Adminer

Adminer is a free PHP based IDE for MySQL and other databases. Simply install Adminer and save the file on your local computer or remote web server directory.

FYI: The Adminer author Jakub Vrana has a patron page, I am a patron of this awesome software.

Snip from Adminers website. “Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch andMongoDB.”

adminer.php file icon screenshot

TIP: The file would be publicly accessible to anyone so don’t save it to a common area, obfuscate the file, protect it of delete the file when you are done using it.

Advertisement:



Once Adminer is installed load it in a web browser, login with your MySQL credentials. Once you login you will see all databases and an Import and Export menu.

Adminer main screen, all databases and import and export menu.

tbtest is a simple database with one table and 4 fields (ID, Key, Value and Modified)

.Click Export to open the export screen.

Export screen showing a list of databases and export options

Click Export, a SQL file will be generated (this is the export of the database).

Here is a save of the file:
https://fearby.com/wp-content/uploads/export.txt

Exported view of https://dev.mysql.com/doc/workbench/en/wb-admin-export-import-management.html

Its that simple.

If I add a binary blob file to the table and upload a PNG file lets see how the export looks.

Screenshot o the new table with a blog field in Adminer UI

Advertisement:



Let export the database again in Adminer and check out the output. I used Sublime Text editor to view the export file.

New Export shows the binary file in the Backup SQL file

Restore (the easiest way) with Adminer

OK lets delete the tbtest database and then restore it with Adminer. I used Adminer to delete (DROP) the database.

Database dropped with Adminer

Database “dbtest” deleted.

Now lets create a blank database to restore to (same name).

Create database screen.

Database created.

dbtest created.

Advertisement:



Now lets import the database backup using Adminer.

Click Import, select the backup file and un-tick Stop on errors.

Import screenshot, dxtest selectded, Restore file selected, stop on errors disabled

TIP: The 2MB next the the choose file button is defined by your web server and PHP configuration. If you are trying to import a larger database (e.g 80MB) first increase the limits in your web server and PHP (via php.ini).

The Import (restore should take seconds)

Import Success

The database was imported from a backup, all tables and records imported just fine.

The database was imported from a backup

Bonus methods.

On Ubuntu use this guide to backup from the command line. If you use the Oracle MySQL Workbench read this.

I hope this helps someone.

Advertisement:



Setting up the free MySQL database server on Windows 10

by

This guide assumes you are a beginner using Windows 10 and maybe have a default website configured on Windows 10 using the built in Information Server (IIS) along with PHP (e.g PHP 7.3.4 or greater).

If you are have never used Internet Information Server (IIS) then XAMPP is a great way (single install) to setup a web server, database server and PHP with little fuss.

Advertisement:



In this case I will manually install MySQL Community Server to add alongside Internet Information Server (IIS) on Windows 10.

Downloading
MySQL Server (Community Edition)

Go to here and click MySQL Installer for Windows.

Screenshot of Download installer link

Click MySQL Installer for Windows.

fyi: I sent myself on a goose chase as I could only see a 32 bit installer (I spent days trying to find a 64-bit installer or manual install binaries that were 64 bit). I should have read the bit that said “MySQL Installer is 32 bit, but will install both 32 bit and 64 bit binaries.“

MySQL Installer is 32 bit, but will install both 32 bit and 64 bit binaries.

You can read the installer documentation here if you wish.

I downloaded the larger of the two available installers (one 16MB the other 300+ MB, same version etc.). I had to login with an Oracle ID to start the download.

Download MySQL Installer screenshot

Install file downloaded

Installing MySQL Server (Community Edition)

I started the installer (accepted the licence agreement)

I accepted the licence agreement

I selected “Full Install“

I could have selected server only or custom.

I selected Full Install

I downloaded and installed Python 3.7, Thanks MySQL Installer for adding a link to the Python download.

Advertisement:



I Installed Python 3.7

After Python was installed I clicked refresh in MySQL and now MySQL can see Python 3.7.

Now MySQL can see Python 3.7.

I had a Visual Studio plugin install error (because Visual Studio was not installed).

Visual Studio plugin install error (Visual Studio is not installed)

Full Install (all components selected) reported the items are ready for install.

Full Install (all components selected)

Installation status complete.

Installation status list (Full Install)

List of items to configure (post install)

List of items to configure.

I setup a standard MySQL Server (not a Cluster)

Advertisement:



I setup a standard MySQL Server (not a Cluster)

I setup MySQL as a standard Development computer on port 3306 over TCP/IP (no Named Pipe, Shared Memory etc).

I setup MySQL as a standard Development computer on port 3306 over TCP/IP.

I enforced strong passwords.

I enforced strong passwords.

I set a root password and added few app usernames (with passwords).

I set a root password and a few app usernames (with passwords)

I named the MySQL Instance and set it to auto start when windows starts as a standard system account.

I named the MySQL Instance and set it to auto start when windows starts as a standard account.

Post installation in progress.

Installation in progress.

I accepted the defaults for the next screen (configuring routers). I tested the connection to MySQL Server.

Connect to MySQL server test screen.

Installation complete.

Installation complete screen.

Advertisement:



MySQL WorkBench

I opened the MySQL Workbench software and viewed the server instance information. Nice.

MySQL Workbench Instance information

MySQL Workbench server performance graphs are nice.

MySQL Workbench performance graphs.

I am used to Adminer for managing MySQL on Linux so I install that now.

Install Adminer (formerly phpMinAdmin)

I prefer the PHP based Admirer database management tool from here.

https://adminer.net website screenshot.

I downloaded a single PHP file and placed it in my IIS website root folder (as /adminer.php).

I tried to connect to my MySQL instance but received this error “The server requested authentication method unknown to the client”.

Unable to connect to MySQL with Adminer: Error "he server requested authentication method unknown to the client"

I googled and checked that I had a MySQL extension in php.ini (It did).

I opened MySQL Workbench and opened Manage Server Connections and located my my.ini file location (“C:\ProgramData\MySQL\MySQL Server 8.0\my.ini“). I opened my my.ini in a text editor and commented out the following line

#default_authentication_plugin=caching_sha2_password

and added

default_authentication_plugin= mysql_native_password

I saved the my.ini file, stopped and started the MySQL Service.

MySQL Workbench restarting database service UI screenshot.

I opened the MySQL Workbench and ran a query (File then “New Query Tab“) to set/reset the password.

ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password
BY 'your_password_goes_here';

Advertisement:



Screenshot

Reset root password SQL statement "ALTER USER

I then tested logging in to MySQL with Adminer.

Adminer view showing default databases installed by MySQL.

Success, I can now created databases and tables.

Adminer create datbase table screenshot showing a table with 4 fields.

I hope this helps someone.

Setup a dedicated Debian subdomain (VM), Install MySQL 14 and connect to it from a WordPress on a different VM

by

This is how I set up a dedicated Debian subdomain (VM), Installed MySQL 14 and connected to it from a WordPress installation on a different VM

Advertisement:



Aside

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving fearby.com from Vultr to UpCloud.

Buy a domain name here

Domain names for just 88 cents!

Now on with the post.

Fearby.com

I will be honest, fearby.com is my play server where I can code, learn about InfoSec and share (It’s also my stroke rehab blog).

There is no faster way to learn than actually doing. The problem is my “doing” usually breaks the live site from time to time (sorry).

I really need to set up a testing environment (DEV-TEST-LIVE or GREEN-BLUE) server(s). GREEN-BLUE has advantages as I can always have a hot spare ready. All I need to do is toggle DNS and I can set the GREEN or BLUE server as the live server.

But first  I need to separate my database from my current fearby.com server and setup up a new web server. Having a Green and Blue server that uses one database server will help with near real-time production website switches.

Dedicated Database Server

I read the following ( Should MySQL and Web Server share the same server? ) at Percona Database Performance Blog. Having a separate database server should not negatively impact performance (It may even help improve speeds).

Deploy a Debian VM (not Ubuntu)

I decided to set up a Debian server instead of Ubuntu (mostly because of the good focus on stability and focus on security within Debian).

I logged into the UpCloud dashboard on my mobile phone and deployed a Debian server in 5 mins.  I will be using my existing how to setup Ubuntu on UpCloud guide (even though this is Debian).

TIP: Sign up to UpCloud using this link to get $25 free UpCloud VM credit.

Deploy Debian Sevrer

Deploy a Debian server setup steps:

  1. Login to UpCloud and go to Create server.
  2. Name your Server (use a fully qualified domain name)
  3. Add a description.
  4. Choose your data centre (Chicago for me)
  5. Choose the server specs (1x CPU, 50GB Disk, 2GB Memory, 2TB Traffic for me)
  6. Name the Primary disk
  7. Choose an operating system (Debian for me)
  8. Select an SSH Key
  9. Choose misc settings
  10. Click Deploy server

After 5 mins your server should be deployed.

Advertisement:



After Deploy

Setup DNS

Login to your DNS provider and create DNS records to the new IP’s (IPv4 and IPv6) provided by UpCloud. It took DNS 12 hours to replicate to my in Australia.

Add a DNS record with your domain registra A NAMe = IPV4 and AAAA Name = IPv6

Setup a Firewall (at UpCloud)

I would recommend you set up a firewall at UpCloud as soon as possible (don’t forget to add the recommended UpCloud DNS IP’s and any whitelisted IP’s your firewall).

Block everything and only allow

  • Port 22: Allow known IP(s) of your ISP or VPN.
  • Port 53: Allow known UpCloud DNS servers
  • Port 80 (ALL)
  • Port 443 (ALL)
  • Port 3306 Allow your WordPress site and known IP(s) of your ISP or VPN.

Read my post on setting up a whitelisted IP on an UpCloud VM… as it is a good idea.

UpCloud thankfully has a copy firewall feature that is very handy.

Copy Firewall rules option at UpCloud

After I set up the firewall I SSH’ed into my server (I use vSSH on OSX buy you could use PUTTY).

I updated the Debian system with the following  command

Get the MySQL Package

Visit http://repo.mysql.com/ and get the URL of the latest apt-config repo deb file (e.g “mysql-apt-config_0.8.9-1_all.deb”). Make a temp folder.

Download the MySQL deb Package

Install the package

Update the system again

Install MySQL on Debian

Secure MySQL

Advertisement:



You can secure the MySQL server deployment (set options as needed)

Install NGINX

I installed NGINX to allow Adminer MySQL GUI to be used

I ran these commands to install NGINX.

I edited my NGINX configuration as required.

  • Set a web server root
  • Set desired headers
  • Optimized NGINX (see past guides here, here and here)

I reloaded NGINX

Advertisement:




Install PHP

I followed this guide to install PHP on Debian.

Install PHP FPM

Increase Upload Limits

You may need to temporarily increase upload limits in NGINX and PHP before you can restore a WordPress database. My feabry.com blog is about 87MB.

Add “client_max_body_size 100M;” to “/etc/nginx/nginx.conf”

Add the following to “/etc/php/7.2/fpm/php.ini”

  • post_max_size = 100M
  • upload_max_filesize = 100M

Restore a backup of my MySQL database in MySQL

You can now use Adminer to restore your blog to MySQL. Read my post here on Adminer here. I used Adminer to move my WordPress site from CPanel to a self-managed server a year ago.

First login to your source server and export your desired database then login to the target server and import the database.

Firewall Check

Don’t forget to allow your WordPress site’s 2x Public IP’s and 1x Private IP to access port 3306 in your UpCloud Firewall.

How to check open ports on your current server

Set MySQL Permissions

Open MySQL

I ran these statements to grant the user logging in on the nominate IP’s access to MySQL.

Reload permissions in MySQL

Allow access to the Debian machine from known IP’s

Edit “/etc/host.allow”

Additions (known safe IP’s that need access to this MySQL remotely).

Tell MySQL to listen on

Edit “/etc/mysql/my.cnf”

Added..

I guess you could change the port to something random???

Restart MySQL

Install a second local firewall on Debian

Advertisement:



Install ufw

Do add the IP of your desired server or VPN to access SSH

Do add the IP of your desired server or VPN to access WWW

Now add the known IP’s (e.g any web servers public (IPv4/IPv6) or Private IP’s) that you wish to grant access to MySQL (e.g the source website that used to have MySQL)

Do add UpCloud DNS Servers to your firewall

Add all other rules as needed (if you stuff up and lock your self out you can login to the server with the Console on UpCloud)

Restart the ufw firewall

Prevent MySQL starting on the source server

Now we can shut down MySQL on the source server (leave it there just in case).

Edit “/etc/init/mysql.conf”

Comment out the line that contains “start on ” and save the file

and run

Reboot

Stop and Disable NGINX on the new DB server

We don’t need NGINX running now the database has been imported with Adminer.

Stop and prevent NGINX from starting up on startup.

Check to see if MySQL is Disabled

Yep

Test access to the database server in PHP code

Add to dbtest.php

<em>SELECT guid FROM wp_posts</em>()<br />
<ul><?php

//External IP (charged after quota hit)
//$servername = 'db.yourserver.com';

//Private IP (free)
//$servername = '10.x.x.x';

$username = 'username';
$password = '********your*password*********';
$dbname = 'database';

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

$sql = 'SELECT guid FROM wp_posts';
$result = $conn->query($sql);

if ($result->num_rows > 0) {
    // output data of each row
    while($row = $result->fetch_assoc()) {
        echo $row["guid"] . "<br>";
    }
} else {
    echo "0 results";
}
$conn->close();
?></ul>
Done

Check for open ports.

You can install nmap on another server and scan for open ports

Advertisement:



Install nmap

Scan a server for open ports with nmap

You should see this on a server that has access to see port 3306 (port 3306 should not be visible by non-whitelisted IP’s).  Port 3shouldoudl not be seen via everyone.

You should see something like this on a server that has access to see port 80/443 (a web server)

I’d recommend you use a service like https://pentest-tools.com/network-vulnerability-scanning/tcp-port-scanner-online-nmap# to check for open ports.  https://hackertarget.com/tcp-port-scan/ is a great tool too.

https://www.infobyip.com/tcpportchecker.php is also a free port checker that you can use to verify individual closed ports.

Screeshot of https://www.infobyip.com/tcpportchecker.php

Hardening MySQL and Debian

Read: https://www.debian.org/doc/manuals/securing-debian-howto/ap-checklist.en.html

Configuring WordPress use the dedicated Debian VM

On the source server that used to have MySQL edit your wp-config.php file for WordPress.

Remove

define('DB_HOST', 'localhost');

add (read the update below, I changed the DNS IP to the Private IP to have free traffic)

//Oriinal localhost
//define('DB_HOST', 'localhost');

//New external host via DNS (Charged after quota hit)
//define('DB_HOST', 'db.fearby.com');

//New external host via Private IP (Free)
define('DB_HOST','10.x.x.x');

Restart NGINX

Restart PHP-FPM

Conclusion

Nice, I seem to have shaved off 0.3 seconds in load times (25% improvement)

1sec gtmtrix load time

Update: Using a Private IP or Public IP between WordPress and MySQL servers

After I released this blog post (version 1.0 with no help from UpCloud) UpCloud contacted me and said the following.

I will have updated my references in this post and replace the public IP address (that is linked to DNS record for db.fearby.com) and instead use the private ip address (e.g 10.x.x.x), your servers private IP address is listed against the public IPv$ and IPv6 address.

I checked that the local ufw firewall did indeed allow the private IP access to MySQL.

On my new Debian MySQL server, I edited the file /etc/mysql/my.cnf and changed the IP to the private IP and not the public IP.

Now it looked like

(10.x.x.x  my Debian servers private IP)

On my WordPress instance, I edited the file  /www-root/wp-config.php

I added the new private host

//Oriinal localhost
//define('DB_HOST', 'localhost');

//New external host via DNS (Charged after quota hit)
//define('DB_HOST', 'db.fearby.com');

//New external host via Private IP (Free)
define('DB_HOST','10.x.x.x');

(10.x.x.x  my Debian servers private IP)

 

Alos on Debian/MySQL ensure you have granted access to the private IP of the WordPress server

Edit /etc/host.allow

Add

Restart MySQL

TIP: Enable UpCloud Backups

Do setup automatic backups (and or take manual backups). Backups are an extra charge but are essential IMHO.

UpCloud backups

Troubleshooting

If you can’t access MySQL log back into MySQL

and run

Reboot

Lower Upload Limits

Don’t forget to lower file upload sizes in NGINX and PHP (e.g 2M) now that the database has been restored.

 

I hope this guide helps someone.

TIP: Sign up to UpCloud using this link to get $25 free UpCloud VM credit.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Advertisement:



Revision History

v1.6 Changed Public IP use to private IP use to ensure we are not charged when the serves sage goes over the quota

v1.5 Fixed 03 type (should have been 0.3)

v1.4 added disable nginx info

v1.3 added https://www.infobyip.com/tcpportchecker.php

v1.1 added https://hackertarget.com/tcp-port-scan/

v1.0 Initial Post

How to backup WordPress on a host that has CPanel

by

This is a simple guide that demonstrates how you can log in to a host that offers the CPanel tools to backup all of your website files (and databases). Backing up your website should be done often and especially before you migrate to any another website host. I used to change hosts every few years (they don’t own your site, you do).

Advertisement:



I have a number of guides on moving away from CPanel, setting up VM’s on UpCloud, AWS, Vultr or Digital Ocean along with installing and managing WordPress from the command line copying files to a server via command line editing remote files locally etc but how do you manage a website with CPanel?

You can normal login to CPanel tools on a shared host by loading www.yourdomainnam.com/cpanel (failing that login to your domain hosts web GUI and find your CPanel interface there).

Step 1: Login to your Host

Login to your web host

Login to Host

Step 2: Find your CPanel Interface

Hosts are a bit different but in this case, I just click my domain to find the CPanel link.

Domain

I found it, I clicked the CPanel login.

Cpanel link

Step 3: CPanel Applications

CPanel does offer good tools to manage your websites like web-based File Manager and Database tool called phpMyAdmin.

CPanel Options

Aside: CPanel/Hosts Downsides

The thing I don’t like about hosts that offer CPanel is they usually limit delivery of your website to extract more money. Nothing worse than receiving Resource Limit Is Reached errors.

cpenal_usage_exceeded

Also shared hosts usually lag way behind in newer software versions like PHP and MySQL (this is a security concern).

TIP: You can scan your site for vulnerabilities using Qualsys Freescan, Zap or Kali Linux.

Here is a security scan of a shared host (with CPanel) that I was using in 1999. Note the high vulnerabilities and old version of Linux.

Static Website

Also, a shared host will often overcharge you (e.g $150 a year) for a poorly configured SSL certificate.

This was an SSL cert I paid $150 a year for (evaluated with SSL Labs SSL Test) on a shared host with CPanel.

Aside: Self Managed Upsides

After I moved my domain to a self-managed virtual machine I migrated WordPress, set up a free SSL certificate, sped up my site with a CDN, setup Cloudflare, setup better TLS security etc

When you manage your own server you can install a free SSL certificate in under 1 minute.

Below is my SSL certificate. A strong SSL certificate will increase search engine traffic

SSL Cert

Aside: Compare Shared host speed v Self Managed

FYI: https://gtmetrix.com/ is a great site for measuring the speed of a website (shared of self-managed). I found great speed improvements after moving away from a host offering CPanel, tweaking the server and setting up cloudflare. A self-managed server will allow you to tweak anything you want.

GTMetrix results:

Shared Host Speed

I like how self-managed servers allow you to scale the server’s resources yourself, move servers or add storage etc.

Aside: SSL Certificate

If you have an SSL cert you should test it often as vulnerabilities pop up from time to time.

FYI: All sites will soon require an SSL certificate to be sent traffic from search engines (no SSL = lower traffic).

SSL Test my site: https://dev.ssllabs.com/ssltest/analyze.html?d=fearby.com&s=104.27.154.69

Now enough with the self-managed serve asides and back to how to backup your website with CPanel tools.

Step 4: Backup your web files in CPanel

Use the File Explorer app in CPanel

File Manager

Highlight all files that you want to backup (highlight everything but not past backup files).

backup-cpanel-006

View the files to compress summary

Compress Summary

Click Compress Files(s) and view the backup progress

Backup Progress

You can now download the backup zip file in your browser (click the file and click Download).

Downaload ZIp

Download Progress.

Download File

Step 5: Backup your database in CPanel

Now we need to backup any MySQL database(s) that may be used by WordPress

Open the phpMyAdmin app in CPanel.

phpMyAdmin

FYI: Alternatively, you can use a free tool called Adminer to backup and restore our database.

Click your WordPress database (on the left). You can identify your current WordPress database by opening the wp-config.php file.

Database Options

The first step is to perform an online cold backup of the WordPress database.

  1. Click the database you want to backup (on the left)
  2. Click the Options tab
  3. Under the Copy To frame enter a new database name to copy the database too
  4. Click Go

Online Backup

Now you have an online cold spare that you can use just in case the original database corrupts itself. You can rename the database or configure WordPress to point to this new database if need be.

backup-cpanel-013

Now let’s download a copy of the database (Repeat for multiple databases).

  1. Click the database you want to backup
  2. Click the Export tab
  3. Click Go

backup-cpanel-014

You should now have a backup of your website in a zip file and an export of your database in a .sql text file, SQL files can be re-imported to databases later.

TIP: Backup often.

Export Done

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.0 Initial post

Using the free Adminer GUI for MySQL on your website

by

Adminer is a free GUI tool that can you can easily install on a PHP web server. Adminer allows you to easily connect to your MySQL instance, create databases/tables/indexes/rows and backup/import databases and much more.

Advertisement:


You can read my other posts on Useful Linux Terminal Commands and Useful OSX Terminal Commands.

I used to use phpMyAdmin to manage MySQL databases on AWS, Digital Ocean and Vultr but switched to Adminer due to forgotten issues.  You can always manage MySQL via command line but that is quite boring.

The below screenshots were taken on my local Development Mac Laptop (with optional OSX Apache SSL Setup (that reports “Not Secure” (but it is good enough to use locally)). I prefer to code in SSL and warn when SSL is not detected.

Downloading and Installing Adminer

Navigate to https://www.adminer.org/ and click Download.

Adminer GUIClick English only (.php file)

Adminer

Save the Adminder for MySQL (.php) file to your web server and give it a random name and put in a folder also with a random name (I use https://www.grc.com/passwords.htm to generate strong password).

Tip: Uploading this file to a live serve offers hackers and unauthorized people potential access to your MySQL server.  I would remove this file from live serves when you are not using it not to be sure.

Tip: Read my guide here on setting up NGINX, MySQL and PHP here.  Basically I did this to setup MySQL on Ubuntu 16.04.

TIP: Ensure MySQL is secure and has a good root password, also consider setting up Ubuntu Firewalls and Securing Ubuntu. Also ensure the Server is patched and does not have exploits like Spectre and meltdown.

Now you can access your Admirer php file on your Web Server (hopefully with an obfiscated name).

Randomize

Login to Adminer with your MySQL root password.

Login

Click Create databaase

Create Database

Give the database a name and choose the character coding standard (e.g UTF8 ceneral ci). Different standards have different performance impacts too.

Save

Now that you have a database you can create a table.

Adminer

Consider adding an auto-incrementing ID and say a Key and Value varchar column.

Adminer

When the table is created you can add a row to the table.

Adminer

I created one with a “TestKey” and “TestValue” row.

Adminer

The row was inserted.

Adminer

The final thing to do is add a database user that code can connect to the database with. Click Privileges.

Adminer

Click Create user

Adminer

Tick All privileges and click Save

Adminer

Now the user is added to the database

Adminer

Let’s create a PHP file and talk to the database. Let’s use parameterized queries

<?php

date_default_timezone_set('Australia/Sydney');
echo "Last modified: " . date ("F d Y H:i:s.", getlastmod()) . "<br /><br />";

// Turn on if you need to see errors
// error_reporting(E_ALL);
// ini_set('display_errors', 0);

$dbhost = '127.0.0.1';
$dbname = 'dbtest';
$dbusername = 'dbtestuser';
$dbpassword = '*****************************************'';

$con = mysqli_connect($dbhost, $dbusername, $dbpassword, $dbname);
 
// Turn on debug stuff if you need it
// echo var_dump($con);
// printf(" - Error: %s.n", $stmt->error);
 
if($con->connect_errno > 0){

    printf(" - Error: %s.n", $stmt->error);
    die("Error: Unable to connect to MySQL");

} else {

    echo "Charset set to utf8<br />";
    mysqli_set_charset($con,"utf8");
}
 
if (!$con) {

    echo "Error: Unable to connect to MySQL (E002)" . PHP_EOL;
    echo "Debugging errno: " . mysqli_connect_errno() . PHP_EOL;
    echo "Debugging error: " . mysqli_connect_error() . PHP_EOL;
    exit;

} else {

    echo "Database Connection OK<br />";
 
    echo "&nbsp; Success: A proper connection to MySQL was made! The $dbname database is great." . PHP_EOL . "<br />";
    echo "&nbsp; &nbsp;- Host information: " . mysqli_get_host_info($con) . PHP_EOL . "<br />";
    echo "&nbsp; &nbsp;- Server Info: '" . mysqli_get_server_info($con) . "'<br />";
    echo "&nbsp; &nbsp;- Server Protocol Info : ". mysqli_get_proto_info($con) . "<br />";
    echo "&nbsp; &nbsp;- Server Version: " . mysqli_get_server_version($con) . "<br />";
    //echo " - Server Connection Stats: " . print_r(vmysqli_get_connection_stats($con)) . "<br />";
    echo "&nbsp; &nbsp;- Client Version: " . mysqli_get_client_version($con) . "<br />";
    echo "&nbsp; &nbsp;- Client Info: '" . mysqli_get_client_info() . "'<br />";
 
    echo "Ready to Query the database '$dbname'.<br />";
 
    // Input Var's that are parameterized/bound into the query statement
    $in_key = mysqli_real_escape_string($con, 'TestKey');
 
    // Output Var's that the query fills after querying the database
    // These variables will be filled with data from the current returned row
    $out_id = "";
    $out_key = "";
    $out_value = "";
 
    echo "1. About to query the database: '$dbname'<br />";
    $stmt = mysqli_stmt_init($con);

    $sql = "SELECT testid, testkey, testvalue FROM tbtest WHERE testkey = ?";
    echo "SQL: $sql (In = $in_key)<br /";

    if (mysqli_stmt_prepare($stmt, $sql)) {

            echo "2. Query Returned<br />";
            /*
                Type specification chars
                Character   Description
                i   corresponding variable has type integer
                d   corresponding variable has type double
                s   corresponding variable has type string
                b   corresponding variable is a blob and will be sent in packets
            */
            mysqli_stmt_bind_param($stmt, 's', $in_key);
            mysqli_stmt_execute($stmt);
            mysqli_stmt_bind_result($stmt, $out_id, $out_key, $out_value);
            mysqli_stmt_fetch($stmt);
     
            // Do something with the 1st returned row        
            echo " - Row: ID: $out_id, KEY: $out_key, VAL: $out_value <br />";//

            // Do we have more rows to process
            while($stmt->fetch()) { 
                
                    // Output returned values
                    echo " - Row: ID: $out_id, KEY: $out_key, VAL: $out_value <br />";//
            
            }
            mysqli_stmt_close($stmt);
            
            echo "Done<br />";
        
        } else {
        
            echo "3. Error Querying<br/>";
            printf(" - Error: %s.n", $stmt->error);
        
        }
}    
?>

Result

Adminer Results

If you don’t have a server check out my guides on AWS, Digital Ocean and Vultr.

Happy coding and I hope this helps someone.

Donate and make this blog better

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.0 Initial Version

PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database

by

Sanitising user input is a golden rule with web developing (see https://www.owasp.org/index.php/Input_Validation_Cheat_Sheet ), here is my code snippet to sanitise and parameterise MySQL queries in PHP 7.0.

Advertisement:



First, watch and follow @jawache (Asim Hussein) demo how common hacks happen and why you should update and patch software often, sanitise user data and set up a firewall.

I have blogged before on how to set up a Vultr server and configure it, How to secure Ubuntu in the Cloud, How to run an Ubuntu System Audit and Beyond SSL with Content Security Policy, Public Key Pinning etc but a 100% secure server is impossible as  zero-day exploits and flaws (e.g WPA WiFi) reminds us how limited technology lifespans can be. Yes, you can setup firewalls on Ubuntu and WordPress but you are only one exploit away from being hacked.Below is my code snippet (in PHP) to sanitise incoming data, query a MySQL database with object-oriented calls in PHP 7.0 and return data in variables. I have set up a firewall to block access to MySQL and non-essential ports (use https://www.shodan.io/ to test your server’s ports). I was using older deprecated PHP 5 era database calls and I researched newer calls available in PHP 7.0.

When you log in to an Ubuntu server and it says the following you should update

Also update software, node, npm etc

This code outputs too much information but will help you setup and get data on your servers (as long as you replace your database, table and field names).

<?php
echo "Last modified: " . date ("F d Y H:i:s.", getlastmod()) . "<br /><br />";
date_default_timezone_set('Australia/Sydney');

$dbhost = '127.0.0.1';
$dbusername = 'themysqlaccount';
$dbpassword = 'themysqlpassword';
$dbname = 'thedatabasename';
$con = mysqli_connect($dbhost, $dbusername, $dbpassword, $dbname);

//Debug stuff
//echo var_dump($con);
//printf(" - Error: %s.\n", $stmt->error);

if($con->connect_errno > 0){
    printf(" - Error: %s.\n", $stmt->error);
    die("Error: Unable to connect to MySQL (E001)");
} else {
 echo "Charset set to utf8<br />";
 mysqli_set_charset($con,"utf8");
}

if (!$con) {
    echo "Error: Unable to connect to MySQL (E002)" . PHP_EOL;
    echo "Debugging errno: " . mysqli_connect_errno() . PHP_EOL;
    echo "Debugging error: " . mysqli_connect_error() . PHP_EOL;
    exit;
} else {
    echo "Database Connection OK<br />";

    echo "&nbsp; Success: A proper connection to MySQL was made! The my_db database is great." . PHP_EOL . "<br />";
    echo "&nbsp; &nbsp;- Host information: " . mysqli_get_host_info($con) . PHP_EOL . "<br />";
    echo "&nbsp; &nbsp;- Server Info: '" . mysqli_get_server_info($con) . "'<br />";
    echo "&nbsp; &nbsp;- Server Protocol Info : ". mysqli_get_proto_info($con) . "<br />";
    echo "&nbsp; &nbsp;- Server Version: " . mysqli_get_server_version($con) . "<br />";
    //echo " - Server Connection Stats: " . print_r(mysqli_get_connection_stats($con)) . "<br />";
    echo "&nbsp; &nbsp;- Client Version: " . mysqli_get_client_version($con) . "<br />";
    echo "&nbsp; &nbsp;- Client Info: '" . mysqli_get_client_info() . "'<br />";

echo "Ready to Query the database '$dbname'.<br />";

// Input Var's that are parameterized/bound into the query statement
// I pre fill three vaiables with known fields in my users table
// Goot article in manual sanitization of strings in PHP http://php.net/manual/en/filter.filters.sanitize.php
    $in_username = mysqli_real_escape_string($con, 'FearTec');
    $in_f_guid = mysqli_real_escape_string($con, '5161a571-4a51-468d-9e96-6a5db5d35b1c');
    $in_mobile = mysqli_real_escape_string($con,'0456629533');

// Output Var's that the query fills after querying the database
// These variables will be filled with data from the current returned row
    $out_id = "";
    $out_f_guid = "";
    $out_username = "";
    $out_mobile = "";

echo "1. About to query the database: '$dbname'<br />";
$stmt = mysqli_stmt_init($con);
if (mysqli_stmt_prepare($stmt,"SELECT id, username, guid, user_mobile FROM thedatabasename WHERE username = ? AND guid = ? AND user_mobile = ?")) {

        echo "2. Query Returned<br />";

        /*
            Type specification chars
            Character   Description
            i   corresponding variable has type integer
            d   corresponding variable has type double
            s   corresponding variable has type string
            b   corresponding variable is a blob and will be sent in packets
        */
        mysqli_stmt_bind_param($stmt, 'sss', $in_username, $in_guid, $in_mobile);

        mysqli_stmt_execute($stmt);

        mysqli_stmt_bind_result($stmt, $out_id, $out_username, $out_guid, $out_mobile);

        mysqli_stmt_fetch($stmt);

        // Do something with the 1st returned row        
        echo " - Row: ID: $out_id, GUID: $out_guid, USR: $out_username, MOB: $out_mobile";//
        // Do we have more rows to process
        while($stmt->fetch()) { 
                // Deal with other rows
                echo " - Row: ID: $out_id, GUID: $out_f_guid, USR: $out_username, MOB: $out_mobile<br />";
        }
        mysqli_stmt_close($stmt);
        
        echo "c<br />";
    }
     else {
        echo "3. Error Querying<br/>";
        printf(" - Error: %s.\n", $stmt->error);
    }

}    

?>

Returned Data

Last modified: November 01 2017 16:43:01.
Charset set to utf8
Database Connection OK
  Success: A proper connection to MySQL was made! The my_db database is great. 
   - Host information: 127.0.0.1 via TCP/IP 
   - Server Info: '5.7.19-0ubuntu0.16.04.1'
   - Server Protocol Info : 10
   - Server Version: 50719
   - Client Version: 50012
   - Client Info: 'mysqlnd 5.0.12-dev - 20150407 - $Id: b5######################## $'
Ready to Query the database 'thedatabasename'.
1. About to query the database: 'thedatabasename'
2. Query Returned
- Row: ID: 1, GUID: 0000000-0000-0000-0000-000000000001, USR: Bob, MOB: 1234567890
- Row: ID: 2, GUID: 0000000-0000-0000-0000-000000000002, USR: Joe, MOB: 1234567891
- Row: ID: 3, GUID: 0000000-0000-0000-0000-000000000003, USR: Jane, MOB: 1234567892

Variable Bind Parameter Types

When you bind an incoming variable you can inform MySQL what the data type is expected to be.

mysqli_stmt_bind_param: Type specification chars
Character   Description
i   corresponding variable has type integer
d   corresponding variable has type double
s   corresponding variable has type string
b   corresponding variable is a blob and will be sent in packets

Debug Options

Errors Enabled: Turn on PHP Debug Errors On

Turning on errors on production servers is bad but on on development.

First, find php.ini files

Edit your appropriate PHP file

And turn on Error reporting.

Restart PHP and NGINX

If you need to view your active php.ini file or see php configuration settings, add this to a .php file on your web server and view it’s contents.

<?php
phpinfo()
?>

It is amazing how clear errors can be

PHP Error

Dump Connection Vars: PHP mysqli_connect: var_dump($con)

echo var_dump($con);

Output:

  public 'affected_rows' => int 0
  public 'client_info' => string 'mysqlnd 5.0.12-dev - 20150407 - $Id: b5###############################3 $' (length=79)
  public 'client_version' => int 50012
  public 'connect_errno' => int 0
  public 'connect_error' => null
  public 'errno' => int 0
  public 'error' => string '' (length=0)
  public 'error_list' => 
    array (size=0)
      empty
  public 'field_count' => int 0
  public 'host_info' => string '127.0.0.1 via TCP/IP' (length=20)
  public 'info' => null
  public 'insert_id' => int 0
  public 'server_info' => string '5.7.19-0ubuntu0.16.04.1' (length=23)
  public 'server_version' => int 50719
  public 'stat' => string 'Uptime: 1828089  Threads: 1  Questions: 15702  Slow queries: 0  Opens: 1529  Flush tables: 1  Open tables: 1461  Queries per second avg: 0.008' (length=142)
  public 'sqlstate' => string '00000' (length=5)
  public 'protocol_version' => int 10
  public 'thread_id' => int 7367
  public 'warning_count' => int 0

Show Environment Vars: mysqli_get_host_info, mysqli_get_proto_info, mysqli_get_server_version, mysqli_get_client_version and mysqli_get_client_info.

echo "&nbsp; Success: Database connection OK." . PHP_EOL . "<br />";
echo "&nbsp; &nbsp;- Host information: " . mysqli_get_host_info($con) . PHP_EOL . "<br />";
echo "&nbsp; &nbsp;- Server Info: '" . mysqli_get_server_info($con) . "'<br />";
echo "&nbsp; &nbsp;- Server Protocol Info : ". mysqli_get_proto_info($con) . "<br />";
echo "&nbsp; &nbsp;- Server Version: " . mysqli_get_server_version($con) . "<br />";
//echo " - Server Connection Stats: " . print_r(mysqli_get_connection_stats($con)) . "<br />";
echo "&nbsp; &nbsp;- Client Version: " . mysqli_get_client_version($con) . "<br />";
echo "&nbsp; &nbsp;- Client Info: '" . mysqli_get_client_info() . "'<br />";

Output:

  Success: Database connection OK. 
   - Host information: 127.0.0.1 via TCP/IP 
   - Server Info: '5.7.19-0ubuntu0.16.04.1'
   - Server Protocol Info : 10
   - Server Version: 50719
   - Client Version: 50012
   - Client Info: 'mysqlnd 5.0.12-dev - 20150407 - $Id: b5######################################## $'

Show errors in failed if  statements: mysqli_stmt_prepare else

printf(" - Error: %s.\n", $stmt->error);

Output:

Error: Table 'thedatabasename.invalidtablename' doesn't exist.

Debugging is your friend.

 

More to come..

 

Donate and make this blog better


Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

V 1.0 initial post

Short (Article):

NodeJS code to handle App logins via API (using MySQL connection pools (1000 connections) and query parameters)

by

The below code is part of what I use to handle logins in an app. You can use this as a base for a NodeJS login routine (in a file called app.js). I was going to use StormPath API but was too slow and unreliable for  (as it turns out StormPath are merging with Okta and are closing down services).

Advertisement:



This code is not functionally complete but you can hack out what you need. Rolling your own  solution can be faster and slow you to fully embed in server side logic but make sure it is secure. 
I am using these node modules (app.js).

var app = require('express')();
var http = require('http').Server(app);
var mysql = require('mysql');
var uuid = require('node-uuid');
var Type = require('type-of-is');
var jsesc = require('jsesc');
var bcrypt = require('bcrypt');
var filessystem = require('fs');

The following code is located in myutils.js is a  handy place to place your common code.  You don’t need it but it is demonstrated here.

module.exports = function(){ 
this.sum = function(a,b){
    return a+b
};
this.multiply = function(a,b){
    return a*b
    
  };
this.samplefunction = function(a){
    return 'Hello World'
    
  };
}

Don’t forget to include it myutils.js in app.js

require('./myutils.js')();  //Remove this if you dont use it

Declare the body parser module (app.js).

app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

Set the MySQL connection details and you can setup the number of ready MySQL connections in the pool here (app.js).

console.log(' mysql_pool.createPool()');
var mysql_pool  = mysql.createPool({
  connectionLimit : 1000,
  host            : 'localhost',
  database        : 'thedatabasename',
  user            : 'thedatabaseuser',
  password        : 'thedatabasepassword'
  
});

Create a static API endpoint (app.js).

// Generic static API Point
app.get('/api/myappname/v1/hello/world/',function(req,res){
	var data = { "Version":"" };
	data["Version"] = "Hello World"; // Add a JSON value
	res.json(data); // Return the JSON
});

Create MySQL functions (app.js).

mysql_pool.getConnection(function(err, connection){
	// See if a connection is available in the pool by querying the database
	console.log(' mysql_pool.getConnection()');
	  	if (err) {
	  		console.log(' error: ' + err);
	  		throw err;
	  	}
	connection.query('select * from thedatabasename;',  function(err2, rows){
	  	if (err2) {
	  		connection.release();
	  		console.log(' error: ' + err2);
	  		throw err2;
	  	} else {
	  		console.log( rows );
	  		console.log('Ready');
	  	}
	  	connection.release();
	});
	console.log(' mysql_pool.release()');
});

mysql_pool.on('connection', function (connection) {
  connection.query('SET SESSION auto_increment_increment=1')
  console.log(' mysql_pool.on(connection) - Set session increment');
});

mysql_pool.on('enqueue', function () {
  console.log(' mysql_pool.on(enqueue). Waiting for available connection slot.');
});

/api/myappname/v1/login function (app.js).

// Login ----------------------------------------------------------------------- 
app.post('/api/myappname/v1/login',function(req,res){
	console.log('API CALL: /api/myappname/v1/login');

	var errData = "";

	// Check to see if the users posted token is valid
	var useraccesstoken = req.headers['x-user-access-token'];
	if (appaccesstoken == undefined) {
		// If a user token is missng then block the login, we only process logins whre valid user tokens are posted. Users are generated a token at acount creation
		console.log(" 498 token required");
		errData = {"Status": "498", "Error": "Invalid user token. Please login again (error 498)."};
		res.statusCode = 499;
		res.send(errData);
		res.end();
	}

	// I usually send an x-access-token in the posted JSON payload to eliminate older apps.
	var appaccesstoken = req.headers['x-access-token'];
	if (appaccesstoken == undefined) {
		// If an app  token is missng then block the login, we only process logins whre valid app tokens are posted.
		console.log(" 499 token required");
		errData = {"Status": "499", "Error": "Invalid app token. Please update your app (error 499)."};
		res.statusCode = 499;
		res.send(errData);
		res.end();

	} else {
		// ok we have a token posted
		var appaccesstokenpassed = false;

		// Check the API Key against known API Keys
		if (appaccesstoken == "appname33ffda6d-4303-4c65-81ca-0ccf0f172a37") { appaccesstokenpassed = true;} // Debug API Key for ICS 0.9.1
		
		// OK Process the posted data
		if (appaccesstokenpassed == true) {
			
			var userenteredemail = req.body.email;
			var userenteredpass = req.body.password; //todo: Add Encryption/Decryption (currenty a BCrypt Hash is sent from the app)

			// If this is Yes a new session token is sent to the user later, this will in valitae older logged in sessions.
			var resetsessions = req.body.resetsessions;
			if (req.body.resetsessions == undefined) {
				resetsessions = "Yes";
			}
			
			// Log for debugging
			console.log(' Email: ' + userenteredemail);
			console.log(' Password: ' + pass);
			console.log(' Reset Sessions: ' + resetsessions);
			
			var data = {}
			
			mysql_pool.getConnection(function(err, connection) {
			if (err) {
				console.log(' mysql_pool.release()');
				connection.release();
		  		console.log(' Error getting mysql_pool connection: ' + err);
		  		throw err;
		  	}
		  		// Revised SQL
		  		connection.query("SELECT * FROM usertablename WHERE usertablename.active=1 AND usertablename.lockedout=0 AND usertablename.email=? LIMIT 1",[userenteredemail],function(err, rows, fields){
				
					if (err) {

						// Log database errors
						console.log(" 503 Service unavailable (database error)");
						errData = {"Status": "503", "Error": "Service unavailable (database error)"};
						res.statusCode = 503;
						res.send(errData);	
						res.end();

					} else {
						
						// Do we have  a recordset
						if (rows.length != 0) {
							
							// Check password hash - https://github.com/ncb000gt/node.bcrypt.js
							var salt = rows[0].password_salt;
							var hash = rows[0].password_hash; 		// bcrypt.hashSync(pass, salt);
							
							hash = hash.replace(/^\$2y(.+)$/i, '\$2a$1');
							var passwordok = false;
							console.log(' About to check password.');
							
							passwordok = bcrypt.compareSync(userenteredpass, hash);

							console.log(passwordok);
							if (passwordok == true) {
							
							// Allow access to records
								var userid = rows[0].id;									// User ID
								var user_guid = rows[0].user_guid;							// Internal App Guid
								var push_guid = rows[0].push_guid;							// Internal GUID user for Push Notifications
								
								
								var newkey = rows[0].usertokenkey;								// Generate a new API secret key
								var newpass = rows[0].usertokenpassword;								// Generate a a new API secret pass

								// This will create two guids to use for all future api hiots from the user
								if (resetsessions == "Yes") {
									newkey = uuid.v4();										// Generate a new API secret key
									newpass = uuid.v4();									// Generate a a new API secret pass
								}

								var stat_total_logins = rows[0].stat_total_logins;			// Read total Logins are stored in my usedatabase
									stat_total_logins = stat_total_logins + 1;				// Increment total logins for later saving 
								
								var username = rows[0].username;							// Read the user Username from the database
								var usertag = rows[0].usertagline;							// Read the user Tagling from the database
								var email = rows[0].email;									// Read the user Email from the database
								
								var block_transactions = rows[0].block_transactions;		// Has the User Blocked Transactions
								var credit = rows[0].credit;								// Credit Remaining
								var currency = rows[0].currency;							// Users Currency (used for the log)

								var smssendalertatlogin = rows[0].user_pref_sms_alert_at_login;  	// Does the user want SMS Sent a Login
								var emailsendalertatlogin = rows[0].user_pref_email_alert_at_login; 	// Does the user want an email
								var pushsendalertatlogin = rows[0].user_pref_push_alert_at_login;	// Push alert the user at login
								
								var user_pref_timezone = rows[0].user_pref_timezone;
								var user_pref_timezone_int = rows[0].user_pref_timezone_int;
								
								var stat_total_password_resets = rows[0].stat_total_password_resets;		// Pass this throgh to the App
								
								var active = rows[0].active;								// Is the user active
								var lockedout = rows[0].lockedout;							// Is the user locked ut
								var changing_password = rows[0].changing_password;

								var querylimit5min = rows[0].QueryLimit5Min;				// Read from he databse how manu API hits the user has a minute
								
								// I like to return the api version and version int with all api JSON payloads (makes checking on the App side easier)
								var jsonvalid = "yes";
								var jsonversion = "2017.01.01.22.00";
								var jsonversionint = "102";
								
								//Update API Keys
								mysql_pool.getConnection(function(err, connection) {
									if (err) {
										console.log(' mysql_pool.release()');
										connection.release();
								  		console.log(' Error getting mysql_pool connection: ' + err);
								  		throw err;
								  	}
								  	// Update the users logins and new keys if thye asked to wipe all past logins.
									connection.query('UPDATE usertablename SET usertokenkey = ?, usertokenpassword = ?, stat_total_logins = ?, last_login = NOW() WHERE id = ?', [newkey, newpass, stat_total_logins, userid],function(errupdate, rowsupdate, fieldsupdate){
										if (errupdate) {
											console.log(" 503 service unavailable (database update error)");
											errData = {"Status": "503", "Error": "503 service unavailable (database update error)"};
											res.statusCode = 503;
											res.send(errData);
											res.end();
										} else {
											if (rowsupdate.length != 0) {
												// Inform User of Valid  Login
												res.statusCode = 200;
												data["Data"] = "Successfully logged in..";									// statis string thta reports the successfull login
												
												// The Key and Pass are used liek OAUTH tokens and are stored in the app and user database. All API hits check these tokens. 
												data["Key"] = newkey;                       								//LOADS IN XCODE v0.9.1
												data["Pass"] = newpass;                      								//LOADS IN XCODE v0.9.1
												
												// User guids are used to store the unique user rather than the database record ID
												data["user_guid"] = user_guid;
												data["push_guid"] = push_guid;												// Internal GUID user for Push Notifications

												data["Logins"] = stat_total_logins;											// Total Logins are stored in my usedatabase
												data["Username"] = username;												// Username
												data["Email"] = email;														// Email
												
												// The notify user code ay ogin has been removed byt can be called here
												data["SMSAtLogin"] = smssendalertatlogin;
												data["EmailAtLogin"] = emailsendalertatlogin;
												data["PushAtLogin"] = pushsendalertatlogin;
												
												// What is the users timezome
												data["timezone"] = user_pref_timezone;
												data["timezoneint"] = user_pref_timezone_int;
												
												
												data["stat_total_password_resets"] = stat_total_password_resets;			// How many times has the user changed passwords
												
												data["active"] = active;													// Is the user active
												data["lockedout"] = lockedout;												// Is th user currently locked out
												data["changing_password"] = changing_password;								// Is the user currebtky changing a password
												
												data["levelquerylimit5min"] = querylimit5min;								// How any times can the user hit the API (handled by the app)
	
												data["jsonvalid"] = jsonvalid;												// JSON ID
												data["jsonversion"] = jsonversion;											// JSON Version
												data["jsonversionint"] = jsonversionint;									// JSON Version
												
												// Send the JSON back to the user
												res.json(data);
												res.end();
												
												//Login Confirmed
												console.log("User " + username + " / " + email + " logged in.");
												
											} else {
												console.log(" 504 service unavailable (database update error)");
												errData = {"Status": "504", "Error": "504 service unavailable (database update error)"};
												res.statusCode = 504;
												res.send(errData);
												res.end();		
											}
										}
										console.log(' mysql_pool.release()');
										connection.release();				// release the MySQL connection pool connection
									});	
								});
								
							} else {
								console.log('402 Invalid Login (' +  req.body.email + ', ' + req.body.password + ')');
								errData = {"Status": "402", "Error": "Invalid Login"};
								res.statusCode = 402;
								res.send(errData);
								res.end();								
							}
	
						} else {
							console.log('403 Account is locked out, not active or unknown. Please visit https://www.myappname.com to reactivate your account. (' +  req.body.email + ')');
							errData = {"Status": "403", "Error": "Account is locked out, not active or unknown."};
							res.statusCode = 403;
							res.send(errData);
							res.end();	
						}
					}
					console.log(' mysql_pool.release()');
					connection.release();
					
				});
			});

		} else {
			console.log("499 token required");
			errData = {"Status": "499", "Error": "Invalid app token. Please update your app (error 499)."};
			res.statusCode = 409;
			res.send(errData);	
			res.end();
		}
	}
});

 

/api/myappname/v1/verifylogin verify login function (app.js).

// Verify Login ----------------------------------------------------------------------- 
app.post('/api/myappname/v1/verifylogin',function(req,res){
	console.log('API CALL: /api/myappname/v1/verifylogin');
	
	// check to see if the App included an api key
	var appaccesstoken = req.headers['x-access-token'];

	var errData = "";
	if (appaccesstoken == undefined) {
		console.log(" 499 token required");
		errData = {"Status": "499", "Error": "valid token required"};
		res.statusCode = 499;
		res.send(errData);
		res.end();

	} else {

	//console.log(" Received appaccesstoken: " + appaccesstoken);
		var appaccesstokenpassed = false;
		// Check the API Key against known API Keys
		if (appaccesstoken == "ics201701-1d95-9271-a6c7-a9a33770287") { appaccesstokenpassed = true;} // Debug API Key
		if (appaccesstokenpassed == true) {
			var key = req.body.key;
			var pass = req.body.pass;
			var data = {} 

			console.log(" - Key: " + key )
			console.log(" - Pass: " + pass )
			
			// Grab the user IP and log it
			var ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress;
			console.log(" - IP: " + ip );
			
			mysql_pool.getConnection(function(err, connection) {
			if (err) {
				console.log(' mysql_pool.release()');
				connection.release();
		  		console.log(' Error getting mysql_pool connection: ' + err);
		  		throw err;
		  	}
		  	// Check the user record (only if the user is active and provided thier token key and passowrd)
		    connection.query("SELECT * FROM usertablename WHERE active=1 AND usertokenkey=? AND usertokenpassword=? LIMIT 1",[key, pass],function(err, rows, fields){
				if (err) {
					console.log(" 405 Login Session Expired");
					errData = {"Status": "405", "Error": "Login Session Expired"};
					res.statusCode = 405;
					res.send(errData);	
					res.end();
				} else {
					if (rows.length != 0) {
						console.log(" 200 Login Session OK");
						res.statusCode = 200;
						data["Status"] = "OK";
						res.json(data);
						res.end();
					} else {
						console.log(" 405 Login Session Expired");
						errData = {"Status": "405", "Error": "Login Session Expired"};
						res.statusCode = 405;
						res.send(errData);	
						res.end();
					}
				}
				console.log(' mysql_pool.release()');
				connection.release();
			});
			});
		} else {
			console.log("499 token required");
			errData = {"Status": "499", "Error": "valid token required"};
			res.statusCode = 409;
			res.send(errData);	
			res.end();
		}
	}
});

 

Start the web service (app.js).

http.listen(3000,function(){
	console.log("Connected & Listen to port 3000 at /api/myappname/v1/ ..");
	console.log(" Do..");
	
});

Hope this helps

 

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

V1.0 Initial Post

Setting up a Vultr VM and configuring it with Runcloud.io

by

I have setup a Vultur VM manually (guide here). I decided to setup a VM software stack with Runcloud to see if it is as secure as fast (and saves me time).

Advertisement:



Buying a Domain

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

I deployed an Ubuntu server in NY for $2.5 a month on Vultr (I chose NY over Sydney as the Sydney $2.5 servers were sold out)

I then signed up for Runcloud.

I opened up port 80, 443 and 34210 as recommended by runcloud.

I then connected to my Vultr server with Runcloud.

Then Runcloud asked that I run a  script on the server as root.

Tip: Don’t run the script on a different IP like I did.

It appears I accidentally ran the Runcloud install command on the wrong IP, what did it install/change? I looked to see if Runcloud offer an uninstallation command? nope.

Snip from runcloud.

No uninstall?

Time to check out RunCLoud IDE at https://manage.runcloud.io to see what it looks like.

View Server Configuration

I was able to start a  NGINX installation/web server within a few clicks (it installed to /etc/nginx–rc)

Runcloud.io Pros

  • Setup was quick.
  • Dashboard looks pretty

Runcloud.io Cons

  • My root access is no longer working (what happened)? I did notice that Fail2Ban was blocking loads of IP’s?

  • I can’t seem to edit website files with runcloud.io?
  • Limited by UI (I could create a database and database user but now set database user permissions or assign database users to databases (there is a guide but no GUI on adding a  user to a DB))
  • I have to trust that runcloud have set things up securely.
  • CPanel UI has more options that Runcloud IMHO.
  • Other free domain managers exist like https://www.phpservermonitor.org

Domain

I linked a domain to the IP, now I just need to wait before continuing.

End of Guide

I have been locked out of my runcloud managed domain, I will stick to manually setup servers.

Being locked out is good for security I guess 🙁

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 404 "Not Found"]

V1.4 added tags and fixed typos.

v0.31 Domain linked

How to setup pooled MySQL connections in Node JS that don’t disconnect

by

Using NodeJS/NGINX to power a custom API or website is quite simple. At some stage you will want to connect NodeJS to a MySQL database.

Advertisement:



Update (June 2018): I moved my domain to UpCloud (they are that awesome). Use this link to signup and get $25 free credit. Read the steps I took to move my domain to UpCloud here.

Upcloud Site Speed in GTMetrix

Buy a domain name from Namecheap here.

Domain names for just 88 cents!

Here is a quick sample of the minimum code needed to get a Node Server up with a API target of “/api/database/status”. What better way to know if a database is up than to read a setting from a table to confirm it is up.

var app = require('express')();
var http = require('http').Server(app);
var mysql = require('mysql');
var bodyParser = require("body-parser");

app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

var db_config = { host : 'localhost', user : 'your_db_username', password : 'your_db_password', database : 'your_databse_name',};
var connection = mysql.createConnection(db_config);

app.get('/api/database/status',function(req,res){
	var retvalSettingValue = "?";
    connection.query('SELECT SettingValue FROM your_status_table WHERE SettingKey =\'DatabaseStatus\'', function(err, rows, fields) {	
    	if (err) {
			var data = { "Time":"", "DatabaseStatus":"" };
			data["Time"] = (new Date()).getTime();
			data["DatabaseStatus"] = "Down";
			res.json(data); 
		} else {
			var dbretval = rows[0].SettingValue;
			if (dbretval == 1 ) {
				var data = { "Time":"", "DatabaseStatus":"" };
				data["Time"] = (new Date()).getTime();
				data["DatabaseStatus"] = "Up";
				res.json(data); 
			} else {
				var data = { "Time":"", "DatabaseStatus":"" };
				data["Time"] = (new Date()).getTime();
				data["DatabaseStatus"] = "Down";
				res.json(data); 
			}
		}
    });

http.listen(3000,function(){
	console.log("Connected & Listen to port 3000 at /api ..");
});

This should return the following JSON data.

{"Time":1440835363892,"DatabaseStatus":"Up"}

The problem for me with the code above was that MySQL kept disconnecting after a few days. Adjusting timeouts, setting up keep alive cron jobs failed to keep the connection up. I then tried introducing the following code solve the disconnected MySQL connection.

function handleDisconnect() {
	console.log('handleDisconnect()');
	connection.destroy();
	connection = mysql.createConnection(db_config);
	connection.connect(function(err) {
	    if(err) {
			console.log(' Error when connecting to db  (DBERR001):', err);
			setTimeout(handleDisconnect, 1000);
	    }
	});

}

I added the following before and connection.query in existing code.

	connection.connect(function(err) {
	if(err) {
		console.log('Connection is asleep (time to wake it up): ', err);
		setTimeout(handleDisconnect, 1000);
		handleDisconnect();
	}
	});

I ended up swapping over to using pooled MySQL connections and this code has been working for days. Pooled connections reserves a number of connections ready for use in the background. You simply ask for a connection and when you are done you release it. The pooled connection closes and refreshed the connection and puts it at the end of the queue.

var app = require('express')();
var http = require('http').Server(app);
var mysql = require('mysql');
var bodyParser = require("body-parser");

app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

var mysql_pool  = mysql.createPool({
  connectionLimit : 100,
  host            : 'localhost',
  user            : 'your_database_user',
  password        : 'your_database_password',
  database        : 'your_database'
});

app.get('/api/database/status',function(req,res) {
	console.log('API CALL: /api/database/status');
	var retvalSettingValue = "?";
	mysql_pool.getConnection(function(err, connection) {
		if (err) {
			connection.release();
	  		console.log(' Error getting mysql_pool connection: ' + err);
	  		throw err;
	  	}
	    connection.query('SELECT SettingValue FROM your_database_table WHERE SettingKey =\'DatabaseStatus\'', function(err2, rows, fields) {	
	    	if (err2) {
				var data = { "Time":"", "DatabaseStatus":"" };
				data["Time"] = (new Date()).getTime();
				data["DatabaseStatus"] = "Down";
				res.json(data); 
			} else {
				var dbretval = rows[0].SettingValue;
				if (dbretval == 1 ) {
					var data = { "Time":"", "DatabaseStatus":"" };
					data["Time"] = (new Date()).getTime();
					data["DatabaseStatus"] = "Up";
					res.json(data); 
				} else {
					var data = { "Time":"", "DatabaseStatus":"" };
					data["Time"] = (new Date()).getTime();
					data["DatabaseStatus"] = "Down";
					res.json(data); 
				}
			}
			console.log(' mysql_pool.release()');
			connection.release();
	    });
	});
});

http.listen(3000,function(){
	console.log("Connected & Listen to port 3000 at /api ..");
});

Throughput

To test the connections/throughput it’s time to fire up more than 100 connections using siege over 60 seconds. I am using HTTPS connections so his would slow down the throughput a fair bit (but I am sticking with HTTPS for customer security/privacy). Read my guide here on adding a A+ level SSL cert to a digital ocean VM.

#!/bin/bash
free -m

siege -b -c1 -t1M https://yourserver.com/api/database/status
cat /var/log/siege.log

free -m

Here are the results from a siege single client hitting the NodeJS/pooled connections for 60 seconds. I am able to get about 6,150 database reads in 60 seconds with no dropped connections.

[email protected]:/home/scripts# bash _Benchmark.sh n
             total       used       free     shared    buffers     cached
Mem:          2001       1784        217         53        186        980
-/+ buffers/cache:        617       1384
Swap:         4095          0       4095

** SIEGE 3.0.5
** Preparing 1 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:                   6150 hits
Availability:                 100.00 %
Elapsed time:                  59.69 secs
Data transferred:               0.26 MB
Response time:                  0.01 secs
Transaction rate:             103.03 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                    0.97
Successful transactions:        6150
Failed transactions:               0
Longest transaction:            0.05
Shortest transaction:           0.00
 
FILE: /var/log/siege.log
You can disable this annoying message by editing
the .siegerc file in your home directory; change
the directive 'show-logfile' to false.
      Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,    OKAY,   Failed
2015-08-29 04:18:35,   6150,      59.69,           0,       0.01,      103.03,        0.00,        0.97,    6150,       0
             total       used       free     shared    buffers     cached


Mem:          2001       1806        195         53        186        981
-/+ buffers/cache:        638       1363
Swap:         4095          0       4095

2 siege clients hitting the server delivers 8,120 database reads with 0 dropped connections.

      Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,    OKAY,   Failed
2015-08-29 04:25:13,   8120,      59.83,           0,       0.01,      135.72,        0.00,        1.96,    8120,       0

10 siege clients hitting the server delivers 16,383 database reads with 0 dropped connections.

      Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,    OKAY,   Failed
2015-08-29 04:29:50,  16383,      59.51,           0,       0.04,      275.30,        0.00,        9.86,   16383,       0

20 siege clients hitting the server delivers 15440 database reads with 0 dropped connections. This looks like the max the server can handle (257 delivered requests a second (19.7 concurrent connections)).

      Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,    OKAY,   Failed
2015-08-29 04:33:06,  15440,      59.93,           0,       0.08,      257.63,        0.00,       19.77,   15440,       0

My SQL, Linux Memory and Cache tweaks

I have tweaked my stock MySQL config a bit (changed values below)

#
# The MySQL database server configuration file.
#
...
[mysqld]
...
key_buffer		= 32M
key_buffer_size = 32M
max_allowed_packet	= 16M
max-connect-errors             = 1000000

#max_connections        = 100
max-connections = 500
thread_stack		= 256K
thread_cache_size       = 8
myisam-recover         = FORCE,BACKUP
#table_cache            = 64
#thread_concurrency     = 10
...
query_cache_limit	= 1M
query_cache_size        = 0
...
expire_logs_days	= 10
max_binlog_size         = 100M
...
# CACHES AND LIMITS #
tmp-table-size                 = 32M
max-heap-table-size            = 32M
thread-cache-size              = 50
open-files-limit               = 65535
# table-definition-cache         = 1024
# table-open-cache               = 2048
..

Advertisement:



Some may notice a 20mb memory jump in memory usage after I benchmarked, I have set my ubuntu to use as much memory as possible and not release too soon. I also have a 4GB swap file (that is ready in reserve but not used much).

  • swappiness this control is used to define how aggressively the kernel swaps out anonymous memory relative to pagecache and other caches. Increasing the value increases the amount of swapping. The default value is 60.
  • vfs_cache_pressure this variable controls the tendency of the kernel to reclaim the memory which is used for caching of VFS caches, versus pagecache and swap. Increasing this value increases the rate at which VFS caches are reclaimed.

/etc/sysctl.conf

vm.swappiness = 10
vm.vfs_cache_pressure = 50
vm.min_free_kbytes= 131072

Tips on setting linux cache usage and snappiness here.

More performance tweaks here: https://gist.github.com/dakull/5629740

Donate and make this blog better




Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 404 "Not Found"]