Backup

Finding software that needs updating on Windows using SUMo (Review)

May 30, 2020 by Simon

Recently I blogged about using Acronis True Image 2020 to backup and restore your Windows computer after a friend had to manually reinstall her Windows PC.

Her Windows PC died and she could not recover Windows with a System Restore snapshot or via the retailers systems image recovery process.

I use Back Blaze software to automatically backup my computer but I do not have an up to date collection of needed installation files in case I need to perform a Clean Installation of Windows 10.

I found SUMo in the process of assisting my friend.

SUMo to the Rescue

I found SUMo (Software Update Monitor) from KC Softwares that allows me to analyse my existing installed software and their versions, find and download new versions (installers or setup files) for my software.

I am using the Pro version of SUMo (buy here).

Tip: The trial version will inform you of an update but it will not show you links to obtain the updates easily.

SUMo Trial Limitations

The best feature of SUMo is I can find apps on my PC that need updating.

Also in one click (Right click) you will be directed to the applications official site (official site (or App page on SUMo)) and be able to download the app installer to the new version of the app.

Rigth click on an outdated app in SUMo to get the new aopo.

The Trial version is limited to informing you that an update is available.

SUMo telling your there is an update available

Also updating SUMo to the PRO version streamlines it’s own updates too. You can update SUMo in the PRO version with one click. 🙂

SUMo Pro Scan

After I opening SUMo PRO (I waited for it to find installed apps) then I clicked “Check” to scan all my apps for updates.

SUMo happily went through the list of my installed apps and gathered the applications versions (and checked online if newer versions existed).

SUMo found 239 installed applications on my PC, 188 were up to date, 53 needed minor (recent updates) and 2 needed major (old) updates.

Its awesome that in one click and I can see what software needs updating. In the process I was able to see what applications I don’t use (they were old versions) and I uninstalled them

SUMo Settings

I will update my software and to ensure my software does not get out of date I enabled “Load at startup” and “Check for new software at startup” (SUMo).

I did not enable “Scan for new software at startup” but seeing SUMo Start and update will remind me to check for updates if I have time.

I also enabled “Show file path” in the SUMo Application list so I can see where the programs are. I found multiple instances of Python installed all over my PC.

Getting Application Updates

I right clicked on each of my applications (that needed updating) and clicked “Get Update from…” then “.. from developer’s Site (Direct)“.

If you don’t want to right click you can click the “Get Update” button in the tool bar to get selected applications update.

This will open a web browser page to the software vendors website.

SUMo also has a page available with stats on the application’s known statistics (nice table of versions, beta or stable status and numbers of users).

Updating Software

I was able to right click “Get Update from…” – “.. from developer’s Site (Direct)” for each application in the list that needed an update. I was directed to the software vendors page for each software where I could download the app updates.

Within minutes I had dozens of vendor pages open with links to new versions of my outdated software.

Dozens of site pages with fresh app installs.

Within a short period I had amasses gigabytes of new application installs.

4.5GB of data of new application installs

I installed each of the new updates and then moved the new installs to my B:\Installs folder (in case I have to manually reinstall Windows in the future).

Checking Application Update Status

After I updated my out of date applications I then right clicked on the apps in SUMo and clicked Check to re check the status of the applications

SUMo Right Click Actions

Fyi: SUMo has a number of actions in the right click menu (on an installed Application).

Delete: Removes item from list. Will appear again on next scan or manually addition
Ignore: Blacklist : this Software/company pair will never appear again, whatever the version, file path, …
Skip: Items remains listed but a SPECIFIC version gets no longer notified as update for a given time (1 day / week / month) or forever.

(snip from here)

I had a few items I Deleted from the list or Ignored.

My Visual Studio (development) software was reporting an older version of Python (2.7). I updated Visual Studio to the latest version but Python 2.7 remained. I deleted Visual Studio from the list and was prompted (this is not deleting the application from the hard disk just from the generated list.

After a few hours updating apps

I whittled down the list of my applications that needed updating

I could sort applications in the list by OK status or apps that needed Minor or Major updates applied.

All Apps Updated

I can now say that all of my Applications are all up to date.

Export

I used the export feature in SUMo to build an excel file listing all my applications.

Fyi: SUMo exported this list as a “;” separated file, I opened the file in Sublime Text Editor and replaced “;” with TAB and then pasted this list back into Excel.

Conclusion

SUMo allowed me to update my installed applications and update my aged cache of application installs I was keeping in case I needed to manually rebuild Windows.

SUMo made this an absolute pleasure to do.

Pros

Price: 19.99 Euros (about $33.27 AUD) is a bargain.
Simplicity (it just works so well, why does Windows not do this)
Saves me time finding out of date apps and update files for an emergency and inevitable manual Windows reinstall.
My apps are more secure after updating them

Cons

I needed this in my life 10 years ago
Would be nice to have an inbuilt help file (instead of email support)

Try the free version here

v 1.0 Initial Post

Using Acronis True Image 2020 to backup and restore your Windows computer

May 24, 2020 by Simon

This is a lengthy (but simple) guide on how I get files ready for a clean install of Windows, backup and restore Windows (C Drive and associated hidden Windows partitions) using Acronis True Image 2020).

Backup and Restore Windows Prerequisites

You will need to buy a licence of Acronis True Image 2020 to..
- Backup Windows
- Restore Windows
You will need a second hard disk larger than your Windows drive (to backup to).

Backup Rules

Backup to Three Places.
Two of the media need to be different
One of the backups need to be offsite

Even though I use Back blaze software to automate backup of all files on my PC for $6.60 (AUD) a month. I pay an extra $2 (AUD) a month Backblaze will retain all of my files for up to 1 year. I also back up data to Backblaze Bucket(s) with Duplicati.

With all the horror stories of Windows updates killing machine’s it’s time to return to using Acronis True Image (not a paid endorsement) to start backing up the drive my Windows is on (all partitions)

I remember using Acronis True Image 8 in 2004 and loved it. Time to buy the new version. I loaded https://acronis.com/ (Australia Region) and noticed a cloud backup in the marketing information. I hope they still do local backups line in 2004.

I loaded the Acronis True Image 2020 link from the downtown menu (Australian link).

Acronis True Image 2020 Download Menu at acronis dot com

$69.99 seems fair,

Acronis True Image 2020 buy now button at acronis dot com

Darn, $69.99 is a yearly subscription. I don’t like subscriptions if I can get away with it.

Acronis True Image 2020 subscription and standalone options at acronis dot com

I clicked Buy under standard licence.

Acronis True Image 2020 was $83.99 AUD at acronis dot com

$83.99 is a bit high

Saving a dollar

TIP: Make sure you install the Honey Extension (from joinhoney.com) and use the Google Chrome Web Browser before purchasing to get a discount on Acronis True Image 2020.

The Honey extension applied the coupon so fast I could not get a screenshot it(so here is a screenshot with American Currency from a different web browser)

Now back to my purchase (and country), $33.60 is a nice saving.

I am paying for this because 16+ years ago I used an older version of Acronis True Image and the free version backed up OK but when I tried to restore from a boot CD I needed to pay?

Honey saved me $33.80

I am happy to pay $50.33 for Acronis True Image 2020.

After I purchased Acronis True Image 2020 I was presented with links to download the related files to use the software and to restore.

I downloaded the main Windows Acronis True Image Application for Windows and the Universal Restore program.

The files were about 900MB in total.

While the downloads were downloading I created an Acronis account

I added my purchased licence to my Account.

I was shown a welcome to Acronis page after creating an account.

The downloads finished and I started the AcronisTrueImage2020.exe setup file

I clicked Install

After the setup completed I was directed to a Quick Start Guide

Starting Acronis True Image 2020

When I loaded Acronis True Image 2020 for the fits time I had to agree to the Licence Agreement

A pasted in my just purchased Licence key

Start a trial or activate with a serial number

I already had an account (created minutes earlier in a web browser) so I clicked Sign In.

I was presented with a help wizard on startup.

Now I am ready to create my first backup.

Creating a Backup

Acronis True Image 2020 wanted me to back up all of my PC but it is 8TB, I don’t have an 8TB drive handy to back up to. For the life of Me I could not see how much 8TB would cost per month to backup.

By default Acronis wants to backup your whole PC

I clicked on my computer name (“ZENIGMA”) and selected “Disks and partitions“

I un-ticked all system drives and partitions that were not my C Drive disk. I have installed Windows 10 onto a Corsair MP600 M.2 PCIE 1TB SSD and I want to back it up. The M.2 SSD connects to my main board.

I selected this drive to backup.

View of all disks and partitions, I ticked the MP600 drive.

Oh dang, I don’t have enough space free on any drive to backup the C Drive to., Time to move some files.

Backup Destination

I have 8 partitions (drive letters) on 5 SATA hard rives and 2 M.2 SSD’s. I think I can move some data and free some space for this backup.

I had to shuffle data across drives to create enough space to backup my c drive

File Musical Chair’s

I used WinDirStat from https://windirstat.net/ to scan my U drive (my newest hard drive) for files to move.

I used WInDIRStat to find files I can move

WinDirStat shows a nice graph of files on my U Drive. WinDirStat colour codes file types by colour and also sizes squared by actual file size.

Windirstats scan results, Loads o Music I can ,move.

I could see 240GB of music that could be moved. My wife and I have converted all CD’s that we have purchased since the 1990’s to live alongside out Apple iTunes collection.

I used the Windows feature to right click on the Music folder and move it the folder to another drive.

Windows confirmed hat I wanted to move this folder.

It took 2 hours to move my Music folder to a smaller slower drive.

Now that I have 800GB free I can backup up my C:\ Disk

I re opened Acronis True image and resumed from Backup point where I left off and made sure I was backing up my C Drive (Corsair MP600 SSD)

My C drive is 500gb and the destination has 888GB free

Acronis showed the Windows drive to web backed up.

I clicked “Select destination“.

I clicked “Browse“

I selected “U:\AcronisBackup” as the target to backup C drive to

I was prompted for a encryption password (nice)

Now the backup is ready to start.

I clicked Back up now

Backup in Progress

The Backup was going to take about 40 minutes. I could see My C and U drive were busy in Task Manager

After the backup is complete Acronis verifies the Backup

After the backup finished I could see that Acronis managed to backup C drive (570GB) to a 329GB file on a different drive

Now I need to restore the backup to confirm it works.

Only a working backup is of value.

Creating Rescue (Recovery) DVD Media?

Now that I have a backup of Windows I need to create a boot media that will allow me to restore the backup image in case Windows dies.

I clicked in the 4th icon from the top in Acronis True Image 2020 then Rescue Media Builder

I clicked Simple

First I created a DVD boot disc.

I selected my DVD Dive (it had a blank DVD in it)

The Rescue Media needs a 800MB CD or DVD.

I clicked Proceed to burn the rescue media to the blank DVD

The Rescue Media DVD was burnt to a blank DVD, a few tips on using the Media were presented

I labelled the DVD Rescue Media and put it in a safe place.

Creating Rescue (Recovery) USB Media?

Now I will create a USB rescue Media Key.

I opened Acronis, Click the Tools Icon (same as the DVD method above), then Rescue Media Builder, click Simple and select your Empty USB key.

TIP: Make sure it is the correct Key as they key will be deleted first.

As with the DVD method above the Rescue Media files to be copied were about 800MB.

When the UBS key is prepared you will receive the same success screen as the DVD method above.

Restoring a backup from the DVD Rescue Media

I inserted the Rescue DVD into my DVD drive, I rebooted my PC and pressed F2 to enter my BIOS (your key may be different).

I navigated to the section in my BIOS where I could choose my Boot device and made the DVD drive boot first, this will allow my Rescue DVD to boot before Windows.

I rebooted my PC and I pressed a key when prompted to boot from the Rescue DVD

The DVD buzzed away for a minute and then I was prompted with Back Up or Recover menu. I clicked U:\Blog\Acronis True Image\DVD Restore.

I navigated to the U Drive (SFEARABY) Drive and selected the Backup I had taken earlier.

FYI: The Drive letters did not match in Windows and the Rescue Media. Take note of tour drive names. My U:\ Was listed as E:\.

I entered the password that I used to encrypt the backup.

I selected the backup of my C Drive to restore. I assume you can choose multiple, drives if you had backed up multiple drives.

I selected Recover whole disk and partitions.

I was asked to select the date to backup from (I assume this is for when you have multiple backup events over many days).

I was again asked to select partitions to recover.

A summary of the restore was presented and I clicked Proceed.

The Recovery was underway, It took about 8 minutes to restore over 500GB.

It worked.

I restarted my PC and removed the Rescue DVD.

Windows was all restored to it’s earlier state.

Restoring a backup from the USB Rescue Media

Restoring is exactly the same as the DVD method.

Reboot and Enter your BIOS
Set the Boos Order to allow the USB key to boot first
Reboot (no need to press a key to boot from USB)
Backup and restore (as needed)

I was able to restore the backup no problems,

FYI: I had the same slowness in the UI in the same spots as the DVD, nothing that breaks the process.

The USB and DVD Recovery offers a backup and restore options.

The Recovery media also allows you to clone a disk and setup a secure hidden partition for backups.

Backup Protection

Acronis True Image 2020 also protects backup locations and prevents access to them. I had to Allow Windows to be able to delete a backup.

Once I allowed Windows to be able to access the backup I was able to delete it.

Acronis allowing windows to manage backups files on Windows

Backup Schedules

Acronis True Image 2020 allows you to set Backup Schedules.

I do not have enough disk space to setup anything other than one full backup at a time

Backblaze

I still have Backblaze automatically backing up my files from within windows and this is my 2nd later of defence.

Backblaze will backup the 300+ GB backup.

Other Features

Active Protection

Acronis True Image 2020 has built in Active Protection to watch processes for ransomware or cryptographic. Read more here: https://www.acronis.com/en-au/active-protection/

I have had one false positive upgrading Java but ill leave it enabled.

A nice log is available of all events and whitelisted apps.

I have had to exclude apps but this has settles down.

A notification appears on blocked or whitelisted events 🙂

I can simply block or allow to deal with each exception.

Thus is a nice feature.

Cloud Backup

I am now using any Cloud backup features in Acronis, ill let Back blaze handle that.

Archive large or old files

The Archive large files or old files is a bit basic for me to use.

I will keep using CCCleaner and WinDirStats for this job.

Even the Windows default Disk Cleanup is good.

Cloud Sync

I do not use Cloud Sync, I would but I can’t find a price on the Acronis site at all.

I noticed that on the DVD and USB recovery boot I can restore a windows from a Acronis cloud backup (nice).

Other Tools

Clone disk (like EASE US?)
Universal restore?
Try and Decide
System Clean Up (backup First)
Acronis Secure Zone
Drive Cleanser
Third Party Tools

Acronis True Image (Conclusion)

I am being picky here as I loved using Acronis 10 years ago and I have rose coloured glasses.

Pros

It works
You can backup Windows while Windows is open
DVD and USB Rescue Media creation is easy
Backup Schedules
Notifications of disk space and operational state
Exclusions of non essential files
Backup Validation (at time of creation and schedule)
Ability to call pre and post backup commands
Choice of Incremental, Full or differential backups schemes
Cleanup of older incremental versions (and full version frequency)
Simple single version backup (single file, overwrites each time)
It works.

Cons

Some Buttons in the GUI are slow to respond
The DVD an USB Rescue Media is little unresponsive at times during the restore (but it works)
The DVD and USB Rescue Media GUI can use a bit of design love (I am sure it was better 10 years ago (basic and clean))
When I restored I had no Idea what C:\ Drive was listed as H:\ (loads of confusion on the internet)

Disclaimer

I am not paid by Acronis to say this, this is just my guide for my friends. I paid with my own cash. I take no responsibility if this does not work for you. Good luck though.

Troubleshooting

DVD Rescue Media

The DVD/USB Rescue Media is a bit laggy. At one point in the recovery I has a white screen for over a minute. I waited and the screen came good

Also the Menu in the DVD/USB toolbar does seem a bit squashed.

Bonus: Windows System Restore

There may be a time when you have created a full Operating system backup but the restore does not work (e.g hardware has failed (and been changed)). Acronis True Image 2020 will allow you to restore files from a backup image (and not the whole partition or disk).

Make sure System Protection is enabled for your C Drive

Create a System Restore Image

If you have Windows 10 it is a good idea to create a System Restore point ever few months just in-case Windows goes bad.

Click Your Start Button then type “Create a restore point“

Click your C Drive and click “Create“

The restore point will be created

Restoring a restore point

At any time you feel that Windows is sick (and still working) you can restore and old system restore.

Click the the desired restore point then click Next

You will be able to restore the old system snapshot.

Bonus: Clean Windows 10 Install

TBA: Blog post coming soon

Bonus: How I prepare all files needed for a clean install

Before you do a clean install of Windows 10 you may need to document what software you have installed so you have a change to reinstall them after you reinstall Windows.

This is possibly the most boring job but the one with the most reward. The hardest party is knowing what software you have an need.

Finding Installed Software with SUMo

I use a free program called SUMo from KC Software (not a plug) to list all installed programs and to see what software is out of date.

SUMo informed me that I have 256 software products installed.

7 were really out of date
62 products had recent updates

I took this list and downloaded and installed new copies of the apps I had installed (as newer version’s of apps can be more secure) and I also saved the installer files to a folder away from C Drive (e.g B:\Installs).

I am a bit OSD and I have created 10 folders under the B:\Installs folder that contain installs from Essential Drivers to non essential apps that I can use in the event of a clean install of Windows

In the Installs folder I have these sub folders

\100 Drivers\
\200 Essential Apps\
\300 Productivity Apps\
\400 Development Apps\
\500 Utils\
\600 Games\
\700 Virtual Machine OS Installs\
\800 Virtual Machine Apps\
\900 Learning\

When reinstalling Windows I start by installing all apps in each group (starting with “100 Drivers”), The larger the parent folders’ number is the less important the files are int hat folder.

I have a Legend.txt with a summary of the contents of each folder

Install Folders Legend

100+ ~ 199 Drivers
     - Mainboard ASUS RUF X570 (WIFI)
     - AMD Chipset Drivers
     - Nvidia Video Card
     - Etc
201+ ~ 299 Essential Apps
     - Printer Drives
     - 1Pasword
     - Antivrus
     - Backblaze
     - VPN
     - Acronis
     - Etc
300+ ~ 399  Productivity Apps
     - Microsoft Office 
     - Microsoft Visio
     - Etc
400+ ~ 499  Development Apps
     - Visual Studio
     - Arduino IDE
     - Fritrzing
     - Microsoft SQL Developer
     - MySQL
     - Python
     - PHP Storm
     - Etc
500+ ~ 599 Utils
     - CPU-Z
     - Core Temp
     - Acrbat Reader
     - Etc
600+ ~ 699 Games
     - Steam Installer
     - Origin Installer
     - Etc
700+ ~ 799 Virtual Machine OS Installs
     - Windows 3.11
     - Windows 95
	 - Etc
800+ ~ 899 Virtual Machine Apps
     - Office 95
     - Office XP
     - Etc
900+ ~ 909 Learning
     - Python Books
     - Etc

You can create whatever numbering scheme you want.

I also have a cached folder of my Development Software (Visual Studio) Install cache at “B:\Installs\vscache“

Under my drivers folder I have all the drivers I need to reinstall windows

An under my “500 Utils” folder is all the utilities I install (many).

Hundreds of sub folders with utilities in them

The main thing is I have all the files and installers needed to do a clean setup of Windows 10 if need be.

Links

Version History

v1.1 Added “Backup and Restore Windows Prerequisites”

v1.0 Initial

Backing up your computer automatically with BackBlaze software (no data limit)

June 2, 2019 by Simon

Backblaze ( https://www.backblaze.com/ ) is an awesome company who not only create a mega reliable backup infrastructure but they also tell us how they do it and tell us what hard drives are good or bad.

Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj

They are so confident and experienced that they offer unlimited backups for $6.60 a month here in Australia. They state they have 750 Petabytes stored on their infrastructure & have restored over 40 billion files to customers.

Check our my other related posts

Main features of Backblaze.

No limit to files that you back up (number or size or files)
Web-based file recovery (if needed)
They will mail you a hard drive if you need to restore large amounts of files.
Ability to locate lost or stolen computers.
Mobile app access your backups.
Business options for large computer fleets.

Creating a Back Blaze Account

Go to backblaze and create an account.

Choose a Backblaze Plan

Go to: https://secure.backblaze.com/buy_plan.htm to choose a backup and payment plan. Use my link and get your first month free.

Australian Prices below (add +10% GST).

$6 AUD a month
or $60 for 12 Months
or $110 for 24 Months

Screenshot $6 AUD a month or $60 for 12 Months or $110 for 24 Months

Choose a payment plan

Enter payment details

Click “Buy Backblaze”

Use my link and get your first month of Backblaze backups free: https://secure.backblaze.com/r/00e1wj

Done, now I can download the client.

Installing the Backblaze Software

When the download is down you can start the install.

You will need to login to the install to streamline the setup.

Backblaze install screen asking me to login

After you login the installation will begin.

Backblaze is installing and checking what files need backing up.

Backblaze now reports that the install has complete. Backblaze reports that I have 379GB to backup.

Pressing OK shows Backblaze is already uploading my files.

Also, I have 15 days of free trial before being billed.

Clicking on the backblaze system tray icon reveals a few options

Backblaze Preferences

Backblaze preferences can be opened by clicking Settings from the main screen.

I..

Renamed the computer.
Disabled backup on battery power
Set upload threads to 5 (then lowering if the network gets busy or when the backup is almost complete)
Disabled Automatic throttling and set faster backups

I set a private encryption key

There is a 50 char limit on encryption keys

50 char limit on encryption keys message

Warning about not forgetting the encryption key

All other preferences look ok.

That’s it the backup is uploading automatically and there is nothing else I need to do.

Backblaze Portal

If you login to the backblaze portal ( here ) you can view your files (you will need to enter the private encryption keys if you opted to encrypt your files).

Initial Backup time

Your initial backup may have a while to upload depending on your backup size and internet connection speed.

I was able to upload 400GB in 3 days on my Internet Plan. Smaller files are uploaded first then larger ones. I have 2 files left to upload totalling 7GB.

Read the official guide on speeding up uploads here.

One way to make the initial backups faster is to see what files are queues to upload in the Backblaze preferences and then excluding unwanted files and folders.

I can see my Internet Explorer cache and Google Chrome Temporary Internet File(s) folders are queued to be back up.

Time to exclude these folders in the Backblaze Settings under exclusions

C:\Users\Simon Fearby\AppData\Local\Microsoft\Windows\INetCache\IE
C:\Users\Simon Fearby\AppData\Local\Google\Chrome\User Data\Profile 1\Code Cache\
C:\ProgramData\MySQL\MySQL Server 8.0\Data\#innodb_temp\
C:\ProgramData\USOShared\Logs\
C:\Users\Simon Fearby\AppData\Roaming\Adobe\SLData\SLCache\

Time to exclude these folders.

That knocked off a few thousand files to backup 🙂

Restore Files from Backblaze

If you restore files you have a few options

Download the files (free)
Save files to B2 Cloud (Backblaze)
USB Flash Drive
USB Hard Drive

restore options. 1) download, 2) save to BS cloud, 3) USB Key, 4) USB Hard Drive

Choosing to download files you are given a treeview to restore files from.

I selected a test file to restore (restores are not instant) and was informed I would be emailed when the files are ready to download. Thats cool

I waited 3 minutes and received an email that my files were ready to download.

I hope this helps someone.

How to back up an iPhone (including photos and videos) multiple ways

June 2, 2019 by Simon

This guide is for and anyone who needs to back up their iPhone (complete device along with separate backups of photos and videos).

This is not a paid promo, I don’t get a kickback for mentioning the awesome app below.

Check our my other related posts

At the time of writing iTunes v12.9.5.7 was the latest version of iTunes.

iTunes is Apple official software for getting files too and from an iPhone, iPod or iPad. If rumours are right Apple will kill off iTunes software at the end of 2019. If Apple kills off the iTunes software I will update this guide in the future.

Disclaimer: I provide no warranty or support for this app, this is what happened to work for me.

Why write a guide on Backing up a mobile phone?

It’s not that simple an I get asked how to do this almost every week by someone.
No one thinks to backup up their photos until they fear their phone is lost or stolen.
I am about to publish a few posts on backing up (development machines, servers etc) with automated software and with free open source software so this post will be handy to link in what data you can back up in that post.
The last time I wrote an article on backups was in March 2016 and its not that great these days.

Golden rules of backing up.

Backup to Three different locations.
Two of the media need to be different.
One of the locations needs to be offsite
Test your backups (a backup is only good if you can restore). (from @Daniel15)
If the backup is encrypted make sure you have the decryption keys. (from @Daniel15)

Backing up the official way (with iTunes)

Apple prefers you just pay them to extend your free 5GB iCloud storage and not worry about backups and not to worry yourself with manual steps.

My wife’s iPhone is always popping up messages saying that her iCloud storage is nearly full and she should upgrade the free space limit.

Screenshot of iCloud setting on iPhone saying your iCloud is nearly full but you have used 5GB or 5GB?

I know she has 100GB+ images and movies on her iPhone but iCloud has only backed up 4GB of photos and videos. Do not trust any backup statuses unless you can verify all files from a restore.

For the record I use an Android phone (Android backup guide coming soon).

iCloud is a good idea for automatic backup but I prefer to see my photos and backup them up myself. Also, Apple iCloud is not free from troubles. If you want Apple to handle backups then I do suggest you upgrade your iCloud storage from the free 5GB to a more sensible 200GB or more.

Screenshot of https://support.apple.com/en-au/HT201318

Here in Australia Apple charge the following for extra storage.

50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

A rough estimate: 50GB is enough to store about 12,000 files (9,000 photos + 3,000 videos) *

* = based on stats from the backup below (depending on the size an length of your videos).

Screenshot of apple iCloud pricing 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

You can upgrade your paid Apple iCloud backup limit from your iCloud Storage Settings from your iPhone’s settings screen.

Screenshot of iPhone iCloud upgrade button.

IMHO: Apple does make it clear what you are getting before you purchase (but they do not push it enough and people assume their data is safe).

iPhone picture of 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

TIP: Please review your existing iPhone data usage and iCloud usage before it’s too late

I don’t want ot pay Apple for more iCloud storage

I like you, I am tight too but I do pay Google $2.49 a month for 100GB backup storage on my Android phone. Google Drive (Google storage is a bit cheaper than Apple).

Screenshot of Google Drive https://www.google.com/drive/

It is nice knowing your phone is auto backed up.

Install iTunes on Windows (10)

On your Windows 10 computer click the start button. If you don’t have a Microsoft Account go here and create one. Click the start button and type “itunes” (you don’t need to type into a box, just start typing when you click the start button)’.

Then click “iTunes, install app”

Screenshot of a Windows start menu and Get iTunes icon

If you are logged into your Windows 10 store with your Microsoft ID click “Get” (if not you will need to login to the store).

Screenshot of me logged into the Windows Microsoft Store.

After you click ‘Get’, iTunes will start downloading.

Screenshot of the Windows Store downloading iTunes

When iTunes is downloaded Windows will install it. When it’s installed click ‘Launch’

iTunes is installed, Click launch screenshot

You will need to agree to Apple’s terms of service. Click ‘Agree’

When iTunes opens click “Agree”

Screenshot of Apple iTunes asking if we can agree to share analytics data

Now login to iTunes with your Apple ID (if you don’t have one create one here)’.

Screenshot of iTunes open with an arrow pointing to the Account menu

Click the ‘Account’ then ‘Sign In’ menu.

Screenshot of the account then sign in menu

Optional: If you have Two Factor Authorisation (2FA) of Apple ID’s turned on (you should) you will need to enter a 6 digit code.

Apple Two Factor Authorisation (2FA) will send one of your other devices a login code that you will need to use to login (confirm you own the account)

Screenshot of apples 2FA web page at https://support.apple.com/en-au/HT204915

TIP: Check out https://twofactorauth.org/ to see what other sites use Two Factor Authorisation (2FA). I use hardware Yubico YubiKeys to protect logins to WordPress, Linux and websites.

Now back to the article, iTunes should be ready to allow us to backup our iPhone.

Before I continue I will click ‘Edit‘ then ‘Preferences‘ menu to view where iTunes will download media too (different ot backup data), not important but I just want ot see if it is not pointed ot my smaller C drive before I backup my phone

I changed the location to a larger S:\Drive’

Screenshot of Apple setting screen (tab Advanced)

I was greeted with a message asking me to confirm that I wanted to sync the iPhone that I just plugged in called ‘EllieRose’, I clicked ‘Continue’

On the iPhone I also clicked ‘Trust’ to allow iTunes to talk to it.

iPhone screenshot to allow iTunes to talk to it.

I was then prompted to download a software update to the iPhone, It appears this phone is not running the latest software

Screenshot of a question to update to iOS v 12.3.1

I was prompted to sync the iPhone to the computer

screenshot of a prompt to sync purchases from the iPhone to the computer

A backup of the phone was underway.

Screenshot of iTunes backup progress with a progress bar

If the screen above does not appear click the icon below to view the backup and restore iPhone menu.

Screenshot showing the iPhone Device button in iTunes

The iPhone was backing up to this folder on my computer:

C:\Users\Simon Fearby\Apple\MobileSync\Backup

I was not prompted for a location to back up so and I will move this backup folder after the backup completes (so my C: drive does not fill up).

My drives.

C:\ = 500GB drive (faster SSD)
S:\ = 2,000GB Drive (slower Magnetic)

Screenshot showing iTunes backup and the C:\Users\Simon Fearby\Apple\MobileSync\Backup folder

The iTunes iPhone backup is now complete (it took about 2 hours).

Now that the iPhone was backed up iTunes the Operating System update started to run.

Screenshot of iTunes verifying the iPhone update

Update verified.

Screenshot of iTunes updating the iPhone OS.

During the update the iPhone was unavailable.

Screenshot of iPhone installing firmware

Done, the iPhone had updated it’s operating system in about 30 minutes.

Screenshot of iTunes showing the newer OS version.

Now lets see how much space I have available on my C Drive.

Screenshot of disk usage on my c and S drive.

It looks like the iPhone backup iTunes made was 60GB in size.

Screenshot of Windows reporting the backup was 60GB in size.

Using Windows Explorer I moved the backup from..

C:\Users\Simon Fearby\Apple\MobileSync\Backup

to..

S:\Backup\AlisonsiPhone\iTunes\31May2019

(right click drag an drop action in explorer)

Screenshot of right click drag and drop move folder.

It took me about 20 minutes to move the 60GB folder.

Screenshot of thew Windows copy dialog progress.

TIP: You can (should) also copy this backup folder to..

Now this backup is available for me to restore in the future if I need it.

In the case of a restore, I just need to move the backup into the iTunes expected location.

Copy from..

S:\Backup\AlisonsiPhone\iTunes\31May2019

..to..

C:\Users\Simon Fearby\Apple\MobileSync\Backup

The Catch

The catch with back-ups made with iTunes is they are useless if you wanted to restore individual files (say to find a photograph or video). iTunes backups are usually tens of thousands of files with random folders and filenames

Screenshot of iTunes nasty backup, i obfuscated all files to random guid filenames

If you want to JUST backup photos and videos read on.

How to backup just photos and videos from your iPhone.

This part of the guide needs a paid version of the iOS App Photo Transfer App. In Australia, the app is free but to turn it into a full version needed to restore everything (via in-app purchase) it cost $10.99.

This app is well worth $10.99 AUD (it may be cheaper in other countries) to have it push photos an videos from your iPhone to a free companion Windows program (a Mac versions exist too).

Screenshot of http://phototransferapp.com/

Buy the iOS version and install it on your iPhone.

Then download the free Windows version: http://phototransferapp.com/win/

Extract the files from the zip file.

I extracted PhotoTransferapp.exe from PhotoTransferapp.zip

When I run PhotoTransferApp.exe I get an error saying I need to install Adobe AIR run times.

Screenshot of me installing Adobe Air run times from https://get.adobe.com/air/

Go to https://get.adobe.com/air/ and download and install the Air run time.

Screenshot of https://get.adobe.com/air/

Install the Adobe Air run time.

Install Adobe Air From https://get.adobe.com/air/

I installed Adobe AIR and again reopened the PhotoTansferApp.exe and was prompted to allow access to the Windows Firewall.

I was on a home network so I enabled firewall access.

Screenshot of Photo Transfer App asking fore firewall access.

The Photo Transfer App prompted me to ‘Discover Devices‘ and to make sure the Transfer app is running on the iPhone.

Before I clicked ‘Discover Devices” on the Photo Transfer App on Windows I opened the Transfer app (mentioned above) on the iPhone.

On the iPhone I clicked ‘Send‘

The transfer app will ask for access to your photos, you will need to press OK.

Screenshot of the transfer app asking for permissions to photos

On the iPhone again I clicked ”Windows‘

The iOS Transfer app now said I should run the ‘Photo Transfer App‘ on Windows.

TIP: You can transfer over WiFi (if your iPhone and Windows device is on the same WiFi Network) or you can transfer over a USB cable.

Before I started the ‘Detect Device‘ or ‘transfer‘, I set the Backup path location to my S: drive by clicking settings in the bottom right and choosing a folder (as my C Drive is a bit small).

Screenshot of Photo Transfer App choosing a folder to backup to.

I noticed the port of 57777, I temporarily disabled the whole Windows Firewall just in case it prevents the photo backup.

In hindsight, it was not a good idea to disable the whole firewall, but because I was at home on a safe network I felt safe to do so. Next time I will not disable the firewall and see if this still works.

If you are on an internet cafe, school or university network do not disable your firewall.

I clicked ”Start” then typed ‘firewall‘ and clicked ‘Windows Defender Firewall‘

Screenshot Windows Defender firewall icon

I turned off my firewall.

I clicked ‘Detect Devices‘ and the iPhone ‘EllieRose‘ appeared on the left. I doubled clicked on the iPhone name and was prompted with an Authorization required message.

Screenshot: Authorisation required message.

I looked at the ‘Transfer‘ app on iOS and clicked ‘Yes, always‘ to allow access.

Screenshot: Authorisation yes/no on the iPhone

After 20 seconds I can see photos on my iPhone on Windows.

Screenshot, all of my iPhone images were appearing on Window.

I selected all camera albums to backup and clicked backup. I prompt to upgrade to the full version will appear, you will need to buy the upgrade.

Full backup happens only if you upgrade to a full version via an in app purchase

After the full version if purchased the backup will be allowed.

You can upgrade from the free to paid full version from the main screen of the Transfer app.

Screenshot get full access by upgrading the free app to a paid version

The upgrade In App Purchase is $10.99 AUD

Screenshot of a $10.99 In App Purchase on the iPhone

When you purchase the In App Purchase you can run the backup again.

Screenshot of all camera categories on the iPhone on windows

Now I could see photos being copied from the iPhone to my defined backup folder.

I went to bed as I knew there were about 100GB of files on the iPhone and this was going to take a while.

In the morning the backup was done.

26,000 files were backed up (over 100GB).

I now had a full iPhone backup made by iTunes and a copy of all photos and videos.

Screenshot showing the 2 backups (a) iTunes made and b) Photo Transfer App made)

I turned on the Firewall again.

I now had 160GB of backed up photos, videos and phone backup.

Backup your iPhone backups to the cloud

The steps to do this two ways will be added soon.

I will add a section on how you can back up the iTunes and manually synced photo and video backups to the cloud automatically and a more complex but cheaper was for 0.005c per GB.

Watch this post.

How to backup and Android Phone

Article coming soon.

How to backup and restore a MySQL database on Windows and Linux

April 21, 2019 by Simon

Why backup and restore

This is a quick guide demonstrating how you can backup and restore a MySQL database on Windows and Linux using Adminer.

You may need to know how to backup a restore a database for a number of reasons..

e.g

Send the database to someone to debug or give feedback while learning.
Move the database from a local machine to the cloud
Move the database from cloud vendor A to cloud vendor B
etc.

Having a backup of the VM is good but having a backup of the database too is better. I use UpCloud for hosting my VM’s and setting backups is easy. But I cannot download those backups.

Murphy’s Law

“If anything can go wrong, it will”

The most important reason for taking a backup and knowing how to restore it is for disaster recovery reasons.

Backup (the easiest way) with Adminer

Adminer is a free PHP based IDE for MySQL and other databases. Simply install Adminer and save the file on your local computer or remote web server directory.

FYI: The Adminer author Jakub Vrana has a patron page, I am a patron of this awesome software.

Snip from Adminers website. “Adminer (formerly phpMinAdmin) is a full-featured database management tool written in PHP. Conversely to phpMyAdmin, it consist of a single file ready to deploy to the target server. Adminer is available for MySQL, MariaDB, PostgreSQL, SQLite, MS SQL, Oracle, Firebird, SimpleDB, Elasticsearch andMongoDB.”

TIP: The file would be publicly accessible to anyone so don’t save it to a common area, obfuscate the file, protect it of delete the file when you are done using it.

Once Adminer is installed load it in a web browser, login with your MySQL credentials. Once you login you will see all databases and an Import and Export menu.

Adminer main screen, all databases and import and export menu.

tbtest is a simple database with one table and 4 fields (ID, Key, Value and Modified)

.Click Export to open the export screen.

Export screen showing a list of databases and export options

Click Export, a SQL file will be generated (this is the export of the database).

Here is a save of the file:
https://fearby.com/wp-content/uploads/export.txt

Exported view of https://dev.mysql.com/doc/workbench/en/wb-admin-export-import-management.html

Its that simple.

If I add a binary blob file to the table and upload a PNG file lets see how the export looks.

Screenshot o the new table with a blog field in Adminer UI

Let export the database again in Adminer and check out the output. I used Sublime Text editor to view the export file.

New Export shows the binary file in the Backup SQL file

Restore (the easiest way) with Adminer

OK lets delete the tbtest database and then restore it with Adminer. I used Adminer to delete (DROP) the database.

Database “dbtest” deleted.

Now lets create a blank database to restore to (same name).

Database created.

Now lets import the database backup using Adminer.

Click Import, select the backup file and un-tick Stop on errors.

Import screenshot, dxtest selectded, Restore file selected, stop on errors disabled

TIP: The 2MB next the the choose file button is defined by your web server and PHP configuration. If you are trying to import a larger database (e.g 80MB) first increase the limits in your web server and PHP (via php.ini).

The Import (restore should take seconds)

The database was imported from a backup, all tables and records imported just fine.

Bonus methods.

On Ubuntu use this guide to backup from the command line. If you use the Oracle MySQL Workbench read this.

I hope this helps someone.

How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains

March 31, 2019 by Simon

This is a quick guide that will show you how you can connect to a cloud server via SFTP with the PHPStorm IDE from Jet Brians and deploy files from your localhost to the cloud. This is my opinion, I am not paid to promote PHPStorm or UpCloud.

Pre-Requisites/Assumptions

This guide will assume you already have (or know how to)..

Buy a domain name and point it’s DNS to a server (I use Namecheap.com for buying domains)
Buy and deploy a server in the cloud. I have used AWS, Digital Ocean, Vultr but now use UpCloud for deploying fast self-managed servers. Read this guide here to see how I create a server from scratch on Up Cloud.
Setup SSH access to your server and configure a firewall.
You have or know how to set up PHP and Web Servers and configure them on your localhost and remote server (guides here, here, here, here, here and here).
etc ( check out all my guides here https://fearby.com/all )

I am using Windows 10 Home (with IIS Web Server (document root redirected to S:\Code\) and a pre built Ubuntu Cloud servers.

Why no FTP? I do not create FTP servers on my serves to increase security and I only access servers via SSH via white-listed IP’s and then authenticate with hardware 2FA keys from YubiCo (read me 2FA guide here and also how to secure *nix servers and WordPress with 2FA).

Background

2 years ago I used to use the Cloud 9 IDE to connect to, and code files on cloud servers and life was good. I could configure and connect to servers, drag and drop files, run bash scripts from a web page and close the Cloud 9 browsers tabs, travel hundreds of kilometres and log back into C9 and all code and bash scripts would reappear.

Here is the Cloud 9 IDE showing code on the left and a Browser on the right.

C9.IDE showing code on the left and web page on the right

With Cloud 9 code could be easily accessed, edited and run.

See screenshot of code running from a Cloud 9 hosted server with properties windows on the right.

C9 IDE showing a bash terminal windows and code

Regrettably, I cancelled my $9/m subscription to Cloud 9 after a minor stroke and since then I have gone back to a terminal screen to code and transfer files. In the last 2 years.

Screenshot of the putty program connected to a Ubuntu box editing a file with nano

Uploading and downloading files on mass is painful via pure SSH.

AWS has since purchased Cloud 9 and I am not sure if it will ditch support of non-AWS servers in the future. AWS is good but servers are very expensive for what you get IMHO. I have found Disk IO on UpCloud is awesome (also UpCloud support is great (I am not paid to say that)).

PHPStorm IDE?

A quick Google of IDE’s like Cloud 9 mentioned PHPStorm from Jet Brains. I have used IntelliJ IDEA from Jet Brains before and PHPStorm seems to be very popular.

Go to
https://www.jetbrains.com/phpstorm/ and see the features of PHPStorm

Watch PHPStorm in action

Whats new in PHPStorm 2019.1

Install PHPStorm

Visit https://www.jetbrains.com/phpstorm/download/ and download and install PHPStorm (free trial 30 days)

System requirements

Microsoft 10/8/7/Vista/2003/XP (incl. 64-bit)
2 GB RAM minimum
4 GB RAM recommended
1024×768 minimum screen resolution

Pricing

PHPStorm is $8.90/m for individual use (or $19.90/m commercial). For the first 12 months of uninterrupted subscription payments qualify you for receiving a perpetual fallback license (20% discount for an uninterrupted subscription for a 2nd year, 40% discount for an uninterrupted subscription for 3rd year onwards).

PHPStorm work on Windows, OSX or Linux. This great an I use Windows locally and Linux remotely but I’m keen to use Linux locally to match local and remote dev environments.

Official PHPStorm Pricing Page:
https://www.jetbrains.com/phpstorm/buy/#edition=commercial

fyi: Jet Brains has free licencing for individual use for Students and faculty members.

Creating a Project

Open PHPStorm and select “Create New Project”.

Choose a project type on the left (e.g “PHP Empty Project“) and choose a location to save too (I chose “S:\code\php001) on my local machine. I chose “S:\code\php001” on my local machine.

New Project screenshot asking for a name and save location

Choose a folder to save to.

Choose a project and location to save to locally

Click “OK” to create the project.

PHPStorm will have created a project for you. You will notice a “.idea“folder under the location you saved with these files.

misc.xml
modules.xml
php001.xml
wordspace.xml

Do not delete these files.

Creating your first PHP file

You can right click on the project root and select New then PHP File

Right click on the root in the tree view then new PHP File

Or clicking the File then New menu choosing PHP File.

Name the file (e.g index.php)

The file has been created and its available in my localhost web server.

Screenshot showing PHPStorm with index.php, S:\Code showing index.php and http://localhpst/php001/index.php loading

Creating a Deploy Target

Now we need to specify a deploy target in PHPStorm to push the file changes to the cloud. Backup your server (yes backup your server just in case).

Open your PHPStorm project and click Tools, Deployment and then Configuration.

Click the plus icon near the top left and choose SFTP

Screenshot showing add new deployment server (SFTP)

Name the deployment target (e.g “server (project)”)

Screenshot of an input box showing a server name

Enter your “server name” or IP and port
Enter your “ssh username” (ensure the SSH user had write access to the wwwroot folder and the web server can read the files written by this user)
Under password I chose “Key pair OpenSSH or Putty (as I had SSH details already setup in Putty details
You can add your ppk private key from Putty (use the puttygen program to conbvert ssh public and private kets to ppk format)
If you have a passphrase on your SSH key add it now
Enter your web servers remote path (for the project)
Enter your web server URL

Screenshot showing a server name, port, username, password, ssh file passphrase, root path and web server url.

I did SSH to my remote server and created the destination folder. This will ensure I can deploy code here (PHPStorm does not create the remote path fpor you ).

mkdir /wwwroot/php001
chown -R www-data:www-data /wwwroot/php001/

Click Test Connection

No we need to click the Mappings tab and add a mapping.

Local path is your local path
Deployment path is / (the web root path is carried forward from the previous tab)
Web path is the web path that is entered in the browser

Screenshot showing a manual file mapping of local and remote file locations

Click Add New Mapping. Now we are ready to deploy

Deploying code to the cloud

I right clicked on the root note in PHPStorm and created an index.php file.

I edited the index.php on my local machine and then click the Tools then Deployment and choose “Upload to fearby.com (php001)” menu.

Manual upload available in Tools menu then Deployment menu

The File Transfer output window showed the transfer progress.

Screenshot showing the file transfer window output saying the file uploaded.

I loaded https://fearby.com/php001/index.php in Google chrome. It worked.

Screenshot showing https://fearby.com/php001/index.php loaded in a bowser

Don’t forget to turn off Automatic uploads under Tools, Deployment menu.

'Screenshot showing Automatic updated turned on

Now when I create new files or change existing files they will auto upload.

Sourc Control

I will add this soon.

Shell Command

You can also open an SSH console to the server and run commands

e.g zip files

zip -r backup.zip .

I can also open a folder window in PHPStorm and show all remote files by clicking Tools then Deployment then Show Remote Files, Zip files can be easily downlaoded or other files uploaded. Nice.

Linux Client

I will review the Linux PHPStorm client soon.

Troubleshooting

Watch the Official guide on Deployment and Remote Hosts in PhpStorm – PhpStorm Video Tutorial

Good Luck. I hope this guide helps someone.

Version

1.2 Removed advertisements

1.1 Minor Updates

1.0 Initial Version

I am moving away from Apple hardware

February 3, 2019 by Simon

My Late 2012 Mac Book Pro Retina laptop is all but dead, it has many dead pixels and because of the poor cooling and is NOT a joy to use anymore. It does not “JUST WORK” and personally, I do not think “thinner” laptops can handle Australian summers as its hardware cooling it inadequate above 40c air temperatures.

My laptop processor would spend more time thermal throttling (at 104c) in Web Browsers and text editors that at normal speeds. Opening up productivity apps like Photoshop or Premiere Pro would send the laptop into meltdown.

Frequent high temps were common.

Attempted Fixes

Warning Disclaimer: My laptop is out of warranty and I know my way around the inside of a computer hardware without zapping it. Do not attempt to open your laptop unless you know what you are doing, have backed up your data and are prepared to brick your computer.

I removed dust from inside the laptop.
I tried to only use the laptop refrigerative air conditioning
I replaced the thermal paste on the CPU and GPU (3 times)
I reinstalled OSX Mojave and reset the SMC and PRAM multiple times.
I ran the fans at 100% (see post here), The fans were operating at full capacity and were not broken.

The stock thermal paste was crusty after 5 years. The plastic CPU/GPU cover was visibly cooked.

I ordered some new Thermal grizzly thermal paste, I had some older silicone paste on hand just in case.

After many reapplications of the Thermal Grizzly, the older silicone paste seemed to work the better???

After a few months, all of the fixes above did not seem to work. OSX Mojave would spin up the CPU and GPU into a frenzy overloading the single heat pipe within minutes.

Time to try some more drastic cooling modifications?

I tried improving the efficiency of the single (copper) heat pipe that is shared between an Intel i7 2.6 GHz and an Nvidia Video Card by removing the black paint by stripping the paint with acetone.

I manually removed paint from in between the heat sink fins with a LED to reveal the metal.

I reinstalled the heat pipe with high hopes? That looks nice 🙂

I removed the old thermal paste and added new paste. First I tried Thermal Grizzly Cryonaut. I re-applied the paste three separate times as each application was not that much better than the old crusty stock paste from Apple. Did I have a bad batch of Thermal Grizzly?, It seemed thick and not very viscous. I ended up using an old tube of silicone paste (the white stuff) as my Arctic Silver was too old to try and I did not want to order more.

With the silicone paste applied and the paint removed temperatures were about 15c lower at max, I still had frequent thermal throttling but at least I had a reserve buffer.

This was all before the Aussie Heatwaves and high temperatures soon returned.

Is there still room for improvement?

How heat pipes work

Heat pipes have an evaporating (hot part) and condensing zone (cool part) on the heat pipe. I noticed Apple’s “stock” condensing fins were small, would improving this zone help?. Time to improve the condensers zones by adding larger copper heat sinks to the bare side of the heat pipe.

I purchased a few copper Xeon/Sun server sized heat sinks and thermal epoxied them to the condensing end of the heat pipe. Yes, they would protrude out the bottom of the case but #Meh. I can fix that by extending the base of the laptop down and making it thicker (old school style).

The server heat sinks arrived

I cut the heat sinks in half.

I packed the fins with paper before cutting to ensure the cut did not damage the fins.

After cutting, I wiped the copper heat sinks with vinegar to restore the surface to a nice copper shine.

I tested the heat sink idea with silicone paste first

Temps were 25c lower, Now it’s time to use Arctic Silver Thermal Epoxy

I applied the Thermal Epoxy to the heat pipe (I temporary had foil strips above the fans so I did not block them while the epoxy dried.

I then stuck the heat sink’s to the heat pipe (with Arctic Silver Thermal Epoxy).

I toyed with a clear case but decided against it for static electricity and stability reasons.

I purchased a second Mac Book base for so I could cut holes for the heat sinks to protrude and use the original base to hide the modification.

I made a 30 mm base wall to so I could use it as a wall between the laptop base and the new 30mm lower base.

I added some 5-volt and 12 -volt fans inside the new extended 30mm base.

Finished Product

A normal looking Mac Book except for the 30mm lower base and internal 5V or 12V fans.

External power plugs on the left side, I will add lights at a later stage.

Are the temps lower?

Videos

Video: Mac Book Pro cooling mod, I can now watch 1080p videos without maxing the CPU

Video: Mac Book Pro cooling mod with external powered 5v or 12v fans

Conclusion

50c lower temps are nicer at idle but in Premiere Pro (exporting video) the laptop was still thermal throttling like mad and temps were terrible (100+). Lets not get started when I start some development VM’s

Conclusion 2 weeks later

This is still not a joy to use. I don’t think I have the right to expect a 5-year-old laptop to keep up running a CPU/GPU intensive OS and applications.

Time to buy a new computer, Apple still makes thin and over heating laptops by the looks of it?

Maybe I need to buy a fridge to stick a computer in a fridge to use these days?

YouTube users indicate Apple has a problem with heat.

What computer do I get next?

Not an Apple made one. I will be moving back to Windows for local development and Linux on servers

Dell Alienware has many heat pipes.

Acer Predator 500

I read a few reviews (e.g this one from Ultra book reviews) and Acer have good cooling.

MSI GT Series laptops look the best if cooling is important.

Or should I build a custom desktop with way more cores

CPU: Threadripper 2950X 16C 32T

SSD: M.2 SSD: Samsung 970 PRO 512GB
MOBO: Asus Zenith Extreme
Power: Corsair RM1000x 1000W
MEM: Quad 3600 Mhz
GPU: AMD Radeon VII Navi 3980

Thanks for reading.

v1.3 Added videos

v1.2 Updated alt tag descriptions

v1.1 Added “I will be moving back to Windows for local development and Linux on servers”

1.0 Initial Draft

No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

August 5, 2018 by Simon

No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Aside

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Now on with the post.

Murphy’s Law

I recently had an issue where I set up a website for a friend. I invested 6 hours into setting up..

I setup…

Debian OS
NGINX Web Server
MySQL Database
PHP 7.2
PHP-FPM Child Workers
HTTPS Certificate
Security (Firewall/Headers/SSH, WordPress, Plugins etc).
Installed WordPress and Plugins
Setup DNSSEC
Etc

I had tested GTMetrix scores = less than 1 second. Security headers were tested and I was happy with the site.

The server and backups were automatically deleted after 7 days while I was away from my keyboard because I assumed the account was valid and had credits.

Lesson Learned

Always have a backup (of the server, setup/ www, MySQL etc).
Script setups (Ansible, Puppet or Scripts) to sate time redeploying if need be.
Backups are not always available.
Do have setup documented (Check)
Do have a disaster plan

I have guides on setting up a server on UpCloud, AWS, Vultr, Digital Ocean but setting up can be rather repetitive so how can you prevent resetting up servers?

Why Plan for the Worst

Companies disappear.
Some hosts go down.
Some hosts have weird trial modes and internal process that could take your site down.
Human error?
Murphy’s Law

How I will prevent this in future

I am building a Java desktop app for Windows/OSX/Linux app that will deploy and set up on UpCloud/Vultr/Digital Ocean providers and allow for 1 click deploy and backup and restore.
I am going to re-establish replication between servers with RSync etc.
I am going to start to automate installs and environments.
I am going to set up a hot (ready to go) Green/Blue mirrored environments (www and DB server) on different providers in case of emergency. Then I can set the active live servers with DNS (blog posts soon).
Consider a server farm (same provider or different providers)

I hope this guide helps someone.

Please consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

v1.0 Initial Post

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

June 25, 2018 by Simon

Upgrading an Ubuntu server on UpCloud to add more CPU, Memory and Disk Space

If you have not read my previous posts I have now moved my blog from Vultr to the awesome UpCloud host (signup using this link to get $25 free credit).

Recently I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Spoiler: UpCloud performance is great.

Why Upgrade

I have 1 CPU, 1 GB memory and 50GB storage and it is running well? I have PHP child workers set up and have set up the preferred use of memory over swap file usage.

Before UpCloud, when I had 512MB ram on Vultr I had multiple NGINX crashed a day so I used a bash script and scheduled a cron job to clear memory cache when memory fell below 100MB (view the script here). To further increase the speed of the WordPress I have configured the OS to use memory over the disk. About once a day free memory falls below 100MB (this is not a problem as my script clears cached items automatically).

I’d like to add more memory as I am working on some things (watch this space) and I will use the extra memory. I’d prefer the server is set up now for the expected workload.

How to Upgrade

This is how I upgraded from 1xCPU/2GB Memory/50GB Storage/2TB Transferred Data to a 2 CPU/4GB Mmeory/80GB Storage/4TB Transferred Data server.

UpCloud Pricing: https://www.upcloud.com/pricing/

Upgrade an UpCloud VM

I shut down my existing VM. Read this guide to setup a VM.

shutdown -P

Login to the UpCloud dashboard, select your server (confirm the server has shut down) and click General Settings, choose the upgrade and click Update.

I confirmed the upgrade options (2x CPU, 4096 MB Memory).

Click Update

After 10 seconds you can start your server from the UpCloud server.

I confirmed the CPU upgrade was visible in the VM

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
stepping        : 2
microcode       : 0x1
cpu MHz         : 3099.978
cache size      : 16384 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 6199.95
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
stepping        : 2
microcode       : 0x1
cpu MHz         : 3099.978
cache size      : 16384 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 6199.95
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Software Tweaks Post Upgrade.

I added these settings to the top of /etc/nginx/nginx.conf to ensure the extra CPU was used.

worker_processes auto;
worker_cpu_affinity auto;

I increased PHP FPM ( /etc/php/7.2/fpm/php.ini ) to increase the memory usage and child workers. I doubled child workers and max memeory limit.

memory_limit = 3072M
pm.max_children = 80
pm.start_servers = 30
pm.min_spare_servers = 10
pm.max_spare_servers = 30

I restarted NGINX and PHP

nginx -t
nginx -s reload
/etc/init.d/nginx restart
service php7.2-fpm restart

I tweaked WordPress max memory limits

define( 'WP_MEMORY_LIMIT','3072M');
define( 'WP_MAX_MEMORY_LIMIT','3072M');

MySQL Tweaks: I logged into MySQL

mysql -u root -p

I ran “SHOW GLOBAL STATUS” to view stats

mysql> SHOW GLOBAL STATUS;
+-----------------------------------------------+--------------------------------------------------+
| Variable_name                                 | Value                                            |
+-----------------------------------------------+--------------------------------------------------+
| Aborted_clients                               | 0                                                |
| Aborted_connects                              | 0                                                |
| Binlog_cache_disk_use                         | 0                                                |
| Binlog_cache_use                              | 0                                                |
| Binlog_stmt_cache_disk_use                    | 0                                                |
| Binlog_stmt_cache_use                         | 0                                                |
| Bytes_received                                | 3179986                                          |
| Bytes_sent                                    | 223872114                                        |
| Com_admin_commands                            | 0                                                |
| Com_assign_to_keycache                        | 0                                                |
| Com_alter_db                                  | 0                                                |
| Com_alter_db_upgrade                          | 0                                                |
| Com_alter_event                               | 0                                                |
| Com_alter_function                            | 0                                                |
| Com_alter_instance                            | 0                                                |
| Com_alter_procedure                           | 0                                                |
| Com_alter_server                              | 0                                                |
| Com_alter_table                               | 0                                                |
| Com_alter_tablespace                          | 0                                                |
| Com_alter_user                                | 0                                                |
| Com_analyze                                   | 0                                                |
| Com_begin                                     | 0                                                |
| Com_binlog                                    | 0                                                |
| Com_call_procedure                            | 0                                                |
| Com_change_db                                 | 284                                              |
| Com_change_master                             | 0                                                |
| Com_change_repl_filter                        | 0                                                |
| Com_check                                     | 0                                                |
| Com_checksum                                  | 0                                                |
| Com_commit                                    | 0                                                |
| Com_create_db                                 | 0                                                |
| Com_create_event                              | 0                                                |
| Com_create_function                           | 0                                                |
| Com_create_index                              | 0                                                |
| Com_create_procedure                          | 0                                                |
| Com_create_server                             | 0                                                |
| Com_create_table                              | 0                                                |
| Com_create_trigger                            | 0                                                |
| Com_create_udf                                | 0                                                |
| Com_create_user                               | 0                                                |
| Com_create_view                               | 0                                                |
| Com_dealloc_sql                               | 0                                                |
| Com_delete                                    | 18                                               |
| Com_delete_multi                              | 0                                                |
| Com_do                                        | 0                                                |
| Com_drop_db                                   | 0                                                |
| Com_drop_event                                | 0                                                |
| Com_drop_function                             | 0                                                |
| Com_drop_index                                | 0                                                |
| Com_drop_procedure                            | 0                                                |
| Com_drop_server                               | 0                                                |
| Com_drop_table                                | 0                                                |
| Com_drop_trigger                              | 0                                                |
| Com_drop_user                                 | 0                                                |
| Com_drop_view                                 | 0                                                |
| Com_empty_query                               | 0                                                |
| Com_execute_sql                               | 0                                                |
| Com_explain_other                             | 0                                                |
| Com_flush                                     | 0                                                |
| Com_get_diagnostics                           | 0                                                |
| Com_grant                                     | 0                                                |
| Com_ha_close                                  | 0                                                |
| Com_ha_open                                   | 0                                                |
| Com_ha_read                                   | 0                                                |
| Com_help                                      | 0                                                |
| Com_insert                                    | 342                                              |
| Com_insert_select                             | 0                                                |
| Com_install_plugin                            | 0                                                |
| Com_kill                                      | 0                                                |
| Com_load                                      | 0                                                |
| Com_lock_tables                               | 0                                                |
| Com_optimize                                  | 0                                                |
| Com_preload_keys                              | 0                                                |
| Com_prepare_sql                               | 0                                                |
| Com_purge                                     | 0                                                |
| Com_purge_before_date                         | 0                                                |
| Com_release_savepoint                         | 0                                                |
| Com_rename_table                              | 0                                                |
| Com_rename_user                               | 0                                                |
| Com_repair                                    | 0                                                |
| Com_replace                                   | 0                                                |
| Com_replace_select                            | 0                                                |
| Com_reset                                     | 0                                                |
| Com_resignal                                  | 0                                                |
| Com_revoke                                    | 0                                                |
| Com_revoke_all                                | 0                                                |
| Com_rollback                                  | 0                                                |
| Com_rollback_to_savepoint                     | 0                                                |
| Com_savepoint                                 | 0                                                |
| Com_select                                    | 16358                                            |
| Com_set_option                                | 849                                              |
| Com_signal                                    | 0                                                |
| Com_show_binlog_events                        | 0                                                |
| Com_show_binlogs                              | 0                                                |
| Com_show_charsets                             | 0                                                |
| Com_show_collations                           | 0                                                |
| Com_show_create_db                            | 0                                                |
| Com_show_create_event                         | 0                                                |
| Com_show_create_func                          | 0                                                |
| Com_show_create_proc                          | 0                                                |
| Com_show_create_table                         | 0                                                |
| Com_show_create_trigger                       | 0                                                |
| Com_show_databases                            | 3                                                |
| Com_show_engine_logs                          | 0                                                |
| Com_show_engine_mutex                         | 0                                                |
| Com_show_engine_status                        | 0                                                |
| Com_show_events                               | 0                                                |
| Com_show_errors                               | 0                                                |
| Com_show_fields                               | 921                                              |
| Com_show_function_code                        | 0                                                |
| Com_show_function_status                      | 0                                                |
| Com_show_grants                               | 0                                                |
| Com_show_keys                                 | 1                                                |
| Com_show_master_status                        | 0                                                |
| Com_show_open_tables                          | 0                                                |
| Com_show_plugins                              | 0                                                |
| Com_show_privileges                           | 0                                                |
| Com_show_procedure_code                       | 0                                                |
| Com_show_procedure_status                     | 0                                                |
| Com_show_processlist                          | 0                                                |
| Com_show_profile                              | 0                                                |
| Com_show_profiles                             | 0                                                |
| Com_show_relaylog_events                      | 0                                                |
| Com_show_slave_hosts                          | 0                                                |
| Com_show_slave_status                         | 0                                                |
| Com_show_status                               | 6                                                |
| Com_show_storage_engines                      | 0                                                |
| Com_show_table_status                         | 0                                                |
| Com_show_tables                               | 2                                                |
| Com_show_triggers                             | 0                                                |
| Com_show_variables                            | 6                                                |
| Com_show_warnings                             | 1                                                |
| Com_show_create_user                          | 0                                                |
| Com_shutdown                                  | 0                                                |
| Com_slave_start                               | 0                                                |
| Com_slave_stop                                | 0                                                |
| Com_group_replication_start                   | 0                                                |
| Com_group_replication_stop                    | 0                                                |
| Com_stmt_execute                              | 4                                                |
| Com_stmt_close                                | 4                                                |
| Com_stmt_fetch                                | 0                                                |
| Com_stmt_prepare                              | 4                                                |
| Com_stmt_reset                                | 0                                                |
| Com_stmt_send_long_data                       | 4                                                |
| Com_truncate                                  | 2                                                |
| Com_uninstall_plugin                          | 0                                                |
| Com_unlock_tables                             | 0                                                |
| Com_update                                    | 70                                               |
| Com_update_multi                              | 0                                                |
| Com_xa_commit                                 | 0                                                |
| Com_xa_end                                    | 0                                                |
| Com_xa_prepare                                | 0                                                |
| Com_xa_recover                                | 0                                                |
| Com_xa_rollback                               | 0                                                |
| Com_xa_start                                  | 0                                                |
| Com_stmt_reprepare                            | 0                                                |
| Connection_errors_accept                      | 0                                                |
| Connection_errors_internal                    | 0                                                |
| Connection_errors_max_connections             | 0                                                |
| Connection_errors_peer_address                | 0                                                |
| Connection_errors_select                      | 0                                                |
| Connection_errors_tcpwrap                     | 0                                                |
| Connections                                   | 292                                              |
| Created_tmp_disk_tables                       | 1124                                             |
| Created_tmp_files                             | 30                                               |
| Created_tmp_tables                            | 1369                                             |
| Delayed_errors                                | 0                                                |
| Delayed_insert_threads                        | 0                                                |
| Delayed_writes                                | 0                                                |
| Flush_commands                                | 1                                                |
| Handler_commit                                | 6094                                             |
| Handler_delete                                | 33                                               |
| Handler_discover                              | 0                                                |
| Handler_external_lock                         | 38571                                            |
| Handler_mrr_init                              | 0                                                |
| Handler_prepare                               | 0                                                |
| Handler_read_first                            | 2299                                             |
| Handler_read_key                              | 134761                                           |
| Handler_read_last                             | 237                                              |
| Handler_read_next                             | 310119                                           |
| Handler_read_prev                             | 2733                                             |
| Handler_read_rnd                              | 222350                                           |
| Handler_read_rnd_next                         | 472820                                           |
| Handler_rollback                              | 0                                                |
| Handler_savepoint                             | 0                                                |
| Handler_savepoint_rollback                    | 0                                                |
| Handler_update                                | 15605                                            |
| Handler_write                                 | 17310                                            |
| Innodb_buffer_pool_dump_status                | Dumping of buffer pool not started               |
| Innodb_buffer_pool_load_status                | Buffer pool(s) load completed at 180624 23:38:01 |
| Innodb_buffer_pool_resize_status              |                                                  |
| Innodb_buffer_pool_pages_data                 | 1035                                             |
| Innodb_buffer_pool_bytes_data                 | 16957440                                         |
| Innodb_buffer_pool_pages_dirty                | 0                                                |
| Innodb_buffer_pool_bytes_dirty                | 0                                                |
| Innodb_buffer_pool_pages_flushed              | 1936                                             |
| Innodb_buffer_pool_pages_free                 | 7144                                             |
| Innodb_buffer_pool_pages_misc                 | 13                                               |
| Innodb_buffer_pool_pages_total                | 8192                                             |
| Innodb_buffer_pool_read_ahead_rnd             | 0                                                |
| Innodb_buffer_pool_read_ahead                 | 0                                                |
| Innodb_buffer_pool_read_ahead_evicted         | 0                                                |
| Innodb_buffer_pool_read_requests              | 306665                                           |
| Innodb_buffer_pool_reads                      | 950                                              |
| Innodb_buffer_pool_wait_free                  | 0                                                |
| Innodb_buffer_pool_write_requests             | 26509                                            |
| Innodb_data_fsyncs                            | 1229                                             |
| Innodb_data_pending_fsyncs                    | 0                                                |
| Innodb_data_pending_reads                     | 0                                                |
| Innodb_data_pending_writes                    | 0                                                |
| Innodb_data_read                              | 16273920                                         |
| Innodb_data_reads                             | 1078                                             |
| Innodb_data_writes                            | 2857                                             |
| Innodb_data_written                           | 53379584                                         |
| Innodb_dblwr_pages_written                    | 1275                                             |
| Innodb_dblwr_writes                           | 109                                              |
| Innodb_log_waits                              | 0                                                |
| Innodb_log_write_requests                     | 450                                              |
| Innodb_log_writes                             | 585                                              |
| Innodb_os_log_fsyncs                          | 793                                              |
| Innodb_os_log_pending_fsyncs                  | 0                                                |
| Innodb_os_log_pending_writes                  | 0                                                |
| Innodb_os_log_written                         | 664064                                           |
| Innodb_page_size                              | 16384                                            |
| Innodb_pages_created                          | 56                                               |
| Innodb_pages_read                             | 988                                              |
| Innodb_pages_written                          | 1936                                             |
| Innodb_row_lock_current_waits                 | 0                                                |
| Innodb_row_lock_time                          | 0                                                |
| Innodb_row_lock_time_avg                      | 0                                                |
| Innodb_row_lock_time_max                      | 0                                                |
| Innodb_row_lock_waits                         | 0                                                |
| Innodb_rows_deleted                           | 2                                                |
| Innodb_rows_inserted                          | 19219                                            |
| Innodb_rows_read                              | 249102                                           |
| Innodb_rows_updated                           | 77                                               |
| Innodb_num_open_files                         | 81                                               |
| Innodb_truncated_status_writes                | 0                                                |
| Innodb_available_undo_logs                    | 128                                              |
| Key_blocks_not_flushed                        | 0                                                |
| Key_blocks_unused                             | 12751                                            |
| Key_blocks_used                               | 645                                              |
| Key_read_requests                             | 321877                                           |
| Key_reads                                     | 648                                              |
| Key_write_requests                            | 196                                              |
| Key_writes                                    | 150                                              |
| Locked_connects                               | 0                                                |
| Max_execution_time_exceeded                   | 0                                                |
| Max_execution_time_set                        | 0                                                |
| Max_execution_time_set_failed                 | 0                                                |
| Max_used_connections                          | 3                                                |
| Max_used_connections_time                     | 2018-06-24 23:43:48                              |
| Not_flushed_delayed_rows                      | 0                                                |
| Ongoing_anonymous_transaction_count           | 0                                                |
| Open_files                                    | 229                                              |
| Open_streams                                  | 0                                                |
| Open_table_definitions                        | 206                                              |
| Open_tables                                   | 786                                              |
| Opened_files                                  | 502                                              |
| Opened_table_definitions                      | 208                                              |
| Opened_tables                                 | 817                                              |
| Performance_schema_accounts_lost              | 0                                                |
| Performance_schema_cond_classes_lost          | 0                                                |
| Performance_schema_cond_instances_lost        | 0                                                |
| Performance_schema_digest_lost                | 0                                                |
| Performance_schema_file_classes_lost          | 0                                                |
| Performance_schema_file_handles_lost          | 0                                                |
| Performance_schema_file_instances_lost        | 0                                                |
| Performance_schema_hosts_lost                 | 0                                                |
| Performance_schema_index_stat_lost            | 0                                                |
| Performance_schema_locker_lost                | 0                                                |
| Performance_schema_memory_classes_lost        | 0                                                |
| Performance_schema_metadata_lock_lost         | 0                                                |
| Performance_schema_mutex_classes_lost         | 0                                                |
| Performance_schema_mutex_instances_lost       | 0                                                |
| Performance_schema_nested_statement_lost      | 0                                                |
| Performance_schema_prepared_statements_lost   | 0                                                |
| Performance_schema_program_lost               | 0                                                |
| Performance_schema_rwlock_classes_lost        | 0                                                |
| Performance_schema_rwlock_instances_lost      | 0                                                |
| Performance_schema_session_connect_attrs_lost | 0                                                |
| Performance_schema_socket_classes_lost        | 0                                                |
| Performance_schema_socket_instances_lost      | 0                                                |
| Performance_schema_stage_classes_lost         | 0                                                |
| Performance_schema_statement_classes_lost     | 0                                                |
| Performance_schema_table_handles_lost         | 0                                                |
| Performance_schema_table_instances_lost       | 0                                                |
| Performance_schema_table_lock_stat_lost       | 0                                                |
| Performance_schema_thread_classes_lost        | 0                                                |
| Performance_schema_thread_instances_lost      | 0                                                |
| Performance_schema_users_lost                 | 0                                                |
| Prepared_stmt_count                           | 0                                                |
| Qcache_free_blocks                            | 1                                                |
| Qcache_free_memory                            | 16760152                                         |
| Qcache_hits                                   | 0                                                |
| Qcache_inserts                                | 0                                                |
| Qcache_lowmem_prunes                          | 0                                                |
| Qcache_not_cached                             | 16355                                            |
| Qcache_queries_in_cache                       | 0                                                |
| Qcache_total_blocks                           | 1                                                |
| Queries                                       | 19164                                            |
| Questions                                     | 19155                                            |
| Select_full_join                              | 0                                                |
| Select_full_range_join                        | 0                                                |
| Select_range                                  | 2677                                             |
| Select_range_check                            | 0                                                |
| Select_scan                                   | 2098                                             |
| Slave_open_temp_tables                        | 0                                                |
| Slow_launch_threads                           | 0                                                |
| Slow_queries                                  | 0                                                |
| Sort_merge_passes                             | 12                                               |
| Sort_range                                    | 4859                                             |
| Sort_rows                                     | 244452                                           |
| Sort_scan                                     | 854                                              |
| Ssl_accept_renegotiates                       | 0                                                |
| Ssl_accepts                                   | 0                                                |
| Ssl_callback_cache_hits                       | 0                                                |
| Ssl_cipher                                    |                                                  |
| Ssl_cipher_list                               |                                                  |
| Ssl_client_connects                           | 0                                                |
| Ssl_connect_renegotiates                      | 0                                                |
| Ssl_ctx_verify_depth                          | 0                                                |
| Ssl_ctx_verify_mode                           | 0                                                |
| Ssl_default_timeout                           | 0                                                |
| Ssl_finished_accepts                          | 0                                                |
| Ssl_finished_connects                         | 0                                                |
| Ssl_server_not_after                          |                                                  |
| Ssl_server_not_before                         |                                                  |
| Ssl_session_cache_hits                        | 0                                                |
| Ssl_session_cache_misses                      | 0                                                |
| Ssl_session_cache_mode                        | NONE                                             |
| Ssl_session_cache_overflows                   | 0                                                |
| Ssl_session_cache_size                        | 0                                                |
| Ssl_session_cache_timeouts                    | 0                                                |
| Ssl_sessions_reused                           | 0                                                |
| Ssl_used_session_cache_entries                | 0                                                |
| Ssl_verify_depth                              | 0                                                |
| Ssl_verify_mode                               | 0                                                |
| Ssl_version                                   |                                                  |
| Table_locks_immediate                         | 11962                                            |
| Table_locks_waited                            | 0                                                |
| Table_open_cache_hits                         | 19395                                            |
| Table_open_cache_misses                       | 817                                              |
| Table_open_cache_overflows                    | 12                                               |
| Tc_log_max_pages_used                         | 0                                                |
| Tc_log_page_size                              | 0                                                |
| Tc_log_page_waits                             | 0                                                |
| Threads_cached                                | 2                                                |
| Threads_connected                             | 1                                                |
| Threads_created                               | 3                                                |
| Threads_running                               | 1                                                |
| Uptime                                        | 2944                                             |
| Uptime_since_flush_status                     | 2944                                             |
+-----------------------------------------------+--------------------------------------------------+
353 rows in set (0.00 sec)

Read more on SHOW GLOBAL STATUS here. Read more on the values here.

I can see NO major errors here (possibly due to UpClouds awesome disk IO) so I won’t be making memory tweaks in MySQL. Sign Up using this link and get $25 credit free on UpCloud and see for yourself how fast they are.

Configure Ubuntu System Memory Usage

Edit /etc/sysctl.conf

Add the following to allow things to sit in ram longer

vm.vfs_cache_pressure=50

Snip from: https://www.kernel.org/doc/Documentation/sysctl/vm.txt

This percentage value controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.

At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim.  Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will never reclaim dentries and inodes due to memory pressure and this can easily lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

Increasing vfs_cache_pressure significantly beyond 100 may have negative performance impact. Reclaim code needs to take various locks to find freeable directory and inode objects. With vfs_cache_pressure=1000, it will look for ten times more freeable objects than there are."

Read these pages here and here regarding setting MySQL memory.

Reboot

shutdown -r now

Resize the disk

The 2x CPU, 4GB memory plan comes with 80GB storage allowance. My disk at present is 50GB and I will update the size soon following this guide soon.

Quick Benchmark

I used loader.io to load 500 users to access my site in 1 minute.

The benchmark worked with no errors.

This benchmark was performed with no Cloudflare caching. I should get Cloudflare caching working again to lower the average response time. I loaded my website manually in Google Chrome while loader.io threw 500 users at my site and it loaded very fast.

Conclusion

After a few days, I checked my memory logs and there were no low memory triggers (just normal internal memory management triggers). Ubuntu was happier.

This graph was taken before I set “vm.vfs_cache_pressure” so I will update this graph in a few days.

I hope this guide helps someone.

Please consider using my referral code and get $25 credit for free.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

1.0 Initial Post

How to use the UpCloud API to manage your UpCloud servers

June 17, 2018 by Simon

How to use the UpCloud API to manage your UpCloud servers.

If you have not read my previous posts I have now moved my blog etc to the awesome UpCloud host. Sign up using this link to get $25 free credit.

I recently compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here).

Here is my blog post on moving from Vultr to UpCloud.

Spoiler: UpCloud performance is great.

I have never had an UpCloud page load take longer than 2 seconds since moving.

UpCloud API

UpCloud has an API that we can opt into to using where we can manage servers. Read the official UpCloud API documentation here.

The API allows you to control:

Accounts
Pricing
Zones
Timezones
Plans
Servers
Storages
IP-Addresses
Firewall
Tags
etc

Create a sub-account to query the API

You should create a new user account (in the UpCloud dasbboard) just for API access. I created two accounts for use on my server and on my home laptop and my server (and set a limiting IP(s) that can access it).

Create a Sub Account for API Access

Click My Accounts,
Click User Accounts,
Click Change on your user and enable API connections.
TIP: Set up an IP rule to limit access to your API for security (I set up a VPN to get a static IP on my dynamic IP Internet host at home)).
Save the changes

TIP: Lock down the account to have the minimum permissions required.

e.g

Disable access to the control panel (Untick).
Allow API Connections (Tick) and specify an IP
Disable access to billing contact (Untick).
Disable access to billing section in the control panel (Untick).
Disable allowing of emails to billing contact (Untick).
Allow or Remove access to all server (or manually add access to desired servers)
Allow or Remove access to modify storage (or manually allow or remove access to desired storage)
etc

Save the account.

Now let’s make our first API call

I use OSX and I use the awesome Paw API testing tool from https://paw.cloud (This is not a plug, they are awesome). Postman is a popular API testing tool too. Any good programing language or CLI will allow you to send API requests.

First, let’s prepare the authorization string (this is a Base64 encoded combination of your username and password) read more here.

Head over to https://www.base64encode.org/
Click the Encode tab
Add your “username:password” (without the quotes).
Click Encode

A Base64 string will be outputted 🙂

e.g > eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk

fyi

You can encode also Encode and Decode Base64 from the Ubuntu Command line

Encode Base64 from the CLI Sample

echo -n 'yourapiusername:yoursupersecurepassword' | base64
eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk

Decode Base64 from the CLI Sample

echo `echo eW91cmFwaXVzZXJuYW1lOnlvdXJzdXBlcnNlY3VyZXBhc3N3b3Jk | base64 --decode`
yourapiusername:yoursupersecurepassword

Now we can add an “Authorization Basic” token to the API request in Paw.

A quick test of the UpCloud Prices API endpoint https://api.upcloud.com/1.2/price reveals the API is working.

I can now see a full breakdown of my service prices in JSON 🙂

Query My Account

OK, Let’s see how much credit I have left by querying the https://api.upcloud.com/1.2/account, I duplicated the item in Paw and changed the URL to https://api.upcloud.com/1.2/account but no data returned?

I had to enable “Access to Billing section in Control Panel” for the user before this data returned from the API (make sense).

> HTTP/1.1 200 OK

Query (GET)

GET /1.2/account HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic *******************************************

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:23:32 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 91
Server: Apache

{
   "account" : {
      "credits" : 2500.00,
      "username" : "yourapiusername"
   }
}

“2500.00” = cents ($25)

Query All of Your Servers

Ok, Let’s get server information by querying https://api.upcloud.com/1.2/server

Query (GET)

GET /1.2/server HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:32:22 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 1154
Server: Apache

{
   "servers" : {
      "server" : [
         {
            "core_number" : "1",
            "hostname" : "server1nameredacted.com",
            "license" : 0,
            "memory_amount" : "2048",
            "plan" : "1xCPU-2GB",
            "plan_ipv4_bytes" : "3472464313",
            "plan_ipv6_bytes" : "166293599",
            "state" : "started",
            "tags" : {
               "tag" : [
                  "tag1"
               ]
            },
            "title" : "server1nameredacted.com",
            "uuid" : "########-####-####-####-############",
            "zone" : "us-chi1"
         },
         {
            "core_number" : "1",
            "hostname" : "server2nameredacted.com",
            "license" : 0,
            "memory_amount" : "1024",
            "plan" : "1xCPU-1GB",
            "plan_ipv4_bytes" : "198911",
            "plan_ipv6_bytes" : "19742",
            "state" : "started",
            "tags" : {
               "tag" : [
                  "tag2"
               ]
            },
            "title" : "server1nameredacted.com",
            "uuid" : "########-####-####-####-############",
            "zone" : "us-chi1"
         }
      ]
   }
}

Query Server Information

I have redated the UUID’s for my servers but once you know them you can query them by hitting https://api.upcloud.com/1.2/server/########-####-####-####-############

Query (GET)

GET /1.2/server/########-####-####-####-############ HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:45:14 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 1656
Server: Apache

{
   "server" : {
      "boot_order" : "cdrom,disk",
      "core_number" : "1",
      "firewall" : "on",
      "host" : redacted,
      "hostname" : "server1nameredacted.com",
      "ip_addresses" : {
         "ip_address" : [
            {
               "access" : "private",
               "address" : "##.#.#.###",
               "family" : "IPv4"
            },
            {
               "access" : "public",
               "address" : "###.###.###.###",
               "family" : "IPv4",
               "part_of_plan" : "yes"
            },
            {
               "access" : "public",
               "address" : "####:####:####:####:####:####:########",
               "family" : "IPv6"
            }
         ]
      },
      "license" : 0,
      "memory_amount" : "2048",
      "nic_model" : "virtio",
      "plan" : "1xCPU-2GB",
      "plan_ipv4_bytes" : "3519033266",
      "plan_ipv6_bytes" : "168200052",
      "state" : "started",
      "storage_devices" : {
         "storage_device" : [
            {
               "address" : "virtio:0",
               "boot_disk" : "0",
               "part_of_plan" : "yes",
               "storage" : "########-####-####-####-############",
               "storage_size" : 50,
               "storage_title" : "system",
               "type" : "disk"
            }
         ]
      },
      "tags" : {
         "tag" : [
            "fearby"
         ]
      },
      "timezone" : "Australia/Sydney",
      "title" : "server1nameredacted.com",
      "uuid" : "########-####-####-####-############",
      "video_model" : "cirrus",
      "vnc" : "off",
      "vnc_password" : "#########################",
      "zone" : "us-chi1"
   }
}

The servers name, IPv4 and IPV6 network adapters are listed, CPU(s), Memory, Disk Sized and UUID’s are all visible 🙂

Surprisingly the VNC password is visible (enabling access to the root console).

TIP: Ensure your API account is safe and secure.

Query Storage Information

Now, Let’s query the storage with the GUID from above by querying https://api.upcloud.com/1.2/storage/########-####-####-####-############

Query (GET)

GET /1.2/storage/########-####-####-####-############ HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic  ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 04:53:36 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 559
Server: Apache

{
   "storage" : {
      "access" : "private",
      "backup_rule" : {},
      "backups" : {
         "backup" : [
            "########-####-####-####-############"
         ]
      },
      "license" : 0,
      "part_of_plan" : "yes",
      "servers" : {
         "server" : [
            "########-####-####-####-############"
         ]
      },
      "size" : 50,
      "state" : "online",
      "tier" : "maxiops",
      "title" : "system",
      "type" : "normal",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

I can see information about the storage’s assigned server and backups 🙂

Query Backup Information

Backup storage can be queried with the same storge API endpoint https://api.upcloud.com/1.2/storage/########-####-####-####-############

Query (GET)

GET /1.2/storage/014fd483-ea90-4055-b445-bf2011951999 HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Authorization: Basic ##############base64hash##############

Output

HTTP/1.1 200 OK
Date: Sun, 17 Jun 2018 05:01:11 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 412
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-16T04:47:56Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "online",
      "title" : "On-Demand Backup",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

Rename Backup

One thing that I would like to be able to do is to rename on-demand backups in the UpCloud dashboard (this is not a feature yet) but I can rename manual backup’s in the API though 🙂

Boring “On-Demand Backup” label.

I tried sending JSON to https://api.upcloud.com/1.2/storage/########-####-####-####-############ to rename a backup but kept getting an error?

JSON

{
> “storage”: {
> “title”: “Latest manual backup , Working NGINX, PHP, MySQL w Tweaks”,
> “size”: “50”
> }
> }

Result

> “error_code” : “CONTENT_TYPE_INVALID”,
> “error_message” : “The Content-Type header has an invalid value.”

I googled and found an old manual for UpClouds API (official support here).

I added these missing content type headers (108 was the length in chars of the payload)

> Content-Type: application/json; Charset=UTF-8'
> Content-Length: 108

Still no go?

I think the content length value is wrong, more here.

I fixed it, it turned out I had a semicolon in the Content-Type value. The JSON RFC always assumes that Content-Type is UTF8 encoded (more here).

This Fails

Content-Type: application/json; charset=utf-8

This Works

Content-Type: application/json

Now I can rename my Backup (storage). I manually calculated the length of the JSON payload and added a “Content-Length” header and value.

Query (PUT)

PUT /1.2/storage/########-####-####-####-############ HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Content-Type: application/json
Content-Length: 113
Authorization: Basic ##############base64hash##############

{"storage":{"size":"50","title":"Latest manual backup , Working NGINX, PHP, MySQL w Tweaks"}}

Output

HTTP/1.1 202 ACCEPTED
Date: Sun, 17 Jun 2018 05:47:02 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 453
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-16T04:47:56Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "online",
      "title" : "Latest manual backup , Working NGINX, PHP, MySQL w Tweaks",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

Success 🙂

Create a Backup

Backups can be performed with a “/backup” added to the end of the query string.

Query (POST)

POST /1.2/storage/########-####-####-####-############/backup HTTP/1.1
Host: api.upcloud.com
User-Agent: Paw/3.1.7 (Macintosh; OS X/10.13.5) NSURLConnection/1452.23
Content-Type: application/json
Content-Length: 100
Authorization: Basic ##############base64hash##############

{
  "storage": {
    "title": "Sunday 17th Latest backup , Working NGINX, PHP, MySQL w Tweaks"
  }
}

Output

HTTP/1.1 201 CREATED
Date: Sun, 17 Jun 2018 06:17:35 GMT
Content-Type: application/json; charset=UTF-8
Connection: close
Content-Length: 487
Server: Apache

{
   "storage" : {
      "access" : "private",
      "created" : "2018-06-17T06:17:35Z",
      "license" : 0,
      "origin" : "########-####-####-####-############",
      "progress" : "0",
      "servers" : {
         "server" : []
      },
      "size" : 50,
      "state" : "maintenance",
      "title" : "Sunday 17th Latest backup , Working NGINX, PHP, MySQL w Tweaks",
      "type" : "backup",
      "uuid" : "########-####-####-####-############",
      "zone" : "us-chi1"
   }
}

Success (UpCloud GUI)

Conclusion

UpCloud does have great API docs.

I can easily integrate this into bash scripts to manage my servers via API and a future Java app for managing servers.

Paw does give CURL output to allow me to copy working API’s for use in BASH 🙂

More to come

BASH Script to Deploy and configure a server on UpCloud via Initalization scripts (or manual) (1 week)
JAVA App to manage your server (3 months)

If you are signing up for UpCloud please consider using my referral code and get $25 credit for free.

Read my setup guide here.

https://www.upcloud.com/register/?promo=D84793

I hope this guide helps someone.

Ask a question or recommend an article

[contact-form-7 404 "Not Found"]

Revision History

V1.1 updated typo

v1.0 Initial Post.

Backup

SUMo to the Rescue

SUMo Trial Limitations

SUMo Pro Scan

SUMo Settings

Getting Application Updates

Updating Software

Checking Application Update Status

SUMo Right Click Actions

After a few hours updating apps

All Apps Updated

Export

Conclusion

Saving a dollar

Starting Acronis True Image 2020

Creating a Backup

Backup Destination

File Musical Chair’s

Now that I have 800GB free I can backup up my C:\ Disk

Backup in Progress

Creating Rescue (Recovery) DVD Media?

Creating Rescue (Recovery) USB Media?

Restoring a backup from the DVD Rescue Media

Restoring a backup from the USB Rescue Media

Backup Protection

Backup Schedules

Backblaze

Other Features

Active Protection

Cloud Backup

Archive large or old files

Cloud Sync

Other Tools

Acronis True Image (Conclusion)

Pros

Cons

Disclaimer

Troubleshooting

DVD Rescue Media

Bonus: Windows System Restore

Create a System Restore Image

Restoring a restore point

Bonus: Clean Windows 10 Install

Bonus: How I prepare all files needed for a clean install

Finding Installed Software with SUMo

Links

Creating a Back Blaze Account

Choose a Backblaze Plan

Installing the Backblaze Software

Backblaze Preferences

Backblaze Portal

Initial Backup time

Restore Files from Backblaze

Other Links

Backing up the official way (with iTunes)

I don’t want ot pay Apple for more iCloud storage

Install iTunes on Windows (10)

How to backup just photos and videos from your iPhone.

Backup your iPhone backups to the cloud

How to backup and Android Phone

Other Links

Why backup and restore

Murphy’s Law

Backup (the easiest way) with Adminer

Restore (the easiest way) with Adminer

Bonus methods.

Pre-Requisites/Assumptions

Background

PHPStorm IDE?

Install PHPStorm

System requirements

Pricing

Creating a Project

Creating your first PHP file

Creating a Deploy Target

Deploying code to the cloud

Sourc Control

Shell Command

Linux Client

Troubleshooting