• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Home
  • Create a VM ($25 Credit)
  • Buy a Domain
  • 1 Month free Back Blaze Backup
  • Other Deals
    • Domain Email
    • Nixstats Server Monitoring
    • ewww.io Auto WordPress Image Resizing and Acceleration
  • About
  • Links

IoT, Code, Security, Server Stuff etc

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

  • Cloud
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to buy a new domain and SSL cert from NameCheap, a Server from Digital Ocean and configure it.
    • Setting up a Vultr VM and configuring it
    • All Cloud Articles
  • Dev
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • How to setup pooled MySQL connections in Node JS that don’t disconnect
    • NodeJS code to handle App logins via API (using MySQL connection pools (1000 connections) and query parameters)
    • Infographic: So you have an idea for an app
    • All Development Articles
  • MySQL
    • Using the free Adminer GUI for MySQL on your website
    • All MySQL Articles
  • Perf
    • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Measuring VM performance (CPU, Disk, Latency, Concurrent Users etc) on Ubuntu and comparing Vultr, Digital Ocean and UpCloud – Part 1 of 4
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Setting up a website to use Cloudflare on a VM hosted on Vultr and Namecheap
    • All Performance Articles
  • Sec
    • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
    • Using OWASP ZAP GUI to scan your Applications for security issues
    • Setting up the Debian Kali Linux distro to perform penetration testing of your systems
    • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
    • PHP implementation to check a password exposure level with Troy Hunt’s pwnedpasswords API
    • Setting strong SSL cryptographic protocols and ciphers on Ubuntu and NGINX
    • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
    • All Security Articles
  • Server
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All Server Articles
  • Ubuntu
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • Useful Linux Terminal Commands
    • All Ubuntu Articles
  • VM
    • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
    • All VM Articles
  • WordPress
    • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
    • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
    • How to backup WordPress on a host that has CPanel
    • Moving WordPress to a new self managed server away from CPanel
    • Moving a CPanel domain with email to a self managed VPS and Gmail
    • All WordPress Articles
  • All

Apple

How to back up an iPhone (including photos and videos) multiple ways

June 2, 2019 by Simon

This guide is for and anyone who needs to back up their iPhone (complete device along with separate backups of photos and videos).

This is not a paid promo, I don’t get a kickback for mentioning the awesome app below.

Check our my other related posts

  • Backing up files to a Backblaze B2 Cloud Bucket with Duplicati
  • Backing up your computer automatically with BackBlaze software (no data limit)

At the time of writing iTunes v12.9.5.7 was the latest version of iTunes.

iTunes is Apple official software for getting files too and from an iPhone, iPod or iPad. If rumours are right Apple will kill off iTunes software at the end of 2019. If Apple kills off the iTunes software I will update this guide in the future.

Disclaimer: I provide no warranty or support for this app, this is what happened to work for me.

Why write a guide on Backing up a mobile phone?

  • It’s not that simple an I get asked how to do this almost every week by someone.
  • No one thinks to backup up their photos until they fear their phone is lost or stolen.
  • I am about to publish a few posts on backing up (development machines, servers etc) with automated software and with free open source software so this post will be handy to link in what data you can back up in that post.
  • The last time I wrote an article on backups was in March 2016 and its not that great these days.

Golden rules of backing up.

  1. Backup to Three different locations.
  2. Two of the media need to be different.
  3. One of the locations needs to be offsite
  4. Test your backups (a backup is only good if you can restore). (from @Daniel15)
  5. If the backup is encrypted make sure you have the decryption keys. (from @Daniel15)

Backing up the official way (with iTunes)

Apple prefers you just pay them to extend your free 5GB iCloud storage and not worry about backups and not to worry yourself with manual steps.

My wife’s iPhone is always popping up messages saying that her iCloud storage is nearly full and she should upgrade the free space limit.

Screenshot of iCloud setting on iPhone saying your iCloud is nearly full but you have used 5GB or 5GB?

I know she has 100GB+ images and movies on her iPhone but iCloud has only backed up 4GB of photos and videos. Do not trust any backup statuses unless you can verify all files from a restore.

For the record I use an Android phone (Android backup guide coming soon).

iCloud is a good idea for automatic backup but I prefer to see my photos and backup them up myself. Also, Apple iCloud is not free from troubles. If you want Apple to handle backups then I do suggest you upgrade your iCloud storage from the free 5GB to a more sensible 200GB or more.

Screenshot of https://support.apple.com/en-au/HT201318

Here in Australia Apple charge the following for extra storage.

  • 50GB = $1.49 a month
  • 200GB = $4.49 a month
  • 2TB (2,0000GB) = $14.99 a month

A rough estimate: 50GB is enough to store about 12,000 files (9,000 photos + 3,000 videos) *

  • * = based on stats from the backup below (depending on the size an length of your videos).
Screenshot of apple iCloud pricing 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

You can upgrade your paid Apple iCloud backup limit from your iCloud Storage Settings from your iPhone’s settings screen.

Screenshot of iPhone iCloud upgrade button.

IMHO: Apple does make it clear what you are getting before you purchase (but they do not push it enough and people assume their data is safe).

iPhone picture of 50GB = $1.49 a month
200GB = $4.49 a month
2TB (2,0000GB) = $14.99 a month

TIP: Please review your existing iPhone data usage and iCloud usage before it’s too late

I don’t want ot pay Apple for more iCloud storage

I like you, I am tight too but I do pay Google $2.49 a month for 100GB backup storage on my Android phone. Google Drive (Google storage is a bit cheaper than Apple).

Screenshot of Google Drive https://www.google.com/drive/

It is nice knowing your phone is auto backed up.

Install iTunes on Windows (10)

On your Windows 10 computer click the start button. If you don’t have a Microsoft Account go here and create one. Click the start button and type “itunes” (you don’t need to type into a box, just start typing when you click the start button)’.

Then click “iTunes, install app”

Screenshot of a Windows start menu and Get iTunes icon

If you are logged into your Windows 10 store with your Microsoft ID click “Get” (if not you will need to login to the store).

Screenshot of me logged into the Windows Microsoft Store.

After you click ‘Get’, iTunes will start downloading.

Screenshot of the Windows Store downloading iTunes

When iTunes is downloaded Windows will install it. When it’s installed click ‘Launch’

iTunes is installed, Click launch screenshot

You will need to agree to Apple’s terms of service. Click ‘Agree’

Apple terms of service screenshot.

When iTunes opens click “Agree”

Screenshot of Apple iTunes asking if we can agree to share analytics data

Now login to iTunes with your Apple ID (if you don’t have one create one here)’.

Screenshot of iTunes open with an arrow pointing to the Account menu

Click the ‘Account’ then ‘Sign In’ menu.

Screenshot of the account then sign in menu

Login to iTunes with your Apple ID

Screenshot of the apple sign in box.

Optional: If you have Two Factor Authorisation (2FA) of Apple ID’s turned on (you should) you will need to enter a 6 digit code.

Screenshot of an apple 6 digit 2FA code

Apple Two Factor Authorisation (2FA) will send one of your other devices a login code that you will need to use to login (confirm you own the account)

Screenshot of apples 2FA web page at https://support.apple.com/en-au/HT204915

TIP: Check out https://twofactorauth.org/ to see what other sites use Two Factor Authorisation (2FA). I use hardware Yubico YubiKeys to protect logins to WordPress, Linux and websites.

Now back to the article, iTunes should be ready to allow us to backup our iPhone.

Screenshot of me logged into iTunes

Before I continue I will click ‘Edit‘ then ‘Preferences‘ menu to view where iTunes will download media too (different ot backup data), not important but I just want ot see if it is not pointed ot my smaller C drive before I backup my phone

I changed the location to a larger S:\Drive’

Screenshot of Apple setting screen (tab Advanced)

I was greeted with a message asking me to confirm that I wanted to sync the iPhone that I just plugged in called ‘EllieRose’, I clicked ‘Continue’

Confirm access to the iPhone screenshot

On the iPhone I also clicked ‘Trust’ to allow iTunes to talk to it.

iPhone screenshot to allow iTunes to talk to it.

I was then prompted to download a software update to the iPhone, It appears this phone is not running the latest software

Screenshot of a question to update to iOS v 12.3.1

I was prompted to sync the iPhone to the computer

screenshot of a prompt to sync purchases from the iPhone to the computer

A backup of the phone was underway.

Screenshot of iTunes backup progress with a progress bar

If the screen above does not appear click the icon below to view the backup and restore iPhone menu.

Screenshot showing the iPhone Device button in iTunes

The iPhone was backing up to this folder on my computer:

C:\Users\Simon Fearby\Apple\MobileSync\Backup

I was not prompted for a location to back up so and I will move this backup folder after the backup completes (so my C: drive does not fill up).

My drives.

  • C:\ = 500GB drive (faster SSD)
  • S:\ = 2,000GB Drive (slower Magnetic)
Screenshot showing  iTunes backup and  the C:\Users\Simon Fearby\Apple\MobileSync\Backup folder

The iTunes iPhone backup is now complete (it took about 2 hours).

Screenshot of the complete backup

Now that the iPhone was backed up iTunes the Operating System update started to run.

Screenshot of iTunes verifying the iPhone update

Update verified.

Screenshot of iTunes updating the iPhone OS.

During the update the iPhone was unavailable.

Screenshot of iPhone installing firmware

Done, the iPhone had updated it’s operating system in about 30 minutes.

Screenshot of iTunes showing the newer OS version.

Now lets see how much space I have available on my C Drive.

Screenshot of disk usage on my c and S drive.

It looks like the iPhone backup iTunes made was 60GB in size.

Screenshot of Windows reporting the backup was 60GB in size.

Using Windows Explorer I moved the backup from..

C:\Users\Simon Fearby\Apple\MobileSync\Backup

to..

S:\Backup\AlisonsiPhone\iTunes\31May2019

(right click drag an drop action in explorer)

Screenshot of right click drag and drop move folder.

It took me about 20 minutes to move the 60GB folder.

Screenshot of thew Windows copy dialog progress.

TIP: You can (should) also copy this backup folder to..

  • A removable hard drive
  • Removable USB Flash Drive
  • A NAS or SAN Drive
  • etc

Now this backup is available for me to restore in the future if I need it.

In the case of a restore, I just need to move the backup into the iTunes expected location.

Copy from..

S:\Backup\AlisonsiPhone\iTunes\31May2019

..to..

C:\Users\Simon Fearby\Apple\MobileSync\Backup

The Catch

The catch with back-ups made with iTunes is they are useless if you wanted to restore individual files (say to find a photograph or video). iTunes backups are usually tens of thousands of files with random folders and filenames

Screenshot of iTunes nasty backup, i obfuscated all files to random guid filenames

If you want to JUST backup photos and videos read on.

How to backup just photos and videos from your iPhone.

This part of the guide needs a paid version of the iOS App Photo Transfer App. In Australia, the app is free but to turn it into a full version needed to restore everything (via in-app purchase) it cost $10.99.

This app is well worth $10.99 AUD (it may be cheaper in other countries) to have it push photos an videos from your iPhone to a free companion Windows program (a Mac versions exist too).

Screenshot of http://phototransferapp.com/

Buy the iOS version and install it on your iPhone.

Then download the free Windows version: http://phototransferapp.com/win/

Extract the files from the zip file.

I extracted PhotoTransferapp.exe from PhotoTransferapp.zip

When I run PhotoTransferApp.exe I get an error saying I need to install Adobe AIR run times.

Screenshot of me installing Adobe Air run times from https://get.adobe.com/air/

Go to https://get.adobe.com/air/ and download and install the Air run time.

Screenshot of https://get.adobe.com/air/

Install the Adobe Air run time.

Install Adobe Air From https://get.adobe.com/air/

I installed Adobe AIR and again reopened the PhotoTansferApp.exe and was prompted to allow access to the Windows Firewall.

I was on a home network so I enabled firewall access.

Screenshot of Photo Transfer App asking fore firewall access.

The Photo Transfer App prompted me to ‘Discover Devices‘ and to make sure the Transfer app is running on the iPhone.

The Photo Transfer App prompted me to 'Discover Devices' and to make sure the Transfer app is running on the iPhone.

Before I clicked ‘Discover Devices” on the Photo Transfer App on Windows I opened the Transfer app (mentioned above) on the iPhone.

Photo of the Transfer app on the iPhone

On the iPhone I clicked ‘Send‘

Click Send

The transfer app will ask for access to your photos, you will need to press OK.

Screenshot of the transfer app asking for permissions to photos

On the iPhone again I clicked ”Windows‘

On the iPhone again I clicked  ''Windows'

The iOS Transfer app now said I should run the ‘Photo Transfer App‘ on Windows.

The iOS Transfer app now said I should run the 'Photo Transfer App' on Windows.

TIP: You can transfer over WiFi (if your iPhone and Windows device is on the same WiFi Network) or you can transfer over a USB cable.

Screenshot: transfer via WiFi or USB?

Before I started the ‘Detect Device‘ or ‘transfer‘, I set the Backup path location to my S: drive by clicking settings in the bottom right and choosing a folder (as my C Drive is a bit small).

Screenshot of Photo Transfer App choosing a folder to backup to.

I noticed the port of 57777, I temporarily disabled the whole Windows Firewall just in case it prevents the photo backup.

In hindsight, it was not a good idea to disable the whole firewall, but because I was at home on a safe network I felt safe to do so. Next time I will not disable the firewall and see if this still works.

If you are on an internet cafe, school or university network do not disable your firewall.

I clicked ”Start” then typed ‘firewall‘ and clicked ‘Windows Defender Firewall‘

Screenshot Windows Defender firewall icon

I turned off my firewall.

Screenshot Firewall disabled.

I clicked ‘Detect Devices‘ and the iPhone ‘EllieRose‘ appeared on the left. I doubled clicked on the iPhone name and was prompted with an Authorization required message.

Screenshot: Authorisation required message.

I looked at the ‘Transfer‘ app on iOS and clicked ‘Yes, always‘ to allow access.

Screenshot: Authorisation yes/no on the iPhone

After 20 seconds I can see photos on my iPhone on Windows.

Screenshot, all of my iPhone images were appearing on Window.

I selected all camera albums to backup and clicked backup. I prompt to upgrade to the full version will appear, you will need to buy the upgrade.

Full backup happens only if you upgrade to a full version via an in app purchase

After the full version if purchased the backup will be allowed.

You can upgrade from the free to paid full version from the main screen of the Transfer app.

Screenshot get full access by upgrading the free app to a paid version

The upgrade In App Purchase is $10.99 AUD

Screenshot of a $10.99 In App Purchase on the iPhone

When you purchase the In App Purchase you can run the backup again.

Screenshot of all camera categories on the iPhone on windows

Now I could see photos being copied from the iPhone to my defined backup folder.

Screenshot underway.

I went to bed as I knew there were about 100GB of files on the iPhone and this was going to take a while.

In the morning the backup was done.

Screenshot backup done.

26,000 files were backed up (over 100GB).

I now had a full iPhone backup made by iTunes and a copy of all photos and videos.

Screenshot showing the 2 backups (a) iTunes made and b) Photo Transfer App made)

I turned on the Firewall again.

Screenshot: I re enabled the firewall.

I now had 160GB of backed up photos, videos and phone backup.

160GB of files  from both backups.

Backup your iPhone backups to the cloud

The steps to do this two ways will be added soon.

I will add a section on how you can back up the iTunes and manually synced photo and video backups to the cloud automatically and a more complex but cheaper was for 0.005c per GB.

Watch this post.

How to backup and Android Phone

Article coming soon.

Other Links

Read the official iTunes/iCloud backup guide from Apple here: https://support.apple.com/en-au/HT203977

Backing up your computer automatically with BackBlaze software (no data limit)

Versions

v1.3 Added back blaze article link

v1.2 Added more images (from a phone that does not have the iOS app already)

v1.1 Added an Android heading.

v1.0 Initial Post

Filed Under: Apple, Backup, Cloud, Google Tagged With: Apple, Backup, iCloud, iPhone

I am moving away from Apple hardware

February 3, 2019 by Simon

My Late 2012 Mac Book Pro Retina laptop is all but dead, it has many dead pixels and because of the poor cooling and is NOT a joy to use anymore. It does not “JUST WORK” and personally, I do not think “thinner” laptops can handle Australian summers as its hardware cooling it inadequate above 40c air temperatures.

My laptop processor would spend more time thermal throttling (at 104c)  in Web Browsers and text editors that at normal speeds. Opening up productivity apps like Photoshop or Premiere Pro would send the laptop into meltdown.

Image of temperate monitoring showing an overheating macbook when the apple is idle

Frequent high temps were common.

Temp monitoring showing 100+c temps

Attempted Fixes

Warning Disclaimer: My laptop is out of warranty and I know my way around the inside of computer hardware without zapping it. Do not attempt to open your laptop unless you know what you are doing, have backed up your data and are prepared to brick your computer.

  • I removed dust from inside the laptop.
  • I tried to only use the laptop refrigerative air conditioning
  • I replaced the thermal paste on the CPU and GPU (3 times)
  • I reinstalled OSX Mojave and reset the SMC and PRAM multiple times.
  • I ran the fans at 100% (see post here), The fans were operating at full capacity and were not broken.

The stock thermal paste was crusty after 5 years. The plastic CPU/GPU cover was visibly cooked.

Picture of dry stock thermal paste

I ordered some new Thermal grizzly thermal paste, I had some older silicone paste on hand just in case.

Picture of thermal paste options

After many reapplications of the Thermal Grizzly, the older silicone paste seemed to work the better???

Picture of thermal paste applied on a processor

After a few months, all of the fixes above did not seem to work. OSX Mojave would spin up the CPU and GPU into a frenzy overloading the single heat pipe within minutes.

Time to try some more drastic cooling modifications?

I tried improving the efficiency of the single (copper) heat pipe that is shared between an Intel i7 2.6 GHz and an Nvidia Video Card by removing the black paint by stripping the paint with acetone.

Picture of the apple heat pipe in a jar of acetone

I manually removed paint from in between the heat sink fins with a LED to reveal the metal.

Picture of paint being removed from the apple heat pipe fins

I reinstalled the heat pipe with high hopes? That looks nice 🙂Picture of the heat pipe minus paint reinstalled

I removed the old thermal paste and added new paste. First I tried Thermal Grizzly Cryonaut. I re-applied the paste three separate times as each application was not that much better than the old crusty stock paste from Apple. Did I have a bad batch of Thermal Grizzly?, It seemed thick and not very viscous. I ended up using an old tube of silicone paste (the white stuff) as my Arctic Silver was too old to try and I did not want to order more.

More heatpipe post re installation pictures

With the silicone paste applied and the paint removed temperatures were about 15c lower at max, I still had frequent thermal throttling but at least I had a reserve buffer.

This was all before the Aussie Heatwaves and high temperatures soon returned.

Is there still room for improvement?

How heat pipes work

Picture of how heatpipes work

Heat pipes have an evaporating (hot part) and condensing zone (cool part) on the heat pipe. I noticed Apple’s “stock” condensing fins were small, would improving this zone help?. Time to improve the condensers zones by adding larger copper heat sinks to the bare side of the heat pipe.

I purchased a few copper Xeon/Sun server sized heat sinks and thermal epoxied them to the condensing end of the heat pipe. Yes, they would protrude out the bottom of the case but #Meh.  I can fix that by extending the base of the laptop down and making it thicker (old school style).

The server heat sinks arrived

Side on picture of server heatpipes

I cut the heat sinks in half.

Picture of a hacksaw cutting heatsinks

I packed the fins with paper before cutting to ensure the cut did not damage the fins.

Picture of a cut heatsink

After cutting, I wiped the copper heat sinks with vinegar to restore the surface to a nice copper shine.

I tested the heat sink idea with silicone paste first

picture of silicone tested on the heatsink

Temps were 25c lower, Now it’s time to use Arctic Silver Thermal Epoxy 

Picture of two part thermal epoxy

I applied the Thermal Epoxy to the heat pipe (I temporary had foil strips above the fans so I did not block them while the epoxy dried.

Picture of epoxy applied

I then stuck the heat sink’s to the heat pipe (with Arctic Silver Thermal Epoxy).

Heatsinks Thermal Expoxied on

I toyed with a clear case but decided against it for static electricity and stability reasons.

Clear Case pon the botom of the laptop picture

I purchased a second Mac Book base for so I could cut holes for the heat sinks to protrude and use the original base to hide the modification.

Cut holes in the base of the laptop base and purchased a second case bottom

I made a 30 mm base wall so I could use it as a wall between the laptop base and the new 30mm lower base.

picture of the base side wall i made

I added some 5-volt and 12 -volt fans inside the new extended 30mm base.

Collage of base assembly, screws and wires

Finished Product

A normal looking Mac Book except for the 30mm lower base and internal 5V or 12V fans.

Picture of the final mod with 50c lower temps and a 30mm bottom slab under the laptop

External power plugs on the left side, I will add lights at a later stage.

picture of the external power plugs flush with the case on the side of the laptop

Are the temps lower?

50 lower temps screenshot

Videos

Video: Mac Book Pro cooling mod, I can now watch 1080p videos without maxing the CPU

Video: Mac Book Pro cooling mod with external powered 5v or 12v fans

Conclusion

50c lower temps are nicer at idle but in Premiere Pro (exporting video) the laptop was still thermal throttling like mad and temps were terrible (100+). Lets not get started when I start some development VM’s

Conclusion 2 weeks later

This is still not a joy to use. I don’t think I have the right to expect a 5-year-old laptop to keep up running a CPU/GPU intensive OS and applications.

Time to buy a new computer, Apple still makes thin and overheating laptops by the looks of it? 

Maybe I need to buy a fridge to stick a computer in a fridge to use these days?

YouTube users indicate Apple has a problem with heat.

What computer do I get next?

Not an Apple made one. I will be moving back to Windows for local development and Linux on servers

Dell Alienware has many heat pipes.

Picture inside a dell alienware laptop with more heatpies

Acer Predator 500

I read a few reviews (e.g this one from Ultra book reviews) and Acer have good cooling.

picture of Acer Predator cooling and heatpipes

MSI GT Series laptops look the best if cooling is important.

Picture of a MSI GT laptop with 9 heatpipes

Or should I build a custom desktop with way more cores

CPU: Threadripper 2950X 16C 32T 

SSD: M.2 SSD: Samsung 970 PRO 512GB
MOBO: Asus Zenith Extreme
Power: Corsair RM1000x 1000W
MEM: Quad 3600 Mhz 
GPU: AMD Radeon VII Navi 3980

Thanks for reading.

 

v1.3 Added videos

v1.2 Updated alt tag descriptions

v1.1 Added “I will be moving back to Windows for local development and Linux on servers”

1.0 Initial Draft

 

Filed Under: Advice, Apple, Backup, Computer, Copper, Disaster Recovery, Heat, Maintenance, Uncategorized Tagged With: Apple, Heat, Macbook

Is OSX Mojave on a 2014 MacBook Pro slower or faster than High Sierra

October 1, 2018 by Simon

This is a quick post to see if OSX Mojave runs slower on a Mid 2014 Mac Book Pro than High Sierra

Aside

If you have not read my previous posts I have now moved my blog to the awesome UpCloud host (signup using this link to get $25 free UpCloud VM credit). I compared Digital Ocean, Vultr and UpCloud Disk IO here and UpCloud came out on top by a long way (read the blog post here). Here is my blog post on moving from Vultr to UpCloud.

Buy a domain name here

Domain names for just 88 cents!

Now on with the post.

New Mac Operating System (Mojave)

I have always been hesitant before upgrading to a new Apple operating system (or performance-impacting patch).

My Mid-2012 Macbook will not be able to install the next 2019 operating system (as it is now considered too old).

MacBook Thermal Cooling

My MacBook is already running at the limit of the stock thermal cooler (read more here). I replaced the thermal paste on my Mid-2012 Mac Book to help lower thermal temps. I often run fans at 100% with TG Pro.

Stock MacBook thermal paste (needs replacing).

Stock Paste

OSX Mojave

What’s new in OSX Mojave: https://help.apple.com/macOS/mojave/whats-new/

  • Dark Mode
  • Folder Stacks
  • Finder Enhancements
  • Quick Look Enhancements
  • New Screen gran
  • iOS to Mac camera.
  • New News App
  • Stocks App
  • Voice Memos
  • Home Control
  • Better Safari Privacy and Security
  • New Mac App Store
  • Take the tour

I currently have High Sierra Installed.

High Sierra About Screen

High Sierra – Black Magic Disk Speed Test 3.1 Speed Test Results

Write:  340.5 MB/s

Read:  348.1 MB/s

Hig Sierra Disk Benchmark

High Sierra – Novabench 4.01 Benchmark Scores

GPU: 0 (known issue)

RAM: 136

GPU: 243

DISK: 57

High Sierra Nova Bench

Downloading Mojave

Mojave is available for download in the App Store.

Download Mojave

Instaling Mojave

A quick wizard and Mojave in ready to install.

Download Mojave

Installation took about 2 hours to install over High Sierra.

OSX Mojave About Screen

Mojave Dark Mode

Dark mode is certainly very pretty, all stock apps on OSX are not optionally available in dark colour themes.

OSX Mojave dark mode

Mojave – Black Magic Disk Speed Test 3.1 Speed Test Results

Write:  348.5 MB/s (8MB/s faster than High Sierra)

Read:  348.1 MB/s (27.1MB/s faster than High Sierra)

Nice

FYI, The first 2 days of Mojave did seem a bit sower but this may because of background indexing.

My home MacBook has a 512GB Apple SSD hard drive. I recently upgraded to Mojave on a 2014 27 iMac that had a Hybrid SSD (128GB SSD + 1TB drive) and it runs really slowly.

High Sierra – Novabench 4.01 Benchmark Scores

GPU: 0 (known issue)

RAM: 136 (same as High Sierra)

GPU:251 (8 higher than High Sierra)

DISK: 57 (same as High Sierra)

Nova Bench on Mojave

Reboot Time in seconds (time taken to reboot and log back into an interactive desktop)

WOW: Reboot average times were 212 seconds in High Sierra but only 124 seconds in Mojave, that’s an 88-second improvement.

Mojave faster reboots

That totally made upgrading to Mojave worth it.

Screen Capture and save speed

Often I screenshot the desktop (or apps), Below is a time in seconds to capture the desktop and open the file in Photoshop on High Sierra and Mojave.

Capture Desktop Speed

Mojave is a lot faster (even with a wait for the file to be saved to the desktop)

IntelliJ

Does Mojave make IntelliJ slower?

Note: Sorry, the scale in the chart zoomed in by default, I am not sure how to reset the scale on the left to starts at 0.

Time to open IntelliJ

4-second improvement. Nice.

Time to opening Adobe CS Premiere Pro in Mojave v High Sierra

How does Adobe Premiere Pro handle Mojave?

Note: Sorry, the scale in the chart zoomed in by default, I am not sure how to reset the scale on the left to starts at 0.

Mojave Opening Premiere Pro

2 seconds slower (I expect updated from Adobe soon)

More to come soon.

I hope this guide helps someone.

Please consider using my referral code and get $25 UpCloud VM credit if you need to create a server online.

https://www.upcloud.com/register/?promo=D84793

Ask a question or recommend an article

[contact-form-7 id=”30″ title=”Ask a Question”]

Revision History

v1.0 Initial Post

Filed Under: Apple, High Sierra, Mojave, OS, OSX Tagged With: faster, Installing, mojave, or, OSX, slower, speed

Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

July 20, 2018 by Simon

This post will explain why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

Background

I used to work in retail selling computers and I would go to great lengths to open a desktop computer chassis and talk someone out of buying a cheaper/slower computer (usually when it had a Cyrix Media GX processor in it). I would do myself out of higher commission and burn time educating customers. I have blogged about what to look for when buying a computer (here).

2012

In 2012 I bought my first Apple Mac computer to write iOS apps (write your first OSX app). I would call myself an Apple fanboy (previously being a PC fanboy for 15 years). I have never rebuilt my OSX system in 6 years buy would rebuild Windows every 6 months. Some Apple things I like.

2017

My Mid 2012 Mac Book Pro i7 processor overheats like crazy. I have blogged about my Mid 2012 MPB overheating issues (read here). I have even gone and installed third party software to control the speeds of my Mac’s fans (read here).

Inside my Mid 2012 Mac Book Pro (heatsink and fans at the top)

Tiny Mac book pro heatsink

Stupidly thin heatsink (IMHO).

Heatsink is 3mm thick

Complete heatsink (CPU and GPU plate)

MBP Heatsink

I am certain this Mac Book heatsink is too small for the processor and graphics card.

As I type this my Mac Book Pro is Thermal throttling (slowing down the CPU) while typing a blog post (not gaming).

Apple 2012 overheating

My only option is to crank up the fans to 100% and overrise Apple silence first mantra.

TgPro fan speed rules

I am currently sitting here at Winter with my MBP 2012 MBP i7 fans running at 100% to try (try) and prevent thermal throtelling killing my productivity. https://t.co/IM6IlnmjC7

— Simon Fearby (Aussie DevSecOps) (@FearbySoftware) July 18, 2018

Intel Power Gadget showing thermal throttling (CPU dropping t0 almost 1Ghz to drop temps).

Thermal Throtelling

Move forward to 2018

Today I learned that Apple is putting an Intel i9 Procesor into a laptop, great? Hold onto your cash, that thing will run very hot and will never operate at its maximum potential.

Reviews are scathing.

I tweeted..

What a joke, why is @Apple putting an Intel i9 into a stupidly thin Mac Book Pro, my i7 can barely keep cool https://t.co/IM6IlnmjC7
— Simon Fearby (Aussie DevSecOps) (@FearbySoftware) July 13, 2018

Apple’s Website: https://www.apple.com/macbook-pro/

Apples website saying it now has i9 macs

What a waste of a good processor.

Below you will see the fallout on YouTube from Apple putting an i9 Processor in the latest 15″ Mac Book Pros.

Dave Lee posted “MacBook Pro 15 (2018) – Beware the Core i9”

TechLinked posted “2018 Macbook ALREADY Overheating?”

AppleInsider – 2018 MacBook Pro i9 Thermal Throttling CONFIRMED!

Best of all, Louis Rossmann summed up the Apple situation perfectly.

 

 

Update 25th July

Apple is doubling down on the lack of cooling (calling it a “missing digital key”).

I will #BoycottAppleProMachines

That’s all.

Revision History

v1.4 Added update 25th July 2018 Missing Digital Key

v1.3 Gizmodo link

v1.2 Test new db server

v1.1 Added Apple Insider video

v1.0 Initial Post

Filed Under: Apple, Heat Tagged With: a, Apple, buy, cooling, fix, Heat, I, issues, l they, Laptop, missing digital key, never, new, the, unti, Why, will

Securing Ubuntu in the cloud

August 9, 2017 by Simon

It is easy to deploy servers to the cloud within a few minutes, you can have a cloud-based server that you (or others can use). ubuntu has a great guide on setting up basic security issues but what do you need to do.

If you do not secure your server expects it to be hacked into. Below are tips on securing your cloud server.

First, read more on scanning your server with Lynis security scan.

Always use up to date software

Always use update software, malicious users can detect what software you use with sites like shodan.io (or use port scan tools) and then look for weaknesses from well-published lists (e.g WordPress, Windows, MySQL, node, LifeRay, Oracle etc). People can even use Google to search for login pages or sites with passwords in HTML (yes that simple).  Once a system is identified by a malicious user they can send automated bots to break into your site (trying millions of passwords a day) or use tools to bypass existing defences (Security researcher Troy Hunt found out it’s child’s play).

Portscan sites like https://mxtoolbox.com/SuperTool.aspx?action=scan are good for knowing what you have exposed.

You can also use local programs like nmap to view open ports

Instal nmap

sudo apt-get install nmap

Find open ports

nmap -v -sT localhost

Starting Nmap 7.01 ( https://nmap.org ) at 2017-08-08 23:57 AEST
Initiating Connect Scan at 23:57
Scanning localhost (127.0.0.1) [1000 ports]
Discovered open port 80/tcp on 127.0.0.1
Discovered open port 3306/tcp on 127.0.0.1
Discovered open port 22/tcp on 127.0.0.1
Discovered open port 9101/tcp on 127.0.0.1
Discovered open port 9102/tcp on 127.0.0.1
Discovered open port 9103/tcp on 127.0.0.1
Completed Connect Scan at 23:57, 0.05s elapsed (1000 total ports)
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00020s latency).
Not shown: 994 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
3306/tcp open  mysql
9101/tcp open  jetdirect
9102/tcp open  jetdirect
9103/tcp open  jetdirect

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds
           Raw packets sent: 0 (0B) | Rcvd: 0 (0B)

Limit ssh connections

Read more here.

Use ufw to set limits on login attempts

sudo ufw limit ssh comment 'Rate limit hit for openssh server'

Only allow known IP’s access to your valuable ports

sudo ufw allow from 123.123.123.123/32 to any port 22

Delete unwanted firewall rules

sudo ufw status numbered
sudo ufw delete 8

Only allow known IP’s to certain ports

sudo ufw allow from 123.123.123.123 to any port 80/tcp

Also, set outgoing traffic to known active servers and ports

sudo ufw allow out from 123.123.123.123 to any port 22

Don’t use weak/common Diffie-Hellman key for SSL certificates, more information here.

openssl req -new -newkey rsa:4096 -nodes -keyout server.key -out server.csr
 
Generating a 4096 bit RSA private key
...

More info on generating SSL certs here and setting here and setting up Public Key Pinning here.

Intrusion Prevention Software

Do run fail2ban: Guide here https://www.linode.com/docs/security/using-fail2ban-for-security

I use iThemes Security to secure my WordPress and block repeat failed logins from certain IP addresses.

iThemes Security can even lock down your WordPress.

You can set iThemes to auto lock out users on x failed logins

Remember to use allowed whitelists though (it is so easy to lock yourself out of servers).

Passwords

Do have strong passwords and change the root password provided by the hosts. https://howsecureismypassword.net/ is a good site to see how strong your password is from brute force password attempts. https://www.grc.com/passwords.htm is a good site to obtain a strong password.  Do follow Troy Hunt’s blog and twitter account to keep up to date with security issues.

Configure a Firewall Basics

You should install a firewall on your Ubuntu and configure it and also configure a firewall with your hosts (e.g AWS, Vultr, Digital Ocean).

Configure a Firewall on AWS

My AWS server setup guide here. AWS allow you to configure the firewall here in the Amazon Console.

Type Protocol Port Range Source Comment
HTTP TCP 80 0.0.0.0/0 Opens a web server port for later
All ICMP ALL N/A 0.0.0.0/0 Allows you to ping
All traffic ALL All 0.0.0.0/0 Not advisable long term but OK for testing today.
SSH TCP 22 0.0.0.0/0 Not advisable, try and limit this to known IP’s only.
HTTPS TCP 443 0.0.0.0/0 Opens a secure web server port for later

Configure a Firewall on Digital Ocean

Configuring a firewall on Digital Ocean (create a $5/m server here).  You can configure your Digital Ocean droplet firewall by clicking Droplet, Networking then Manage Firewall after logging into Digital Ocean.

Configure a Firewall on Vultr

Configuring a firewall on Vultr (create a $2.5/m server here).

Don’t forget to set IP rules for IPV4 and IPV6, Only set the post you need to allow and ensure applications have strong passwords.

Ubuntu has a firewall built in (documentation).

sudo ufw status

Enable the firewall

sudo ufw enable

Adding common ports

sudo ufw allow ssh/tcp
sudo ufw logging on
sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 53
sudo ufw allow 443
sudo ufw allow 873
sudo ufw enable
sudo ufw status
sudo ufw allow http
sudo ufw allow https

Add a whitelist for your IP (use http://icanhazip.com/ to get your IP) to ensure you won’t get kicked out of your server.

sudo ufw allow from 123.123.123.123/24 to any port 22

More help here.  Here is a  good guide on ufw commands. Info on port numbers here.

https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

If you don’t have a  Digital Ocean server for $5 a month click here and if a $2.5 a month Vultr server here.

Backups

rsync is a good way to copy files to another server or use Bacula

sudo apt install bacula

Basics

Initial server setup guide (Digital Ocean).

Sudo (admin user)

Read this guide on the Linux sudo command (the equivalent if run as administrator on Windows).

Users

List users on an Ubuntu OS (or compgen -u)

cut -d: -f1 /etc/passwd

Common output

cut -d: -f1 /etc/passwd
root
daemon
bin
sys
sync
games
man
lp
mail
news
uucp
proxy
www-data
backup
list
irc
gnats
nobody
systemd-timesync
systemd-network
systemd-resolve
systemd-bus-proxy
syslog
_apt
lxd
messagebus
uuidd
dnsmasq
sshd
pollinate
ntp
mysql
clamav

Add User

sudo adduser new_username

e.g

sudo adduser bob
Adding user `bob' ...
Adding new group `bob' (1000) ...
Adding new user `bob' (1000) with group `bob' ...
Creating home directory `/home/bob' ...
etc..

Add user to a group

sudo usermod -a -G MyGroup bob

Show users in a group

getent group MyGroup | awk -F: '{print $4}'

This will show users in a group

Remove a user

sudo userdel username
sudo rm -r /home/username

Rename user

usermod -l new_username old_username

Change user password

sudo passwd username

Groups

Show all groups

compgen -ug

Common output

compgen -g
root
daemon
bin
sys
adm
tty
disk
lp
mail
proxy
sudo
www-data
backup
irc
etc

You can create your own groups but first, you must be aware of group ids

cat /etc/group

Then you can see your systems groups and ids.

Create a group

groupadd -g 999 MyGroup

Permissions

Read this https://help.ubuntu.com/community/FilePermissions

How to list users on Ubuntu.

Read more on setting permissions here.

Chmod help can be found here.

Install Fail2Ban

I used this guide on installing Fail2Ban.

apt-get install fail2ban

Check Fail2Ban often and add blocks to the firewall of known bad IPs

fail2ban-client status

Best practices

Ubuntu has a guide on basic security setup here.

Startup Processes

It is a good idea to review startup processes from time to time.

sudo apt-get install rcconf
sudo rcconf

Accounts

  • Read up on the concept of least privilege access for apps and services here.
  • Read up on chmod permissions.

Updates

Do update your operating system often.

sudo apt-get update
sudo apt-get upgrade

Minimal software

Only install what software you need

Exploits and Keeping up to date

Do keep up to date with exploits and vulnerabilities

  • Follow 0xDUDE on twitter.
  • Read the GDI.Foundation page.
  • Visit the Exploit Database
  • Vulnerability & Exploit Database
  • Subscribe to the Security Now podcast.

Secure your applications

  • NodeJS: Enable logging in applications you install or develop.

Ban repeat Login attempts with FailBan

Fail2Ban config

sudo nano /etc/fail2ban/jail.conf
[sshd]

enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 3

Hosts File Hardening

sudo nano /etc/host.conf

Add

order bind,hosts
nospoof on

Add a whitelist with your ip on /etc/fail2ban/jail.conf (see this)

[DEFAULT]
# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not                          
# ban a host which matches an address in this list. Several addresses can be                             
# defined using space separator.
                                                                         
ignoreip = 127.0.0.1 192.168.1.0/24 8.8.8.8

Restart the service

sudo service fail2ban restart
sudo service fail2ban status

Intrusion detection (logging) systems

Tripwire will not block or prevent intrusions but it will log and give you a heads up with risks and things of concern

Install Tripwire.

sudo apt-get install tiger tripwire

Running Tripwire

sudo tiger

This will scan your system for issues of note

sudo tiger
Tiger UN*X security checking system
   Developed by Texas A&M University, 1994
   Updated by the Advanced Research Corporation, 1999-2002
   Further updated by Javier Fernandez-Sanguino, 2001-2015
   Contributions by Francisco Manuel Garcia Claramonte, 2009-2010
   Covered by the GNU General Public License (GPL)

Configuring...

Will try to check using config for 'x86_64' running Linux 4.4.0-89-generic...
--CONFIG-- [con005c] Using configuration files for Linux 4.4.0-89-generic. Using
           configuration files for generic Linux 4.
Tiger security scripts *** 3.2.3, 2008.09.10.09.30 ***
20:42> Beginning security report for simon.
20:42> Starting file systems scans in background...
20:42> Checking password files...
20:42> Checking group files...
20:42> Checking user accounts...
20:42> Checking .rhosts files...
20:42> Checking .netrc files...
20:42> Checking ttytab, securetty, and login configuration files...
20:42> Checking PATH settings...
20:42> Checking anonymous ftp setup...
20:42> Checking mail aliases...
20:42> Checking cron entries...
20:42> Checking 'services' configuration...
20:42> Checking NFS export entries...
20:42> Checking permissions and ownership of system files...
--CONFIG-- [con010c] Filesystem 'fuse.lxcfs' used by 'lxcfs' is not recognised as a valid filesystem
20:42> Checking for indications of break-in...
--CONFIG-- [con010c] Filesystem 'fuse.lxcfs' used by 'lxcfs' is not recognised as a valid filesystem
20:42> Performing rootkit checks...
20:42> Performing system specific checks...
20:46> Performing root directory checks...
20:46> Checking for secure backup devices...
20:46> Checking for the presence of log files...
20:46> Checking for the setting of user's umask...
20:46> Checking for listening processes...
20:46> Checking SSHD's configuration...
20:46> Checking the printers control file...
20:46> Checking ftpusers configuration...
20:46> Checking NTP configuration...
20:46> Waiting for filesystems scans to complete...
20:46> Filesystems scans completed...
20:46> Performing check of embedded pathnames...
20:47> Security report completed for simon.
Security report is in `/var/log/tiger/security.report.simon.170809-20:42'.

My Output.

sudo nano /var/log/tiger/security.report.username.170809-18:42

Security scripts *** 3.2.3, 2008.09.10.09.30 ***
Wed Aug  9 18:42:24 AEST 2017
20:42> Beginning security report for username (x86_64 Linux 4.4.0-89-generic).

# Performing check of passwd files...
# Checking entries from /etc/passwd.
--WARN-- [pass014w] Login (bob) is disabled, but has a valid shell.
--WARN-- [pass014w] Login (root) is disabled, but has a valid shell.
--WARN-- [pass015w] Login ID sync does not have a valid shell (/bin/sync).
--WARN-- [pass012w] Home directory /nonexistent exists multiple times (3) in
         /etc/passwd.
--WARN-- [pass012w] Home directory /run/systemd exists multiple times (2) in
         /etc/passwd.
--WARN-- [pass006w] Integrity of password files questionable (/usr/sbin/pwck
         -r).

# Performing check of group files...

# Performing check of user accounts...
# Checking accounts from /etc/passwd.
--WARN-- [acc021w] Login ID dnsmasq appears to be a dormant account.
--WARN-- [acc022w] Login ID nobody home directory (/nonexistent) is not
         accessible.

# Performing check of /etc/hosts.equiv and .rhosts files...

# Checking accounts from /etc/passwd...

# Performing check of .netrc files...

# Checking accounts from /etc/passwd...

# Performing common access checks for root (in /etc/default/login, /securetty, and /etc/ttytab...
--WARN-- [root001w] Remote root login allowed in /etc/ssh/sshd_config

# Performing check of PATH components...
--WARN-- [path009w] /etc/profile does not export an initial setting for PATH.
# Only checking user 'root'

# Performing check of anonymous FTP...

# Performing checks of mail aliases...
# Checking aliases from /etc/aliases.

# Performing check of `cron' entries...
--WARN-- [cron005w] Use of cron is not restricted

# Performing check of 'services' ...
# Checking services from /etc/services.
--WARN-- [inet003w] The port for service ssmtp is also assigned to service
         urd.
--WARN-- [inet003w] The port for service pipe-server is also assigned to
         service search.

# Performing NFS exports check...

# Performing check of system file permissions...
--ALERT-- [perm023a] /bin/su is setuid to `root'.
--ALERT-- [perm023a] /usr/bin/at is setuid to `daemon'.
--ALERT-- [perm024a] /usr/bin/at is setgid to `daemon'.
--WARN-- [perm001w] The owner of /usr/bin/at should be root (owned by daemon).
--WARN-- [perm002w] The group owner of /usr/bin/at should be root.
--ALERT-- [perm023a] /usr/bin/passwd is setuid to `root'.
--ALERT-- [perm024a] /usr/bin/wall is setgid to `tty'.

# Checking for known intrusion signs...
# Testing for promiscuous interfaces with /bin/ip
# Testing for backdoors in inetd.conf

# Performing check of files in system mail spool...

# Performing check for rookits...
# Running chkrootkit (/usr/sbin/chkrootkit) to perform further checks...
--WARN-- [rootkit004w] Chkrootkit has detected a possible rootkit installation
Possible Linux/Ebury - Operation Windigo installetd

# Performing system specific checks...
# Performing checks for Linux/4...

# Checking boot loader file permissions...
--WARN-- [boot02] The configuration file /boot/grub/menu.lst has group
         permissions. Should be 0600
--FAIL-- [boot02] The configuration file /boot/grub/menu.lst has world
         permissions. Should be 0600
--WARN-- [boot06] The Grub bootloader does not have a password configured.

# Checking for vulnerabilities in inittab configuration...

# Checking for correct umask settings for init scripts...
--WARN-- [misc021w] There are no umask entries in /etc/init.d/rcS

# Checking Logins not used on the system ...

# Checking network configuration
--FAIL-- [lin013f] The system is not protected against Syn flooding attacks
--WARN-- [lin017w] The system is not configured to log suspicious (martian)
         packets

# Verifying system specific password checks...

# Checking OS release...
--WARN-- [osv004w] Unreleased Debian GNU/Linux version `stretch/sid'

# Checking installed packages vs Debian Security Advisories...

# Checking md5sums of installed files

# Checking installed files against packages...
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-87-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.devname' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.softdep' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.alias' does not
         belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.builtin.bin'
         does not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.symbols' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/modules/4.4.0-89-generic/modules.dep.bin' does
         not belong to any package.
--WARN-- [lin001w] File `/lib/udev/hwdb.bin' does not belong to any package.

# Performing check of root directory...

# Checking device permissions...
--WARN-- [dev003w] The directory /dev/block resides in a device directory.
--WARN-- [dev003w] The directory /dev/char resides in a device directory.
--WARN-- [dev003w] The directory /dev/cpu resides in a device directory.
--FAIL-- [dev002f] /dev/fuse has world permissions
--WARN-- [dev003w] The directory /dev/hugepages resides in a device directory.
--FAIL-- [dev002f] /dev/kmsg has world permissions
--WARN-- [dev003w] The directory /dev/lightnvm resides in a device directory.
--WARN-- [dev003w] The directory /dev/mqueue resides in a device directory.
--FAIL-- [dev002f] /dev/rfkill has world permissions
--WARN-- [dev003w] The directory /dev/vfio resides in a device directory.

# Checking for existence of log files...
--FAIL-- [logf005f] Log file /var/log/btmp permission should be 660
--FAIL-- [logf007f] Log file /var/log/messages does not exist

# Checking for correct umask settings for user login shells...
--WARN-- [misc021w] There is no umask definition for the dash shell
--WARN-- [misc021w] There is no umask definition for the bash shell

# Checking symbolic links...

# Performing check of embedded pathnames...
20:47> Security report completed for username.

More on Tripwire here.

Hardening PHP

Hardening PHP config (and backing the PHP config it up), first create an info.php file in your website root folder with this info

<?php
phpinfo()
?>

Now look for what PHP file is loadingPHP Config

Back that your PHP config file

TIP: Delete the file with phpinfo() in it as it is a security risk to leave it there.

TIP: Read the OWASP cheat sheet on using PHP securely here and securing php.ini here.

Some common security changes

file_uploads = On
expose_php = Off
error_reporting = E_ALL
display_errors          = Off
display_startup_errors  = Off
log_errors              = On
error_log = /php_errors.log
ignore_repeated_errors  = Off

Don’t forget to review logs, more config changes here.

Antivirus

Yes, it is a good idea to run antivirus in Ubuntu, here is a good list of antivirus software

I am installing ClamAV as it can be installed on the command line and is open source.

sudo apt-get install clamav

ClamAV help here.

Scan a folder

sudo clamscan --max-filesize=3999M --max-scansize=3999M --exclude-dir=/www/* -i -r /

Setup auto-update antivirus definitions

sudo dpkg-reconfigure clamav-freshclam

I set auto updates 24 times a day (every hour) via daemon updates.

tip: Download manual antivirus update definitions. If you only have a 512MB server your update may fail and you may want to stop fresh claim/php/nginx and mysql before you update to ensure the antivirus definitions update. You can move this to a con job and set this to update at set times over daemon to ensure updates happen.

sudo /etc/init.d/clamav-freshclam stop

sudo service php7.0-fpm stop
sudo /etc/init.d/nginx stop
sudo /etc/init.d/mysql stop

sudo freshclam -v
Current working dir is /var/lib/clamav
Max retries == 5
ClamAV update process started at Tue Aug  8 22:22:02 2017
Using IPv6 aware code
Querying current.cvd.clamav.net
TTL: 1152
Software version from DNS: 0.99.2
Retrieving http://db.au.clamav.net/main.cvd
Trying to download http://db.au.clamav.net/main.cvd (IP: 193.1.193.64)
Downloading main.cvd [100%]
Loading signatures from main.cvd
Properly loaded 4566249 signatures from new main.cvd
main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Querying main.58.82.1.0.C101C140.ping.clamav.net
Retrieving http://db.au.clamav.net/daily.cvd
Trying to download http://db.au.clamav.net/daily.cvd (IP: 193.1.193.64)
Downloading daily.cvd [100%]
Loading signatures from daily.cvd
Properly loaded 1742284 signatures from new daily.cvd
daily.cvd updated (version: 23644, sigs: 1742284, f-level: 63, builder: neo)
Querying daily.23644.82.1.0.C101C140.ping.clamav.net
Retrieving http://db.au.clamav.net/bytecode.cvd
Trying to download http://db.au.clamav.net/bytecode.cvd (IP: 193.1.193.64)
Downloading bytecode.cvd [100%]
Loading signatures from bytecode.cvd
Properly loaded 66 signatures from new bytecode.cvd
bytecode.cvd updated (version: 308, sigs: 66, f-level: 63, builder: anvilleg)
Querying bytecode.308.82.1.0.C101C140.ping.clamav.net
Database updated (6308599 signatures) from db.au.clamav.net (IP: 193.1.193.64)

sudo service php7.0-fpm restart
sudo /etc/init.d/nginx restart
sudo /etc/init.d/mysql restart 

sudo /etc/init.d/clamav-freshclam start

Manual scan with a bash script

Create a bash script

mkdir /script
sudo nano /scripts/updateandscanav.sh

# Include contents below.
# Save and quit

chmod +X /scripts/updateandscanav.sh

Bash script contents to update antivirus definitions.

sudo /etc/init.d/clamav-freshclam stop

sudo service php7.0-fpm stop
sudo /etc/init.d/nginx stop
sudo /etc/init.d/mysql stop

sudo freshclam -v

sudo service php7.0-fpm restart
sudo /etc/init.d/nginx restart
sudo /etc/init.d/mysql restart

sudo /etc/init.d/clamav-freshclam start

sudo clamscan --max-filesize=3999M --max-scansize=3999M -v -r /

Edit the crontab to run the script every hour

crontab -e
1 * * * * /bin/bash /scripts/updateandscanav.sh > /dev/null 2>&1

Uninstalling Clam AV

You may need to uninstall Clamav if you don’t have a lot of memory or find updates are too big.

sudo apt-get remove --auto-remove clamav
sudo apt-get purge --auto-remove clamav

Setup Unattended Ubuntu Security updates

sudo apt-get install unattended-upgrades
sudo unattended-upgrades -d

At login, you should receive

0 updates are security updates.

Other

  • Read this awesome guide.
  • install Fail2Ban
  • Do check your log files if you suspect suspicious activity.

Check out the extensive Hardening a Linux Server guide at thecloud.org.uk: https://thecloud.org.uk/wiki/index.php?title=Hardening_a_Linux_Server

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

v1.92 added hardening a linux server link

Filed Under: Ads, Advice, Analitics, Analytics, Android, API, App, Apple, Atlassian, AWS, Backup, BitBucket, Blog, Business, Cache, Cloud, Community, Computer, CoronaLabs, Cost, CPI, DB, Development, Digital Ocean, DNS, Domain, Email, Feedback, Firewall, Free, Git, GitHub, GUI, Hosting, Investor, IoT, JIRA, LetsEncrypt, Linux, Malware, Marketing, mobile app, Monatization, Monetization, MongoDB, MySQL, Networking, NGINX, NodeJS, NoSQL, OS, Planning, Project, Project Management, Psychology, push notifications, Raspberry Pi, Redis, Route53, Ruby, Scalability, Scalable, Security, SEO, Server, Share, Software, ssl, Status, Strength, Tech Advice, Terminal, Transfer, Trello, Twitter, Ubuntu, Uncategorized, Video Editing, VLOG, VM, Vultr, Weakness, Web Design, Website, Wordpress Tagged With: antivirus, brute force, Firewall

How to develop software ideas

July 9, 2017 by Simon

I was recently at a public talk by Alan Jones at the UNE Smart Region Incubator where Alan talked about launching startups and developing ideas.

Alan put it quite eloquently that “With change comes opportunity” and we are all very capable of building the next best thing as technological barriers and costs are a lot lower than 5 years ago but Alan also mentioned 19 start-ups-ups fail but “if you focus on solving customer problems you have a better chance of succeeding”. Regions need to share knowledge and you can learn from other peoples mistakes.”

I was asked after this event to share thoughts on “how do I learn to develop an app” and “how do you get the knowledge”. Here is my poor “brain dump” on how to develop software ideas (It’s hard to condense 30 years experience developing software). I will revise this post over the coming weeks so check back often.

If you have never programmed before check out this programming 101 guides here.

I have blogged on technology/knowledge things in the past at www.fearby.com and recently I blogged about how to develop cloud-based services (here, here, here, here and here) but this blog post assumes you have a validated “app idea” and you want to know how to develop yourself. If you do not want to develop an app yourself you may want to speak with Blue Chilli.

Find a good mentor.


True App Development Quotes

  • Finding development information is easy, following a plan is hard.
  • Aim for progress and not perfection.
  • Learn one thing at a time (Multitasking can kill your brain).
  • Fail fast and fail early and get feedback as early as possible from customers.
  • 10 engaged customers are better than 10,000 disengaged users.

And a bit of humour before we start.

Project Mangement Lol

(click for larger image)

Here is a funny video on startup/entrepreneur life/lingo


This is a good funny, open and honest video about programming on YouTube.

Follow Seth F Samuel on twitter here.

Don’t be afraid to learn from others before you develop

My fav tips from over 200 failed startups (from https://www.cbinsights.com/blog/startup-failure-post-mortem/ )

  • Simpler websites shouldn’t take more than 2-3 months.You can always iterate and extrapolate later. Wet your feet asap
  • As products became more and more complex, the performance degrades. Speed is a feature for all web apps. You can spend hundreds of hours trying to speed of the app with little success. Benchmarking tools incorporated into the development cycle from the beginning is a good idea
  • Outsource or buy in talent if you don’t know something (e.g marketing). Time is money.
  • Make an environment where you will be productive. Working from home can be convenient, but often times will be much less productive than a separate space. Also it’s a good idea to have separate spaces so you’ll have some work/life balance.
  • Not giving enough time to stress and load testing or leaving it until the last minute is something startups are known for — especially true of small teams — but it means things tend to get pretty tricky at scale, particularly if you start adding a user every four seconds.
  • It’s possible to make a little money from a lot of people, or a lot of money from a few people. Making a little money from a few people doesn’t add up. If you’re not selling something, you better have a LOT of eyeballs. We didn’t.
  • We received conflicting advice from lots of smart people about which is more important. We focused on engagement, which we improved by orders of magnitude. No one cared. Lesson learned: Growth is the only thing that matters if you are building a social network. Period. Engagement is great but you aren’t even going to get the meeting unless your top-line numbers reach a certain threshold (which is different for seed vs. series A vs. selling advertising).
  • We most definitely committed the all-too-common sin of premature scaling. Driven by the desire to hit significant numbers to prove the road for future fundraising and encouraged by our great initial traction in the student market, we embarked on significant work developing paid marketing channels and distribution channels that we could use to demonstrate scalable customer acquisition. This all fell flat due to our lack of product/market fit in the new markets, distracted significantly from product work to fix the fit (double fail) and cost a whole bunch of our runway.
  • If you’re bootstrapping, cash flow is king. If you want to possibly build a product while your revenue is coming from other sources, you have to get those sources stable before you can focus on the product.
  • Don’t multiply big numbers. Multiply $30 times 1.000 clients times 24 months. WOW, we will be rich! Oh, silly you, you have no idea how hard it is to get 1.000 clients paying anything monthly for 24 months. Here is my advice: get your first client. Then get your first 10. Then get more and more. Until you have your first 10 clients, you have proved nothing, only that you can multiply numbers.
  • Customers pay for information, not raw data. Customers are willing to pay a lot more for information and most are not interested in data. Your service should make your customers look intelligent in front of their stakeholders. Follow up with inactive users. This is especially true when your service does not give intermediate values to your users. Our system should have been smarter about checking up on our users at various stages.
  • Do not launch a startup if you do not have enough funding for multiple iterations. The chances of getting it right the first time are about the equivalent of winning the lotto.

Here are my tips on staying on track developing apps. What is the difference between a website, app, API, web app, hybrid app and software (my blog post here)?

I have seen quite a few projects fail because:

  • The wrong technology was mandated.
  • The software was not documented (by the developers).
  • The software was shelved because new developers hated it or did not want to support it.

Project Roles (hats)

It is important to understand the roles in a project (project management methodology aside) and know when you are being a “decision maker” or a “technical developer”. A project usually has these roles.

  • Sponsor/owner (usually fund the project and have the final say).
  • Executive/Team leader/scrum master (manage day to day operations, people, tasks and resources).
  • Team members (UI, UX, Marketers, Developers (DevOps, Web, Design etc) are usually the doers.
  • Stakeholders (people who are impacted (operations, owners, Helpdesk)).
  • Subject Matter Experts (people who should guide the work and not be ignored).
  • Testers (people who test the product and give feedback).

It can be hard as a developer to switch hats in a one-person team.

How do you develop and gain knowledge?

First, document what you need to develop (what problem are you solving and what value will your idea bring). Does this solution exist already? Don’t solve a problem that already exists.

Developing software is not hard, you just need to be logical, research, be patient and follow a plan. The hardest part can be gluing components together.

I like to think of developing software like making a car if you need 4 wheels do you have 4 wheels? If you want to build it yourself and save some money can you make wheels (make rubber strips with steel reinforced/vulcanized rubber, make alloys and add bearings and have them pass regulations) or should you buy wheels (some things are cheaper to make than other things)? Developing software can be easy if you know what your are doing and have the experience and are aware of the costs and risks.  Developing software can lead you down a rabbit hole of endless research, development, and testing if you don’t know what you are doing.

Examples 1:

I “need a webpage”:

  • Research: Will Wix, Shopify or a hosted WordPress website do (is it flexible or cheap enough) or do I install WordPress (guide here) or do I  learn and build an HTML website and buy a theme and modify it (and have a custom/flexible solution)?

Example 2:

I “need an iPhone and Android app”:

Research: You will need to learn iOS and Android programming and you may need a server or two to hold the apps data, webpage and API. You will also need to set up and secure the servers or choose to install a database or go with a “database as a service” like cloud.mongodb.com or google firebase.

Money can buy anything (but will it be flexible/cheap enough), time can build anything (but will it be secure enough).

Developing software can be easy if you know what your are doing and have the experience and are aware of the costs and risks but developing software can lead you down a rabbit hole of endless research, development and testing if you don’t know what you are doing.

Almost all systems will need a central database to store all data, you can choose a traditional relational SQL database or a newer NoSQL database. MySQL is a good/cheap relational SQL database and MongoDB is a good NoSQL database. You will need to decide on how your app talks to the database (directly or via an API (protected by OAuth or limited access tokens)).  It is a bad idea to open a database directly to the world with no security. Sites like www.shodan.io will automatically scan the Internet looking for open databases or systems and report this as an insecure site to anyone. It is in your interest to develop secure systems in all stages of development.

CRUD (Create, Read, Update and Delete) is a common group of database tasks that you can do to prove you can read, write, update and delete from a database. While performing CRUD operations is a good to benchmark to also see how fast the database it.  if a database is the slowest link then you can use memory to cache database values (read my guide here). Caching can turn a cheap server into a faster server. Learning by doing can quickly build skills so “research”, “do” and “learn”.

Most solutions will need a website (and a web server). Here is a good article comparing Apache and Nginx (the leading open source web servers).

Stacks and Technology – There are loads of development environments (stacks), frameworks and technologies that you can choose. Frameworks supposedly make things easier and faster but frameworks and technologies change (See 2016 frameworks to learn guide and 2017 frameworks to learn guide) frequently (and can be abandoned). Frameworks supposedly make things easier and faster but be careful most frameworks run 30% slower than raw server-side and client code. I’d recommend you learn a few technologies like NGINX, NodeJS, PHP and MySQL and move up from there.

The Mean Stack is a  popular web development platform (MEAN = MongoDB, ExpressJS, Angular and NodeJS.).

Apps can be developed for Apple platforms by signing up here (about $150 AUD a year) and using the XCode IDE. Apps can be developed for the Android Platform by using Android Studio (for about $20 (one-off fee)). Microsoft has a developer portal for the Windows Platform. Google also has an online scalable database as a service called Firebase. If you look hard enough you will find a service for everything but connecting those services can be timely, costly or make security and a scalable solution impossible so beware of using as-a-service platforms. I used the Corona SDK to develop an app but abandoned the platform due to changes in the vendor’s communication and enforced policies.

If you are not sure don’t be afraid of ask for help on Twitter.

Twitter is awesome for finding experts

Recent twitter replies to a problem I had.

Learning about new Technology and Stacks

To build the knowledge you need to learn stuff, build stuff, test (benchmark), get feedback and build more stuff. I like to learn about new technology and stacks by watching Udemy courses and they have a huge list of development courses (Web Development, Mobile Apps, Programming Languages, Game Development, Databases,  Software Testing,  Software Engineering etc).

I am currently watching a Practical iOS 11 course by Stephen DeStefano on Udemy to learn about unreleased/upcoming features on the Apple iPhone (learning about XCode 9, Swift 4, What’s new in iOS 11, Drag and drop, PDF and ARKit etc).

Udemy is awesome (Udemy often have courses for $15).

If you want to learn HTML go to https://www.w3schools.com/.

https://devslopes.com/have a number or development related courses and an active community of developers in a chat system.

You can also do formal study via an education provider (e.g. Bachelor of computer sciences at UNE or Certificate IV in programming or Diploma in Software Development at TAFE).

I would recommend you use Twitter and follow keywords (hashtags) around key topics (e.g #www, #css, #sql, #nosql, #nginx, #mongodb, #ios, #apple, #android, #swift, #objectivec, #java, #kotlin) and identify users to follow. Twitter is great for picking up new information.

I follow the following developers on YouTube (TheSwiftGuy, AppleProgrammer, AwesomeTuts, LetsBuildThatApp, CodingTech etc)

Companies like https://www.civo.com/ offer developer-friendly features with hosting, https://www.pebbled.io/ offer to develop for you and https://serverpilot.io/ help you spin up software on hosting providers.

What To Develop

First, you need to break down what you need. (e.g ” I want an app for iOS and Android in 5 months that does XYZ. The app must be secure and be fast. Users must be able to register an account and update their profile”).

Choosing how high to ensure your development project scales depends on your peak expected/active concurrent users (ratio of paying and free users). You can develop your app to scale very high but this may cost more money initially, it can be bad to pay to ensure scalability early. As long as you have a good product and robust networking/retry routines and UI you don’t need to scale high early.

Once you know what you need you can search the open-source community for code that you can use. I use Alamofire for iOS network requests, SwiftyJSON for processing JSON data and other open-source software. The only downside of using open source software is it may be abandoned by the creators and break in the future. Saving your time early may cost you time later.

Then you can break down what you don’t want. (e.g “I don’t want a web app or a windows phone or windows desktop app”). From here you will have a list of what you need and what you can avoid.

You will also need to choose a project management methodology (I have blogged about this here). Having a list of action item’s and a plan and you can work through developing your app.

While you are researching it is a good idea to develop smaller fun projects to refine your skills.  There are a number of System Development Life Cycles (SDLC’s) but don’t worry if you get stuck, seek advice or move on. It is a  good idea to get users beta testing your app early and seek feedback. Apple has the TestFlight app where you can send beta versions of apps to best testers. Here is a good guide on Android beta testing.

If you are unsure about certain user interface options or features divide your beta testers and perform A/B or split testing to determine the most popular user interfaces. Capturing user data and logs can also help with debugging and user usage actions.

Practice

Develop smaller proof of concept apps in new technologies or frameworks and you will build your knowledge and uncover limitations in certain frameworks and how to move forward with confidence. It is advisable to save your source code for later use and to share with others.

I have shared quite a bit of code at https://simon.fearby.com/blog/ that I refer to from time to time. I should have shared this on GitHub but I know Google will find this if people want it.

Get as much feedback as you can on what you do and choose (don’t trust the first blog post you read (me included)).

Most companies offer Webinars on their products. I like the NGINX webinars. Tutorialspoint have courses on development topics. Sitepoint is a  good development site that offers free books, courses, and articles. What are API’s information by Programmable web.

You may want to document your application flow to better understand how the user interface works.

Useful Tools

Balsamic Mockups and Blueprint are handy for mocking up applications.

C9.io is a great web-based IDE that can connect to a VM on AWS or Digital Ocean.  I have a guide here on connecting Cloud 9 to an AWS VM here.

I use the Sublime Text 3 text editor when editing websites locally.

(image courtesy of https://www.sublimetext.com/ )

I use the Mac Paw app to help test API’s I develop locally.

(image courtesy of https://paw.cloud )

Snippets is a great application for the Mac for storing code snippets.

I use the Cornerstone Subversion app for backing up my code on my Mac.

Webservers: https://www.iis.net/IIS Webserver, NGINX Webserver, Apache Webserver.

NodeJS programming manual and tutorials.

I use Little Snitch (guide here) for simulating network down in app development.

I use the Forklift file manager on OSX.

Databases: SQL tutorials, NoSQL Tutorials, MySQL documentation.

Siege is a command-line HTTP load testing tool.

CPU Busy

http://loader.io/ is a nice web-based benchmarking tool.

Bootstrap is an essential mobile responsive framework.

Atlassian Jira is an essential project tracking tool. More on Agile Epics v Stories v Tasks on the Atlassian community website here. I have a post on developing software and staying on track here using Jira.

Jsfiddle is a good site that allows you to share code you are working on or having trouble with.

Dribbble is a “show and tell” site for designers and creatives.

Stackoverflow is the go-to place to ask for help.

Things I care about during development phases.

  • Scalability
  • Flexibility
  • Risk
  • Cost
  • Speed

Concentrating too much on one facet can risk exposing other facets. Good programmers can recommend a deliver a solution that can be strong in all areas ( I hate developing apps that are slow but secure or scalable and complex).

Platforms

You can signup for online servers like Azure, AWS (my guide here) or you can use a cheaper CPanel based hosting. Read my guide on the costs of running a cloud-based service.

Use my link to get a free Digital Ocean server for two months by using this link. Read my blog post here to help setup you VM. You can always use Ubuntu on your local machine to use Ubuntu (read my guide here). Don’t forget to use a GIT code repository like GitHub or Bitbucket.

Locally you can install Ubuntu (developers edition) and have a similar environment as cloud platforms.

Lessons Learned

  • Deploy servers close to the customers (Digital Ocean is too far away to scale in Australia).
  • Accessibility and testing (make things accessible from the start).
  • Backup regularly (Use GIT, backup your server and use Rsync to copy files to remote servers and use services like backblaze.com to backup your machine).
  • Transportability of technology (Use open technology and don’t lock yours into one platform or service).
  • Cost (expensive and convenient solutions may be costly).
  • Buy in themes and solutions (wrapbootstrap.com).
  • Do improve what you have done (make things better over time). Thing progress and not perfection.

There is no shortage of online comments bagging certain frameworks or platforms so look for trends and success stories and don’t go with the first framework you find. Try candidate frameworks and services and make up your own mind.

A good plan, violently executed now, is better than a perfect plan next week. – General George S. Patton

Costs

Sometimes cost is not the deciding factor (read my blog post on Alibaba cloud). You should estimate your apps costs per 1000 users. What do light v heavy users cost you? I have a blog post on the approx cost of cloud services.  I started researching a scalable NoSQL platform on IBM Cloudant and it was going to cost $4,000 USD a month and integrating my own App logic and security was hard. I ended up testing MongoDB Cloud where I can scale to three servers for $80 a month but for now, I am developing my current project on my own AWS server with MongoDB instance. Read my blog post here on setting up MongoDB and read my blog post on the best MongoDB GUI.

Here is a great infographic for viewing what’s involved in mobile app development.

You can choose a number of tools or technologies to achieve your goals, for me it is doing it economically, securely and in a scalable way that has predictable costs. It is quite easy to develop something that is costly, won’t scale or not secure or flexible. Don’t get locked into expensive technologies. For example, AWS has a user pays Node JS service called Lambada where you get Million of free hits a month and then you get charged $0.0000002 per request thereafter. This sounds good but I prefer fixed pricing/DIY servers better as it allows me to build my own logic into apps (this is more important than scalability).

Using open-source software of off the shelf solutions may speed things up initially? Will It slow you down later though? Ensure free solutions are complete and supported and Ensure frameworks are helping. Do you need one server or multiple servers (guide on setting up a distributed MySQL environment )? You can read about my scalability on a budget journey here. You can speed up a server in two ways Scale Up (Add more Mhz or CPU cores) or scale-out (add more servers).

Start small and use free frameworks and platforms but have a tested scale-up plan, I researched cheap Digital Ocean servers and moved to AWS to improve latency and tested MongoDB on Digital Ocean and AWS but have a plan to scale up to cloud.mongodb.com if need be.

Outsource (contractors) 

Remember outsourcing work tasks (or complete outsourcing of development) can buy you time and or deliver software faster. Outsourcing can also introduce risks and be expensive. Ask for examples of previous work and get raw numbers on costs (now and in the future) and concurrent users that a particular bit of outsourcing work will achieve.

If you are looking to outsource work do look at work that the person or company has done before (if is fast, compliant, mobile scalable, secure, robust, backup up, do you have rights to edit/own and own the IP etc). I’d be cautious of companies who say they can do everything and don’t show live demos.

Also, beware of restrictions on your code set by the contractors. Can they do everything you need (compare with your list of Moscow must haves)? Sometimes contractors only code or do what they are comfortable with that can impact your deliverables.

Do use a private Git repository (that you own) like GitHub or BitBucket to secure your code and use software like Trello or Atlassian JIRA to track your project. Insist the contractors use your repository to retain control.

You can always sell equity in your idea to an investor and get feedback/development from companies like Bluechilli.

Monetization and data

Do have multiple monetization streams (initial app purchase cost, in-app purchase, subscriptions, in-app credit, advertising, selling code/components etc). Monthly revenue over yearly subscription works best to ensure cash flow.

Capture usage data and determine trends around successful engagement, Improve what works. Use A/B testing to roll out new features.

I like Backblaze post on getting your first 1,000 customers.

Maintenance, support risk and benefits

Building your own service can be cheaper but also riskier if you fail to secure an app you are in trouble if you cannot scale you are in trouble. If you don’t update your server when vulnerabilities come out you are in trouble. Also, Google on monetization strategies. Apple apps do appear to deliver more profits over Android. Developers often joke “Apple devices offer 90% of the profits and 10% of the problems and Android apps offer 90% of the problems and 10% of the profits”.

Also, Apple users tend to update to the latest operating system sooner where Android devices are rather fragmented.

Do inform you users with self-service status pages and informative error messages and don’t annoy users.

Use Free Trials and Credit

Most vendors have free trials so use them

https://aws.amazon.com/free/AWS have 12 month free tiers.

Use this link to get two months free with Digital Ocean.

Microsoft Azure also give away free credit.

Google cloud also have free credit.

Don’t be afraid to ask.

MongoDB Cloud also gives away free credit if you ask.

Security

Sites like Shodan.io will quickly reveal weaknesses in your server (and services), this will help you build robust solutions from the start before hackers find them. Read https://www.owasp.org/index.php/Main_Page to know h0w to develop secure websites. Listen to the SecurityNow podcast to learn how the technology works and is broken. Following TroyHunt is recommended to keep up to date with security in general. @0xDUDE is a good ethical hacker to follow to stay up-to date on security exploits also @GDI_FDN is a good non-profit organization that helps defend sites that use open source software.

White hack hackers exist but so do black hat ones.

Read the Open Web Application Security site here. Read my guide on setting up public key pinning in security certificates here.

I use the ASafaWeb site to test your sites from common ASP security flaws. If you have a secure certificate on your site you will need to ensure the certificate is secure and up to date with the SSL Labs SSL Test site.

SSL Cert

Once your websites IP address is known (get it from SSL Labs) run a scan over your site with https://www.shodan.io/ to find open ports or security weaknesses.

Shodan.io allows you and others to see public information about your server and services. You can read about well-known internet ports here.

Anyone can find your server if you are running older (or current) web servers and or services.

It is a  good idea to follow security researchers like Steve Gibson and Troy Hunt and stay up to date with live exploits. http://blog.talosintelligence.com is also a good site for reading technical breakdowns of exploits.

Networking

Do share and talk about what you do with other developers. You can learn a lot from other developers and this can save you loads of time and mistakes. True developers love talking about their code and solutions.

Decision Making

Quite a lot of time can be spent on deciding on what technology or platform to use, I decide by factoring in cost, risk and security over flexibility, support and scalability. If I need flexibility, lower support or scalability then I’ll choose a different technology/platform. Generally, technology can help with support. Scalable solutions need effort from start to finish (it is quite easy to slow down any technology or service).

Don’t be afraid to admit you have chosen the wrong technology or platform. It is far easier to research and move on than live with poor technology.

If you have chosen the wrong technology and stick with it, you (and others) will loath working with it (impacting productivity/velocity).  Do you spend time swapping technology or platforms now or be less productive later?

Intellectual property and Trademarks

Ensure you search international trademarks for your app terms before you start using them. The Australian ATO has a good Australian business name checker here.

https://namechk.com/ is also a good place to search for your app ideas name before you buy or register any social media accounts.

Using https://namechk.com/ you can see “mystartupidea” name is mostly free.

And the name “microsoft’ is mostly taken.

Seek advice from a start-up experts from https://www.bluechilli.com/ like Alan Jones.

See my guide on how to get useful feedback for your ideas here.

Tips

  1. Use Git Source Control systems like GitHub or Bitbucket from the start and offsite backup your server and environments frequently. Digital Ocean charges 20% of your servers costs to back it up. AWS has multiple backup offerings.
  2. Start small and scale up when needed.
  3. Do lots of research and test different platforms, frameworks, and technologies and you will know what you should choose to develop with.

(Image above found at http://startupquotes.startupvitamins.com/ Follow Startup Vitamins on Twitter here.).

You will know when you are a developer when you have gained knowledge and experience and can automatically avoid technologies that will not fit a  solution.

Share

Don’t be afraid to share what you know (read my blog post on this here). Sharing allows you to solidify your knowledge and get new information. Shane Bishop from EWWW Image Optimizer  WordPress plugin wrote Setting up a fast distributed MySQL environment with SSL for us. If you have something to share on here please let me know here on twitter.

It’s never too late to do

One final tip is knowledge is not everything, planning and research is key, a mind that can’t develop may be better than a mind that can because they have no experience (or baggage) and may find faster ways to do things. Thanks to http://zachvo.com/ for teaching me this during a recent WordPress re-deployment. Sometimes the simplest solution is.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

DRAFT: 1.86 added short link

Short: https://fearby.com/go2/develop/

Filed Under: Advice, Android, Apple, Atlassian, Backup, BitBucket, Blog, Business, Cloud, CoronaLabs, Cost, Development, Domain, Firewall, Free, Git, GitHub, Hosting, JIRA, mobile app, MySQL, Networking, NodeJS, OS, Project Management, Scalability, Scalable, Security, Server, Software, Status, Trello, VM Tagged With: ideas

Blocking XCode iOS Simulator App traffic with the help of Little Snitch firewall

January 14, 2017 by Simon

Backend Considerations

I have been developing a mobile app using XCode and Swift 3 for a while now, I have been very focused on developing a scalable and robust back ends on multiple servers using different services. Loads of experts say you only have two chances to keep users engaged before they leave your app because of speed or silly error messages or slow apps.  Part of having a trustworthy app is giving users true error messages when errors pop up.  This involves testing and developing for each error scenario.

I have tried to guess every possible failure point and use HTTP status codes in my app API to inform the user of reasons why something has failed (not just a success/fail).

A typical API’s for my app returns these possible HTTP status error codes.

  • 400 – No Body.
  • 401 – Invalid payload (automatically validated multiple ways using node modules like validator).
  • 402 – Invalid input payload (manually validated by my code).
  • 403 – Backend NoSQL database down.
  • 404 – Invalid User
  • 405 – Query returned no results from back end NoSQL database
  • 406 – Invalid Output Payload (from various sources).
  • 407 – User has reached max queries in xx minutes.
  • 408 – Invalid Request
  • etc

I use http://keymetrics.io to monitor my node processes, custom code to notify me when things go down and I use the node package winston to log anything for later review.  I am happy with the throughput, even with the excessive logging and access controls and I am now moving onto the front end of my application.

Front End

I decided to use Swift 3 in XCode 8.2.1 to talk to my JSON API, I am using the Alamofire Networking module in Xcode to handle the network stack (as pure Swift 3 networking code was horrid).

Tip: I had issues with CocoPods to manage the installation of Alamofire so I ended up dropping it and installing it manually.

Once I setup my API setup the code below to query my API (‘/appname/api/v1/login/’) and process the data returned.

let parameters: Parameters = [
    "email": "\(sUsername)",
    "password": "\(sEncryptedHashedPassword)"
]
let headers: HTTPHeaders = [
    "x-access-token": "\(sSingleUseUserAccessToken)",
    "Content-Type": "application/json",
    "Accept": "application/json",
    "DNT": "1 (Do Not Track Enabled)"
]
let theAPIURL = globalSettings.API_LOGIN_PAGE

Alamofire.request("\(theAPIURL!)", method: .post, parameters: parameters, encoding: JSONEncoding.default, headers: headers)
    .validate(statusCode: 200..<201)
    .validate(contentType: ["application/json"])
    .responseData { response in
        switch response.result {
        case .success(let data):
            
            # Debug 
            print("Login Success (200)")
            print("response.request: \(response.request)")  // original URL request
            print("response.response: \(response.response)") // HTTP URL response
            print("reposnse.data: \(response.data)")        // server data
            print("result: \(response.result)")   // result of response serialization
            print("Login Time: \(response.timeline)")  // time
            
            // Processing Successful Login.
            
            let json = try! JSONSerialization.jsonObject(with: data)
            
            // Debug Return Payload
            print(json)
                        
            // Unwraping Payload Preferences

        
            // Assume the App Loads OK (Checks below)
            var local_LoadedOk = "Yes"
                local_LoadedOk = "Yes"
            var local_LoadedNotOKMessage = "Unknown Error"
                local_LoadedNotOKMessage = "Unknown Error"
            
            // API Payload Validation
            
            var local_LoginChecksPassingOK = true
                local_LoginChecksPassingOK = true
            
            // Get the users guid from the API Payload ( simple validation )
            var local_f_guid = (json as! NSDictionary)["f_guid"] as! String
            print("local_f_guid: \(local_f_guid)")
            if local_f_guid != "" {
                // Is the guis the right length?
                if local_f_guid.characters.count == 36 {
                    // ok
                    print(" - guid ok")
                } else {
                    local_f_guid = ""
                    // login guid too short
                    local_LoginChecksPassingOK = false
                    local_LoadedOk = "No"
                    local_LoadedNotOKMessage = "There was an error with your account (login guid error). Please contact support (LoginError_001)"
                }
            } else {
                local_f_guid = ""
                // Error Login Guid Missing
                local_LoginChecksPassingOK = false
                local_LoadedOk = "No"
                local_LoadedNotOKMessage = "There was an error with your account (login guid error). Please contact support (LoginError_002)"
            }
            
        // Get Username from API Payload
            var local_Username = (json as! NSDictionary)["Username"] as! String
            print("local_Username: \(local_Username)")
            if (local_LoginChecksPassingOK == true) {
                print("login validation ok so far")
                // Get Username
                if local_Username != "" {
                    // Check Email for @ symbol
                    if local_Username.characters.count > 0 {
                        // ok
                        print(" - username ok")
                    } else {
                        local_Username = ""
                        // Username Empty
                        local_LoginChecksPassingOK = false
                        local_LoadedOk = "No"
                        local_LoadedNotOKMessage = "There was an error with your account (email invalid). Please contact support (LoginError_003)"
                    }
                    
                } else {
                    local_Username = ""
                    // Username Empty
                    local_LoginChecksPassingOK = false
                    local_LoadedOk = "No"
                    local_LoadedNotOKMessage = "There was an error with your account (email invalid). Please contact support (LoginError_004)"
                }
            }

            
        // Get Email from API Payload
            var local_Email = (json as! NSDictionary)["Email"] as! String
            print("local_Email: \(local_Email)")
            if (local_LoginChecksPassingOK == true) {
                if (local_LoginChecksPassingOK == true) {
                    if local_Email != "" {
                        // Check Email for @ symbol
                        if local_Email.contains("@") == true {
                            print(" - email ok")
                        } else {
                            local_Email = ""
                            // Username Empty
                            local_LoginChecksPassingOK = false
                            local_LoadedOk = "No"
                            local_LoadedNotOKMessage = "There was an error with your account (email invalid). Please contact support (LoginError_005)"
                        }
                    } else {
                        local_Email = ""
                        // Username Empty
                        local_LoginChecksPassingOK = false
                        local_LoadedOk = "No"
                        local_LoadedNotOKMessage = "There was an error with your account (email invalid). Please contact support (LoginError_006)"
                    }
                }
            }
            
            
            // Was there and Error Loading or Saving
            // ... Code Removed

            // Unload other values
            // ... Code Removed
            
            // Save Preferences
            // ... Code Removed
            
            self.loginActivityIndicator.stopAnimating()
            
            // Redirect Back to Main View
            self.lblLoginProcessing.text = "Returning to the Main Screen........."
            let vc = ( self.storyboard?.instantiateViewController( withIdentifier: "mainViewController") )!
            //vc.view.backgroundColor = UIColor.orange()
            vc.modalTransitionStyle = .crossDissolve
            self.present(vc, animated: true, completion: nil)
            
        case .failure(let error):
            
            var sErrorTitle = ""
            var sErrorBody = ""
            print(" - Login Error")
            print(" - - \(error._code)")
            print(" - - \(error)")
            
            if error._code == NSURLErrorTimedOut {
                //timeout
                print("Error: Server Timeout (NSURLErrorTimedOut)")
                sErrorTitle = "Server Timeout"
                sErrorBody = "The login server timed out.\r\n\r\n Error: \(error)"
            }
            
            if (response.response?.statusCode == 402) {
                print("Error: Invalid Password (402)")
                sErrorTitle = "Invalid Password"
                sErrorBody = "The password you entered was invalid.\r\n\r\n Error: \(error)"
                
            } else if response.response?.statusCode == 403 {
                print("Error: Unknown Account (403)")
                sErrorTitle = "Unknown Account"
                sErrorBody = "The account you entered was not found.\r\n\r\n Error: \(error)"
           
            } else if response.response?.statusCode == 408 {
                print("Error: Server Timeout2 (NSURLErrorTimedOut)")
                sErrorTitle = "Server Timeout2"
                sErrorBody = "The login server timed out.\r\n\r\n Error: \(error)"
                
            } else if response.response?.statusCode == 499 {
                print("Error: Invalid or missing token, please update your app (499) ")
                sErrorTitle = "Invalid Version"
                sErrorBody = "The app token was invalid (or outdated), plaase update your app and try again.\r\n\r\n Error: \(error)"
                
            } else if response.response?.statusCode == 503 {
                print("Error: Database Read Error")
                sErrorTitle = "Sever Error (503)"
                sErrorBody = "The server cannot process your login at this time (Error 503).\r\n\r\n Error: \(error)"
                
            } else if response.response?.statusCode == 504 {
                print("Error: Database Write Error (504)")
                sErrorTitle = "Sever Error"
                sErrorBody = "The server cannot process your login at this time (Error 504).\r\n\r\n Error: \(error)"
                
            } else  {
                print("Unknwon Error (\(response.response?.statusCode)")
                sErrorTitle = "Unable to login"
                
                sErrorBody = "The App was unable to login. Please check your mobile and or wifi settings and try again."
                //sErrorBody = "There was an unknown error loging in (Error (\(response.response?.statusCode))\r\n\r\n Error: \(error)"
                self.resignFirstResponder()
            }

            
            self.loginActivityIndicator.stopAnimating()
            self.resignFirstResponder()
            
            // Show Loading Alert
            let alert = UIAlertController(title: "\(sErrorTitle)", message: "\(sErrorBody)", preferredStyle: .alert)
            self.present(alert, animated: true, completion: nil)
            let when = DispatchTime.now() + 5
            DispatchQueue.main.asyncAfter(deadline: when){
                alert.dismiss(animated: true, completion: nil)
            }
            
        }
}

Everything is working like a real app.  If I enter valid credentials my app logs me in.

If I enter incorrect credentials I get an error.

LittleSnitch001

If I stop my Node login service and try and log in I get an appropriate error message.

LittleSnitch002

Simulating full or partial network request failures on different endpoints

I checked the iOS Simulator (10.0 running iOS 10.2) that comes with XCode 8.2.1 to find a way to turn off the network to the simulator and I coudl not find an option???

The iOS Simulator lacks the usual Wifi and Mobile configuration options found on iOS devices.

LittleSnitch003

XCode Simulator is lacking network control features.

LittleSnitch004

LittleSnitch005

XCode allows me to see the network stats within my app but not adjust the network layer status.

LittleSnitch007

Like all good developers I opened google and typed “Is it possible to disable the network in iOS Simulator? and found many solutions on how to disable the network in the simulator like:

  • Close the simulator, disconnect from the internet, start XCode and your project and simulator and then connect to the network (that way the simulator stays disconnected until the simulator reboots). – This does not work.
  • “Build a simple Faraday cage to block or limit the external RF signal level”.

w6ehv

  • “Create a walk-in Faraday cage with a desk inside, the Mac will be much easier to work with”.

I did not want to spend minutes disconnecting and reconnecting to the internet or build a faraday cage so I took Felix advice and downloaded an application for OSX called Little Snitch from Objective Development.

Little Snitch

Reading the Little Snitch website the software reminds me of the good old days of controlling everything before Operating System Vendors buried these features.

Snip: “Whenever an application attempts to connect to a server on the Internet, Little Snitch shows a connection alert, allowing you to decide whether to allow or deny the connection. Your decision gets stored as a rule which will automatically be applied to future, similar connection attempts from the same application.”

Time to give Little Snitch a go, $34.95 is a bargain if it works as good as it says it does.

Little Snitch

Little Snitch took 5 mins to install (low level).  After it rebooted the Little Snitch Configuration program popped up.

LittleSnitch008

Little Snitch – System Tray Options were available too.

LittleSnitch009

Default Configuration (will take a number of minutes).

Little Snitch was now prompting me to approve many network connections for background apps.  Currently as we speak MongoDB and AWS Elasticsearch servers are being hit with ransomware. I might be patient and manually approve every process wanting to use my network with Little Snitch.

I opened many apps and responded to the network access prompts when the apps tried to talk to the network.

LittleSnitch0010d

Manually invoking an application to use the network (software update) results in an approval pop-up.

LittleSnitch0010c

After a number of minutes reviewing app network permissions, I loaded up the Little Snitch Network Monitor. Nice.

LittleSnitch0011

The Network Monitor is handy for reviewing in real-time what is happening on your Network/Machine.

Note: My BitDefender is rather busy.

LittleSnitch0012

I have digressed,  let’s see if Little Snitch can block my iOS App to assist with debugging API’s.

Blocking iOS Simulator Traffic with Little Snitch

XCode itself wanted access to the internet before I opened my project.

As soon as I started the XCode iOS Simulator I blocked all simulator related processes (I can turn it back on later).

LittleSnitch0013

Tip” I just found out if you move the mouse above the forever button in the dark grey area you can view more information.

I blocked the following iOS Simulator related processes from making any connection forever.

LittleSnitch0014

LittleSnitch0015

LittleSnitch0016

LittleSnitch0017

Now to start my app on the simulator from XCode and invoke a network call and see if we can block it to trigger my error pop-up.  Yes, I was able to block the iOS Simulated app with Little Snitch 🙂

LittleSnitch0019

I received this “correct” error in my app, Excellent, now I can customize the error messages in my app.

LittleSnitch0020

🙂

The eagle eyes will notice that the error message above is the same as when I turned off the Node Server that handles the login.  Now I need to add some XCode code in to detect “Is Wifi Network Up”,”Is Mobile Network Up”Can Access Network” and “Can Ping Server” etc. This would provide true error messages and not give the user any doubt to what the problem was.

If it was their device blocking my app they need a different message to one that reports a general data connection error or server down error.

Now how do I enable the blocked network traffic in Little Snitch?

Open the Little Snitch Configuration app.

LittleSnitch008

You can easily see what processes are allowed/blocked and change the setting (double click then change the connection to/from to Allow/Deny).

LittleSnitch0022

Summary

As it turns out I did not need to block the other iOS processes (just my app) so in future, I will just Deny or Allow for my app (until quite).

LittleSnitch0019

Little Snitch from Objective Development is an awesome app and allows me to block traffic where XCode would not.  As a bonus, it will secure your machine and help keep it safe.

I will update this guide when I learn more about Little Snitch.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

Filed Under: Apple, Firewall, Networking, Tech Advice Tagged With: Block, Firewall, Networking

Quick guide to using Adobe Premiere Pro CC to make videos.

June 30, 2016 by Simon Fearby

This is a simple guide for using Adobe Premiere Pro to edit simple videos.

Adobe used to sell a Master Collection of all Adobe software that cost thousands of dollars. Adobe have moved to a monthly subscription model for single or bundled software packages. Goto http://www.adobe.com/au/creativecloud.html and review the software options.

Adobe Photoshop is a must for editing Photographs and Adobe Premiere Pro is a must for video editing.

Adobe CC

You will need to choose your Adobe Software Package or Bundle (above) and follow the prompts to purchase it. You will be required to link the software to an Adobe ID (Adobe Account). Once you link your Software to an Adobe ID you can login into any PC/Mac and install your software using the Adobe App Download Utility.

Fyi: If you can buy a subsidised 1 year subscription of Creative Cloud at your work you will need to..

  • 1: Buy the 1 year redemption from your work.
  • 2: Create an Adobe account.
  • 3: Redeem the subscription.
  • 4: Download the apps with the Adobe Download Manager app.

The Adobe Download manager will allow you to download or update any of your purchased software packages

Update installed apps

cc_1

Download new apps.

Adobe Creative Cloud

Once you have installed Adobe Premiere Pro you can open it.

premiere_pro_01

Before you start.

  • Have you recorded all of your footage?
  • Do you have at least 20GB free space?
  • Save your videos to a folder that will not be moved later as Premier Pro links to the videos where they reside are and moving the videos will trigger you to re-link them.
  • Do you have a good story to tell?

Bookmark the Adobe Premiere Pro CC help pages here https://helpx.adobe.com/premiere-pro.html and visit YouTube and search for Adobe premiere Pro CC Tutorial.

Adobe Premiere Pro user Interface Basics.

I created a new project called “snow” and will save the project in the default place. Select “HDV” for High Definition. Click on the scratch disc tab.

premiere_pro_02

Adobe Premiere Pro will use these locations to copy andy of the videos or audio instead of touching the original videos. You can delete cached content later (these folders will grow a lot).

premiere_pro_03

The first thing you need to learn is how to reset your workspace. Go to the Window then Workspaces the Editing menu to reset your view.

premiere_pro_04

Click the “Project: Snow” tab in the lower left-hand side to get ready to start importing videos.

You could just start dragging and dropping in dozens of videos but I would suggest you create a number of folders (called “bins”) to store your video and other media assets into.

premiere_pro_05

I will create 4 folders (bin’s) called:

  • 01. On our way to the snow
  • 02. Above Sheba Dam
  • 03. Near Ponderosa
  • 04. Misc

Now I can drag the videos (and pictures) into each bin folder.

premiere_pro_06

Once you have filled each folder take some time reviewing each folder and think about what story you can create. I am going to create a fun mash-up for the day in order.

Creating My Snow Video

First, I double click on my “01. On our way to the snow” bin folder and drag the video onto the timeline sequence.

premiere_pro_07

Hopefully, you can now see your video in the timeline

premiere_pro_08

Shortcut Keys:

  • SPACEBAR plays and pauses the clip.
  • MINUS will Zoom Out of the timeline.
  • PLUS will Zoom Into the timeline.
  • BACKSPACE will zoom out and show all of your timeline.

More shortcuts keys here.

If you were skilled at recording you can just add more video to the timeline and be finished in no time but we have loads of editing and re arranging to do.

Adding a Title

Move your playback cursor to the start pf the video (press HOME ) and go to the Title, New Title then Default Still to start creating a new title.

premiere_pro_09

If you ever used Photoshop this will be very familiar. Close the title screen to save it.

premiere_pro_10

Now I dragged and dropped the timeline above the existing video track. I moved the driving time-lapse to the right and dragged a photo before the time-lapse. Now you can see how you can mix together Videos, Picture and text.

premiere_pro_11

Before I get too carried away I am going to source a background soundtrack from https://www.youtube.com/user/NoCopyrightSounds/videos I am going to choose this one https://www.youtube.com/watch?v=UkUweq5FAcE

If I have enough footage I like to have my videos match the audio where possible for greater impact.

Basic Editing Tools

These are the tool’s that you will use 99% of the time.

premiere_pro_12

If you want to cut a video in half use the Razor tool, if you want to move or interact with clips switch back to the selection tool.

Cutting, Trimming and Cropping Videos and audio

Move the selection tool to the end of a video and click and drag when you see this icon to truncate the length of a video without cutting it.

premiere_pro_13

Move the selection tool to the start of a video and click and drag when you see this icon to truncate the length of a video without cutting it

premiere_pro_14

You could cut a video to size but truncate it (like above) will allow you to extend the clip again without rejoining the video sections.

Layers

Adobe Premiere Pro (like Photoshop) allows you to have layers of videos to allow you to show a “picture in picture” or (picture in picture in picture).

premiere_pro_15

Building your Video.

Now comes the time-consuming bit of finding each video to insert, placing it, trimming it, cutting it and or moving it. This can take hours or days (depending on how fussy you are).

Caching your files.

Adobe Premiere Pro likes to cache your files to speed up the preview and export process. The green, yellow and red lines indicate what parts of your timeline are cached.

premiere_pro_16

Exporting your Video

premiere_pro_17

Exports can take up to 1 hour per 5 minutes on a 3 year old computer.

My VLOG Trip to the Snow video.

My 1st VLOG video.

Tutorials to Watch:

  • Create a Ken Burns Effect in Premiere Pro CC
  • Adobe Premiere Pro CC – Editing 101: Basic Audio (Part 3)
  • Episode 19 – Adding Video and Audio Transitions – Tutorial for Adobe Premiere Pro CC 2015

If you need more help consider buying a professional Adobe Premiere Pro CC course from Udemy.com or search YouTube for Adobe Premiere Pro CC Tutorials

The Adobe Help site is great too: https://helpx.adobe.com/premiere-pro.html

Good luck and let me know what you create.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

Filed Under: Apple, Cloud, Video Editing Tagged With: adobe, editing, premiere, pro, Video

How to get started in programming

June 22, 2016 by Simon Fearby

Today I was asked: Should I learn to code in Swift and what can I build? Swift 1.0 (http://www.swift.org) was launched in 2014 by Apple and it is a multi-platform programming language for iOS, macOS and Linux (but not Windows). Swift has gone through rapid changes recently and Swift 4.0 is the latest stable version. Swift updates can break a number of previous swift coding standards set in Swift 1.x, 2.x and 3.x so don’t get comfortable.

Should a beginner learn Swift? Yes if you only want to develop apps for iOS and macOS and ignore Windows and Android platforms. Apple has released a learn to code in Swift app that will make learning swift much easier http://www.apple.com/swift/playgrounds/

Previously apple recommended developers use the Objective C language to code and compile apps for iOS and macOS. Objective C has been in production since 1983 and is very complex (loads of squiggles, square brackets and legacy classes).

Most smartphones and tablets run Android, not because they are better but because they are cheaper (that’s my opinion). A top of the line iPhone 6S+ costs $1500 where a reasonable Android phone will set you back about $130 to $400. I personally think Apple devices are faster, better and more secure but if you are developing you need to publish apps on android also.  Apple devices are supported for a lot longer than Android devices ( Android support lifetime v Apple iOS ).  Even the 5-year-old iPad 2 is getting the iOS 10 software update in September 2016.  In 2017 iOS 11 does not run on an iPad 2.

If you wanted to native develop android apps you would need to learn Java in the https://developer.android.com/studio/index.html IDE. Be prepared to be confused as the Android Studio has a steep learning curve.

Ok so where do beginners start.

What Companies look for when hiring programmers

https://youtu.be/QbSD4EtpVdY

Jumping right into Swift, Java, Objective C or Lua may not be a good idea if a plain old website will do. It depends on what you want to develop before you start coding. All developers should be able to knock up a website and database before jumping into making mobile apps. PHP ( http://www.php.net ) and MySQL ( http://www.mysql.com ) are good options for beginners making websites.

http://www.udemy.com and http://www.w3schools.com/ is a great place to go to learn more about coding. If you want to see what the Pro’s are doing check out http://www.sitepoint.com are great places for learning WHAT you need to know fast.

But I really want to develop a mobile app.

Development platforms like the Corona SDK http://www.coronalabs.com are a great option for beginners as it is easy to pick up and is super fast and supports eye-popping OpenGL animations and apps along with business apps. Corona allows you to code in a programming language called Lua ( https://www.coronalabs.com/learn-lua ) and compiles your app to the iOS/Android/macOS or Windows desktops. How cool is that.

Corona APp

Another possible solution is using the Electron technology

Corona wraps a common interface (API  https://docs.coronalabs.com/API/index.html) over each platforms API so your code calls the corona API and when you compile your app the platforms native API methods are called.

Corona has great guides and support pages:

https://docs.coronalabs.com/guide/programming/index.html – Getting Started

https://docs.coronalabs.com/guide/index.html –

https://coronalabs.com/blog – Keep up to date with Corona and read guides on many topics.

https://coronalabs.com/resources – Corona Resources.

https://www.youtube.com/user/CoronaGeek – Weekly Corona video podcast.

https://forum.coronalabs.com – Talk to hundreds of Corona developers and ask questions.

https://docs.coronalabs.com/api/index.html – Corona API

What tools do you need

  • A Mac Computer with a retina display.
  • Sublime text Editor https://www.sublimetext.com/3 (and Sublime to Corona Plugin https://coronalabs.com/products/editor/ )
  • Source Version Software http://www.zenaware.com/cornerstone
  • A good code snippet saving app is http://snippets.me/
  • Patience and drive.

Knowing what you want to develop will narrow down the technologies you need to learn.

Summary:

  • If you want to make websites learn HTML and PHP.
  • If you want to build business apps inside corporations learn Visual Studio.NET
  • If you want to make mobile apps fast learn Corona.
  • If you want to make advanced iOS apps learn Swift
  • If you want to make advanced Android apps learn Android Studio
  • If you want to make online database learn MySQL

Check out my guides here on:

How to build your first cross-platform mobile app with corona

Creating a development server for $5 a month

What is the difference between a website, app, web app, hybrid app and software?

..and many more free guides here.

Happy coding.

Still reading?  Check out the beginner guides on Sitepoint.

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

V1.2. Added short link

Sort: https:/fearby.com/go2/learn/

Filed Under: Apple, Cloud, CoronaLabs, Development, MySQL, Scalable, Security Tagged With: Android, build, code, corona, iOS, test

How to build your first cross platform mobile app with corona

November 6, 2015 by Simon Fearby

cross_platform_app-2

Ok, you have an Apple or Android device and want to develop mobile/tablet apps. The first thing you will need is an Apple computer, Apple Developer Subscriptions to develop Apple apps and a Google developer subscription for Android along with some software.  You can develop on Windows but it requires emulating the Mac OSX and it is not recommended. I started developing on a $699 Mac Mini computer running the Apple OSX Operating system.

This guide covers Apple app development steps now, I will be adding Android and Windows Phone building steps soon..

Before you begin I will assume..

  • you have an Apple computer (running the latest system version of Apple’s operating system).
  • you have a paid Apple Developer subscription ($150 AUD a year).
  • you have installed XCode, Sublime Text 3 (free trial is ok) and Corona Labs SDK.
  • you have installed the Corona Editor Sublime package (info).

1. Apple Developer Setup

1.01 Login to the Apple Developer Portal.

apple_app_dev_001

1.02 Use your paid apple developer subscription to login.

apple_app_dev_002

1.03 Developer Subscription Options

The Apple developer portal is quite busy at first sight.

  • SDK’s will list all the downloads you have available as a developer.
  • Certificates, Identifiers & Profiles is where you prepare certificates that are required to make apps.
  • iTunes Connect is where you track your app sales and submit apps to the app store.
  • other options hold no surprises.

apple_app_dev_003

1.04 Clicking the Certificates, Identifiers & Profiles above will load this screen.

apple_app_dev_004

1.05 App submission workflow

Apple have detailed guides on how you develop apps and submit them to the stores.  If in doubt ask on the Apple developer forums.

apple_app_dev_005

1.06 Creating certificates for your development environment.

Apple have very robust security around apps and distribution.  On Windows desktops you can make a program, upload it and have very little restrictions on putting malicious code in your app. Apple, on the other hand, do not allow you to put an app on your iPhone that you develop yourself unless you go through many hoops to first link your machine to a development account, application and development account/team. Follow the steps below to setup a brand new development machine to develop and distribute apps to a number of test devices locally.

apple_app_dev_006

1.07 Open XCode and go to XCode menu, Preferences and Accounts and click the + and Add Apple ID.

apple_app_dev_007

1.08 Sign in with your Apple developer account.

apple_app_dev_008

1.09 You will notice one or more Team names (depending on your account), Select the first team account in the bottom right list and click View Details.

apple_app_dev_009

1.10 Clicking Create on each of the Signing Identities will run a series of scripts to setup encryption keys and sync them with Apple.

apple_app_dev_010

1.11 Click Create next to all of the items except Developer ID Application and Developer ID Installer is what we need to make mobile and mac desktop apps. Developer ID signing identities are required for apps that are to be published outside of the Apple Desktop Gatekeeper protections.

apple_app_dev_011

1.12 OK, what happened above?  Xcode created a bunch of certificates that are linked to your machine, apple servers and your development account.  The certificates will last for 1 year only; you will need to refresh them in 11 months time.  You can view these certificates by opening the Apple Keychain Access program.

Certificates found under the My Certificates group.

apple_app_dev_012

1.13 There are more certificates found under the Certificates group.

apple_app_dev_013

1.14 Although not mandatory I would highlight the new certificates and back them up.  Select the certificates (shift click) and go to the Keychain Access File menu then click Export Items.

apple_app_dev_014

1.15 Click Save and enter a strong password (write the password down).

apple_app_dev_015

1.16 Also backup the newly created My Certificates too.

apple_app_dev_016

1.17 Now put these files somewhere safe away from your main development machine just incase you need to reactivate the certificates on a different machine

apple_app_dev_017

1.18 Now we can log back into https://developer.apple.com and open Certificates, Identifiers and Profiles and click on All under Certificates. You will see the certificates that XCode created for you.

apple_app_dev_018

1.19 But first we need to add some real test devices (iPhone’s and iPad’s) to the list of approved devices).  Open the Devices area of your developer portal.

apple_app_dev_019

1.20 Every Apple device has a unique UDID inside it and we need to find it and add it to the list of known/approved devices we want to test apps on.  We can only add 100 devices a year so don’t go nuts adding devices of mates.

http://whatsmyudid.com/ is a great site that explains how you can find the UDID using iTunes or via the web.

To add your first test device plug it into iTunes and open the devices page and mouse click on the serial number.

apple_app_dev_020

1.21 When you see the UDID appear press CMD+C to copy the UDID to the clipboard.

apple_app_dev_021

1.22 Now go back to the Apple Developer Portal, open Certificates, Identifiers and Profiles, open Devices select iPhone (or your device type) and click the + and enter your device description and UDID.  Be descriptive “Simon’s iPhone 6 Plus” and not just “iPhone”.

apple_app_dev_023

1.23 Click Continue above then Register.

apple_app_dev_024

Add more test devices now (this will save regenerating certificates later).

1.24 Now we can create an Application ID.  Open Identifiers then click App IDs.

apple_app_dev_025

1.25 Your first App certificate should be a Wildcard App certificate.  A wildcard certificate allows you to build many different test apps using the one certificate (the downside is a wildcard app cannot receive push notifications and some other functions).  A wildcard allows you to quickly mock up a new app and build it in seconds.

Enter “App Wildcard” in the name, select Wildcard App ID radio button, and “com.yourcompany.*” (change your company name to your company name) in the Wildcard App ID.

apple_app_dev_026

1.26 No need to select any of the extra options provided.

apple_app_dev_027

1.27 Submit the App ID.

apple_app_dev_028

1.28 You can read more on the extra services here and here.

apple_app_dev_029

1.29 Now you can create dedicated App IDs for known apps that you want to build with additional services. Choose a good name and use the “com.yourcomany.yourapp” App ID.

apple_app_dev_030

1.30 You can choose more app services for dedicated App IDs. Below are the ones I prefer as standard.

apple_app_dev_031

1.31 Confirm your App ID and submit it.

apple_app_dev_032

1.32 Now the final step of creating development certificates to allow us to put the apps on devices.

Open the Development node under Provisioning Profiles.

apple_app_dev_034

1.33 Click iOS App Development (note we are selecting development and not distribution, we will return here to create a distribution certificate when we are ready to publish to the app store)

apple_app_dev_035

1.34 Choose your desired App ID to build a provisioning profile for (wildcard or dedicated)

apple_app_dev_036

1.35 Select the Certificate (or teams) to use.

apple_app_dev_037

1.36 Select the devices you wish to be able to run the app.

apple_app_dev_038

1.37 Review and click generate.

apple_app_dev_039

1.38 There is no need to download these as Xcode can do this.

apple_app_dev_040

Create as many App IDs as you like now.

1.39 Open XCode and return the file XCode then Preferences menu then Accounts tab.  Xcode should auto refresh and sync down your App IDs.

fyi: There are errors in my graphic below (all names appear the same but the list will be the same as you created them.

apple_app_dev_041

1.40 Now the fun part, go to http://coronalabs.com and signup and download the editor.

Corona comes with loads of demo projects and you can open a project and view it in the simulator and or build it straight away,

apple_app_dev_044b

1.41 I opened the Button Events demo and instantly went to Corona’s File, Build then iOS menu. Select a certificate that you want to use to build the app.  A device can have hundreds of apps on it that were built with the Wildcard certificate or one app per dedicated app certificate.

apple_app_dev_042

1.42 Corona saves an app file on the desktop (if that is where you saved to).  Open Xcode and click Window then Devices menu

apple_app_dev_043

1.43 Plug in your device and it should appear in Xcode’s list of devices.  Simply drag the app to your device to upload it to your device.

apple_app_dev_044

1.44 You can see the compiled app on the desktop and the uploaded app on the device.

apple_app_dev_045

1.55 You can see the simulated app on the desktop and the real app on the device. Congratulations.

apple_app_dev_046

2. Android device setup.

2.01 You will need to install the Apple Java Runtime to be able to build Android apps on Corona.

2.02 Optional: Install the Android Developer Studio ( also requires JDK7 ). The full blown Android Developer Studio allows you to run older versions of Android in simulators.

2.02.01 Open the Android Developer Studio app.

Android Studio

2.02.02 Click Next.

Android Studio

2.02.03 If you do not have JSK installed goto the next step to download it.

Android Studio

2.02.04 Goto Oracle and download the latest JDK7 and install it (or skip to 2.02.05)

Android Studio

2.02.05 If you just installed JDK click Previous then Next to detect the JDK location.

Android Studio

2.02.06 Select Custom and click Next

Android Studio

2.02.07 Choose your Theme and click Next

Android Studio

2.02.08 Tick the optional “Android Virtual Device – (1 GB)” and click next.

Android Studio

2.02.09 Select the recommended ram settings for the virtual device.

Android Studio

2.02.10 Wait for the components to download.

Android_010

2.02.11 take note of the virtual device name and click Finish.

Android_011

2.02.12 Android Studio software options.

Android_012

2.02.13 I searched in the Android Developer Guides and Using the Android Emulator guide to see where the Simulator was (no luck). I had to create a blank project to see where the virtual Android simulator was installed.

Android_013

2.02.14 Manually starting the Android Emulator.

  1. Open the Terminal
  2. Type “cd /Users/USERNAME/Library/Android/sdk/tools/”
  3. Type “./emulator -avd Nexus_5_API_23_x86 -netspeed full -netdelay none”

2.02.15 Listing Connected Development Devices.

You may have troubles actually getting files onto your Android 4.x or 5.x devices as some distributions of Android block USB mode.  You may need to enable running of apps from unknown sources (Settings -> Security).

  1. Open the Terminal
  2. Type “cd /Users/USERNAME/Library/Android/sdk/platform-tools”
  3. Type “./adb devices”

2.02.16 How to copy new apps to the device via Command Line (if you do not want to use vendor software to transfer files)

To be continued (Android guide is under construction).

3. Windows phone setup

Guide not available yet.

4. Where to now?

Review the Corona API documentation, hang about the forums, follow Corona on social media, watch the Corona Geeks video cast..

Install the Corona Editor into the Sublime Text Editor (guide here)

Now Code 🙂

crona-sublime

Donate and make this blog better




Ask a question or recommend an article
[contact-form-7 id=”30″ title=”Ask a Question”]

Filed Under: Apple, CoronaLabs, Development, mobile app Tagged With: Android, Apple, Corona Labs, Developer, iOS

Primary Sidebar

Poll

What would you like to see more posts about?
Results

Support this Blog

Create your own server today (support me by using these links

Create your own server on UpCloud here ($25 free credit).

Create your own server on Vultr here.

Create your own server on Digital Ocean here ($10 free credit).

Remember you can install the Runcloud server management dashboard here if you need DevOps help.

Advertisement:

Tags

2FA (9) Advice (17) Analytics (9) App (9) Apple (10) AWS (9) Backup (21) Business (8) CDN (8) Cloud (49) Cloudflare (8) Code (8) Development (26) Digital Ocean (13) DNS (11) Domain (27) Firewall (12) Git (7) Hosting (18) IoT (9) LetsEncrypt (7) Linux (21) Marketing (11) MySQL (24) NGINX (11) NodeJS (11) OS (10) Performance (6) PHP (13) Scalability (12) Scalable (14) Security (45) SEO (7) Server (26) Software (7) SSH (7) ssl (17) Tech Advice (9) Ubuntu (39) Uncategorized (23) UpCloud (12) VM (45) Vultr (24) Website (14) Wordpress (25)

Disclaimer

Terms And Conditions Of Use All content provided on this "www.fearby.com" blog is for informational purposes only. Views are his own and not his employers. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. Never make changes to a live site without backing it up first.

Advertisement:

Footer

Popular

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • I moved my domain to UpCloud (on the other side of the world) from Vultr (Sydney) and could not be happier with the performance.
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Add Google AdWords to your WordPress blog

Security

  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Setup two factor authenticator protection at login on Ubuntu or Debian
  • Using the Yubico YubiKey NEO hardware-based two-factor authentication device to improve authentication and logins to OSX and software
  • Setting up DNSSEC on a Namecheap domain hosted on UpCloud using CloudFlare
  • Set up Feature-Policy, Referrer-Policy and Content Security Policy headers in Nginx
  • Securing Google G Suite email by setting up SPF, DKIM and DMARC with Cloudflare
  • Enabling TLS 1.3 SSL on a NGINX Website (Ubuntu 16.04 server) that is using Cloudflare
  • Using the Qualys FreeScan Scanner to test your website for online vulnerabilities
  • Beyond SSL with Content Security Policy, Public Key Pinning etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Run an Ubuntu VM system audit with Lynis
  • Securing Ubuntu in the cloud
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider

Code

  • How to code PHP on your localhost and deploy to the cloud via SFTP with PHPStorm by Jet Brains
  • Useful Java FX Code I use in a project using IntelliJ IDEA and jdk1.8.0_161.jdk
  • No matter what server-provider you are using I strongly recommend you have a hot spare ready on a different provider
  • How to setup PHP FPM on demand child workers in PHP 7.x to increase website traffic
  • Installing Android Studio 3 and creating your first Kotlin Android App
  • PHP 7 code to send object oriented sanitised input data via bound parameters to a MYSQL database
  • How to use Sublime Text editor locally to edit code files on a remote server via SSH
  • Creating your first Java FX app and using the Gluon Scene Builder in the IntelliJ IDEA IDE
  • Deploying nodejs apps in the background and monitoring them with PM2 from keymetrics.io

Tech

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Is OSX Mojave on a 2014 MacBook Pro slower or faster than High Sierra
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • The case of the overheating Mac Book Pro and Occam’s Razor
  • Useful Linux Terminal Commands
  • Useful OSX Terminal Commands
  • Useful Linux Terminal Commands
  • What is the difference between 2D, 3D, 360 Video, AR, AR2D, AR3D, MR, VR and HR?
  • Application scalability on a budget (my journey)
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.

Wordpress

  • Replacing Google Analytics with Piwik/Matomo for a locally hosted privacy focused open source analytics solution
  • Setting web push notifications in WordPress with OneSignal
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..
  • Check the compatibility of your WordPress theme and plugin code with PHP Compatibility Checker
  • Add two factor auth login protection to WordPress with YubiCo hardware YubiKeys and or 2FA Authenticator App
  • Monitor server performance with NixStats and receive alerts by SMS, Push, Email, Telegram etc
  • Upgraded to Wordfence Premium to get real-time login defence, malware scanner and two-factor authentication for WordPress logins
  • Wordfence Security Plugin for WordPress
  • Speeding up WordPress with the ewww.io ExactDN CDN and Image Compression Plugin
  • Installing and managing WordPress with WP-CLI from the command line on Ubuntu
  • Moving WordPress to a new self managed server away from CPanel
  • Moving WordPress to a new self managed server away from CPanel

General

  • Backing up your computer automatically with BackBlaze software (no data limit)
  • How to back up an iPhone (including photos and videos) multiple ways
  • US v Huawei: The battle for 5G
  • Using the WinSCP Client on Windows to transfer files to and from a Linux server over SFTP
  • Connecting to a server via SSH with Putty
  • Setting web push notifications in WordPress with OneSignal
  • Infographic: So you have an idea for an app
  • Restoring lost files on a Windows FAT, FAT32, NTFS or Linux EXT, Linux XFS volume with iRecover from diydatarecovery.nl
  • Building faster web apps with google tools and exceed user expectations
  • Why I will never buy a new Apple Laptop until they fix the hardware cooling issues.
  • Telstra promised Fibre to the house (FTTP) when I had FTTN and this is what happened..

Copyright © 2023 · News Pro on Genesis Framework · WordPress · Log in

Some ads on this site use cookies. You can opt-out if of local analytics tracking by scrolling to the bottom of the front page or any article and clicking "You are not opted out. Click here to opt out.". Accept Reject Read More
GDPR, Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT