Simon Fearby

Views are personal and not my employers.

Header Right

Development Blog

Coding for fun since 1996, Learn by doing and sharing.

 

Advertisement:

Main navigation

Backup

How to backup an Ubuntu VM in the cloud via crontab entries that trigger Bash Scripts, SSH, rsync and email backup alerts

by 2 Comments

Here is how I backup a number of Ubuntu servers with crontab entries, bash scripts and rsync and send backup email.

Advertisement:



Read more on useful terminal commands here for as low as $2.5 a month. Read on setting up a Digital Ocean Ubuntu server here for as low as $5 a month here ($10 free credit). Read more on setting up an AWS Ubuntu server here.

I have  6 numbered scripts in my scripts folder that handle backups, I call these scripts at set times via the crontab list.

fyi: Paths below have been changed for the purpose of this post (security).

https://crontab.guru/ is great for specifying times to run jobs on each server (I backup one server at 1 AM,, another at 2 AM etc (never at the same time))

Bring up your crontab list

Check out the Crontab schedule generator here.

Below is the contents of my /scripts/0.backupfiles.sh (sensitive information removed).

I use this script to backup folders and configuration data

Below is the contents of my /scripts-folder/1.backupdbs.sh (sensitive information removed).

I use this script to dump my MySQL database.

Below is the contents of my /scripts-folder/2.shrinkmysql.sh (sensitive information removed).

I use this script to tar my SQL dumps as these files can be quite big

Below is the contents of my /scripts-folder/3.addtobackuplog.sh (sensitive information removed).

This script is handy for dumping extra information.

Below is the contents of my /scripts-folder/4.syncfiles.sh (sensitive information removed).

This script is the workhorse routine that rsyncs files to the source to the backup server (a dedicated Vulr server with an A Name record attaching the server to my domain).

I installed sshpass to pass in the ssh user password (after ssh is connected (authorized_keys set), I tried to setup a rsync daemon but had no luck).  I ensured appropriate ports were opened on the source (OUT 22, 873) and backup server (IN 22 873).

ufw firewall has great rules for allowing certain IP’s to talk on ports.

Set Outbound firewall rules (to certain IP’s)

Change 123.123.123.123 to your backup server.

Set Inbound firewall rules (to certain IP’s)

Change 123.123.123.123 to your sending server.

You can and should setup rate limits on IP’s hitting certain ports.

Install Fail2Ban to automatically ban certain users. Fail2Ban reads log file that contains password failure report
and bans the corresponding IP addresses using firewall rules.  Read more on securing Ubuntu in the cloud here.

Below is the contents of my /scripts-folder/5.sendbackupemail.sh (sensitive information removed).

This script sends an email and attaches a zip file of all log files generated through the backup process.

Read my guide on setting up sendmail here.

Security Considerations

You should never store passwords in scripts that talk to SSH connections, create MySQL dumps or when talking to email servers, I will update this guide when I solving all of these cases.  Also, create the least access required for user accounts where possible.

Target Server Configuration

Alos you can see in /scripts-folder/4.syncfiles.sh that I am saving to the ‘/backup-folder/1.www.server01.com/’ folder, you can make as many folders as you want to make the most of the backup server.  I would advise you not use the server for anything else like web servers and apps as this server is holding important stuff.

I have a handy script to delete all backups (handy during testing).

Ensure your backup server is just for backups and only allows traffic from known IP’s

Change the 123.x.x.x servers to your servers IP’s

Tip: Keep an eye on the backups with tools like ncdu

Installing SSH on OSX

If you want to backup to this server with OSyouou will need to install sshpass

sshpass should be installed

I have not got sshp[ass working yet error “Host key verification failed.”  I had to remove the back known host from ~/.ssh/known_hosts” on OSX

But this worked on OSX

Note: Enter the servers sshuser@ before the hostname or rsync will use the logged in OSX username

Don’t forget to check the backup serves disk usage often.

disk usage screenshot

Output from backing up an incremental update (1x new folder)

Update with no files to upload

Backup is easy..

If you want incremental and full backups try Duplicity.

Hope this helps.

 

 

Donate and make this blog better




Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.

v1.7 Duplicity

Moving WordPress to a new self managed server away from CPanel

by 6 Comments

I recently moved my domain from a C-Panel hosted domain (and Email to Google G Suite (my guide here)) to a self-managed Digital Ocean domain (my LetsEncrypt Guide here, my Digital Ocean guide here, my AWS setup guide here, my Vultr setup guide here) and needed to transfer my WordPress site.

I had issues with a CPanel domain on the hosts (Uber/Netregistry) as they were migrating domains and the NetRegstry chat rep said I needed to phone Uber for support. No thanks, I’m going self-managed and saving a dollar.

 

Advertisement:



I backed up my old WordPress site files with FTP (and my database (sql expor)t) on my old site and zipped it up and uploaded it to a public location as I did not want to setup FTP on my new site. More info on this here.

This downloaded mg 1GB backup zip.

I install unzip onto Ubuntu.

I unzipped the WordPress instance with Unzip (my blog lives under a blog folder).

I created an /index.html to redirect people to the blog folder.

I updated my NGINX configuration to ensure WordPress worked.

I checked my databases with mysql (command line) to ensure MySQL was up.

I uploaded https://www.adminer.org/ to my site. I opened my wp-config.php and got db and username and created the user and assigned it to the database in Adminer.

I had a SQL backup of my site buy from experience I had not modify the SQL file before importing it (replace MyISAM with InnoDB).

You You can read more on me previously moving a CPanel domain with email to a self managed VPS and Gmail.

I tried to import my 7mb sql file into the database with Adminer.

But it kept timing out (I suspect php upload limits were set to 2MB).

I fond my php.ini files (4 of them). No php file was found when I ran which php command was run and opened that folder.

I had to make the following PHP.ini changes to allow the larger file size restore uploads with the Adminer utility (default is 2mb).

I had to make the following changes to nginx.conf (to prevent 404 errors on the database upload)

I navigated to the WordPress and all was good (even had SSL as I set that up on the new server)..\\

Conclusion

I now have a faster WordPress on a server I manager with SSL.

Donate and make this blog better




Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.

v1.1 NetRegistry info

Installing Webmin Server Management web GUI on Ubuntu

by Leave a Comment

Setting up Ubuntu in the cloud is great if you are not afraid of using the command line but for those that need it, there are web based server management GUI’s you can install.

Advertisement:



Once Ubuntu is installed (I have many guides at www.fearby.com) you can install http://www.webmin.com/ (web based management GUI for Ubuntu).  It is similar to https://runcloud.io but it’s free. Read the Webadmin install guide here.

Ubuntu Install Steps

Run this in your root terminal to install Webmin on Ubuntu after you install and configure a web server, MySQL and SSL.

Allow port 1000 on your firewall (in my case I am limiting this to my public IP).

I opened port 10,000 on my Digital Ocean firewall screen too.

I installed an SSL certificate on my server (my guide here).  I am having issues with Google Chrome and my SSL certificate so I switched ot safari.

I navigated to https://mydomain.com:10000 to log in to the Webmin dashboard

I could now log in to Webmin so I reset the password???

Now I can login to Webmin

Webmin has a  load of options available, I zeroed in on viewing log files.

I think I need to ban a few IP’s in my firewall based on the login attempts above. View my guide on securing Ubuntu in the cloud.

Done

Webmin has a great section on Ubuntu Users

View User Details

Securing Webmin Configuration then IP Access Control and whitelist your IP address to ensure no one else can view your server (I went to http://icanhazip.com to get my IP address).

I then used the Webmin backup screen to backup my system.

This is great, it is backing up my system 🙂

I used pydf to see how much storage space I have (I am only using 2.65GB but have 17GB available).

Gzip is taking a load of time to backup my system.

I did not need to use (htop) to monitor the CPU, Webmin has a CPU monitor built in.

Webmin also allows you to add users

Beware though you need to assign a user to modules or they won’t be able to do/see much when they login.

If you provide your MySQL root password you can create databases in Webmin under Servers then MySQL Database Server

Loads of database options that I would normally use Adminer for 🙂

File Browser

There is a file browser but (I can’t seem to drag and drop).

Official documentation is found here. I’ll add more to this post as I discover it.

Check out my other posts at www.fearby.com.

 

Donate and make this blog better




Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.

v1.0 Initial Post

Securing Ubuntu in the cloud

by 10 Comments

It is easy to deploy servers to the cloud within a few minutes, you can have a cloud-based server that you (or others can use). ubuntu has a great guide on setting up basic security issues but what do you need to do.

Advertisement:



If you do not secure your server expect it to be hacked into. Below are tips on securing your cloud server.

First, read more on scanning your server with Lynis security scan.

Always use up to date software

Always use update software, malicious users can detect what software you use with sites like shodan.io (or use port scan tools) and then look for weaknesses from well-published lists (e.g WordPress, Windows, MySQL, node, LifeRay, Oracle etc). People can even use Google to search for login pages or sites with passwords in HTML (yes that simple).  Once a system is identified by a malicious user they can send automated bots to break into your site (trying millions of passwords a day) or use tools to bypass existing defences (Security researcher Troy Hunt found out it’s child’s play).

Portscan sites like https://mxtoolbox.com/SuperTool.aspx?action=scan are good for knowing what you have exposed.

You can also use local programs like nmap to view open ports

Instal nmap

Find open ports

Limit ssh connections

Read more here.

Use ufw to set limits on login attempts

Only allow known IP’s access to your valuable ports

Delete unwanted firewall rules

Only allow known IP’s to certain ports

Also, set outgoing traffic to known active servers and ports

Don’t use weak/common Diffie-Hellman key for SSL certificates, more information here.

More info on generating SSL certs here and setting here and setting up Public Key Pinning here.

Intrusion Prevention Software

Do run fail2ban: Guide here https://www.linode.com/docs/security/using-fail2ban-for-security

I use iThemes Security to secure my WordPress and block repeat failed logins from certain IP addresses.

iThemes Security can even lock down your WordPress.

You can set iThemes to auto lock out users on x failed logins

Remember to use allowed whitelists though (it is so easy to lock yourself out of servers).

Passwords

Do have strong passwords and change the root password provided by the hosts. https://howsecureismypassword.net/ is a good site to see how strong your password is from brute force password attempts. https://www.grc.com/passwords.htm is a good site to obtain a strong password.  Do follow Troy Hunt’s blog and twitter account to keep up to date with security issues.

Configure a Firewall Basics

You should install a firewall on your Ubuntu and configure it and also configure a firewall with your hosts (e.g AWS, Vultr, Digital Ocean).

Configure a Firewall on AWS

My AWS server setup guide here. AWS allow you to configure the firewall here in the Amazon Console.

Type Protocol Port Range Source Comment
HTTP TCP 80 0.0.0.0/0 Opens a web server port for later
All ICMP ALL N/A 0.0.0.0/0 Allows you to ping
All traffic ALL All 0.0.0.0/0 Not advisable long term but OK for testing today.
SSH TCP 22 0.0.0.0/0 Not advisable, try and limit this to known IP’s only.
HTTPS TCP 443 0.0.0.0/0 Opens a secure web server port for later

Configure a Firewall on Digital Ocean

Configuring a firewall on Digital Ocean (create a $5/m server here).  You can configure your Digital Ocean droplet firewall by clicking Droplet, Networking then Manage Firewall after logging into Digital Ocean.

Configure a Firewall on Vultr

Configuring a firewall on Vultr (create a $2.5/m server here).

Don’t forget to set IP rules for IPV4 and IPV6, Only set the post you need to allow and ensure applications have strong passwords.

Ubuntu has a firewall built in (documentation).

Enable the firewall

Adding common ports

Add a whitelist for your IP (use http://icanhazip.com/ to get your IP) to ensure you won’t get kicked out of your server.

More help here.  Here is a  good guide on ufw commands. Info on port numbers here.

If you don’t have a  Digital Ocean server for $5 a month click here and if a $2.5 a month Vultr server here.

Backups

rsync is a good way to copy files to another server or use Bacula

Basics

Initial server setup guide (Digital Ocean).

Sudo (admin user)

Read this guide on the Linux sudo command (the equivalent if run as administrator on Windows).

Users

List users on an Ubuntu OS (or compgen -u)

Common output

Add User

e.g

Add user to a group

Show users in a group

This will show users in a group

Remove a user

Rename user

Change user password

Groups

Show all groups

Common output

You can create your own groups but first, you must be aware of group ids

Then you can see your systems groups and ids.

Create a group

Permissions

Read this https://help.ubuntu.com/community/FilePermissions

How to list users on Ubuntu.

Read more on setting permissions here.

Chmod help can be found here.

Install Fail2Ban

I used this guide on installing Fail2Ban.

Check Fail2Ban often and add blocks to the firewall of known bad IPs

Best practices

Ubuntu has a guide on basic security setup here.

Startup Processes

It is a good idea to review startup processes from time to time.

Accounts

  • Read up on the concept of least privilege access for apps and services here.
  • Read up on chmod permissions.

Updates

Do update your operating system often.

Minimal software

Only install what software you need

Exploits and Keeping up to date

Do keep up to date with exploits and vulnerabilities

Secure your applications

  • NodeJS: Enable logging in applications you install or develop.

Ban repeat Login attempts with FailBan

Fail2Ban config

Hosts File Hardening

Add

Add a whitelist with your ip.

Restart the service

Intrusion detection (logging) systems

Tripwire will not block or prevent intrusions but it will log and give you a heads up with risks and things of concern

Install Tripwire.

Running Tripwire

This will scan your system for issues of note

My Output.

More on Tripwire here.

Hardening PHP

Hardening PHP config (and backing the PHP config it up), first create an info.php file in your website root folder with this info

Now look for what PHP file is loadingPHP Config

Back that your PHP config file

TIP: Delete the file with phpinfo() in it as it is a security risk to leave it there.

TIP: Read the OWASP cheat sheet on using PHP securely here and securing php.ini here.

Some common security changes

Don’t forget to review logs, more config changes here.

Antivirus

Yes, it is a good idea to run antivirus in Ubuntu, here is a good list of antivirus software

I am installing ClamAV as it can be installed by the command line and is open source.

ClamAV help here.

Scan a folder

Setup auto update antivirus definitions

I set auto updates 24 times a day (every hour) via daemon updates.

tip: Download manual antivirus update definitions. If you only have a 512MB server your update may fail and you may want to stop fresh claim/php/nginx and mysql before you update to ensure the antivirus definitions update. You can move this to a con job and set this to update at set times over daemon to ensure updates happen.

Manual scan with a bash script

Create a bash script

Bash script contents to update antivirus definitions.

Edit the crontab to run the script every hour

Setup Unattended Ubuntu Security updates

At login, you should receive

Other

  • Read this awesome guide.
  • install Fail2Ban
  • Do check your log files if you suspect suspicious activity.

Donate and make this blog better




Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.

v1.9 added more hardening info

How to develop software ideas

by 3 Comments

I was recently at a public talk by Alan Jones at the UNE Smart Region Incubator where Alan talked about launching startups and developing ideas.

Advertisement:



Alan put it quite eloquently that “With change comes opportunity” and we are all very capable of building the next best thing as technological barriers and costs are a lot lower than 5 years ago but Alan also mentioned 19 start-ups-ups fail but “if you focus on solving customer problems you have a better chance of succeeding”. Regions need to share knowledge and you can a learn from others peoples mistakes.”

I was asked after this event to share thoughts on “how do I learn to develop an app” and “how do you get the knowledge”. Here is my poor “brain dump” on how to develop software ideas (It’s hard to condense 30 years experience developing software). I will revise this post over the coming weeks so check back often.

If you have never programmed before check out this programming 101 guide here.

I have blogged on technology/knowledge things in the past at www.fearby.com and recently I blogged about how to develop cloud-based services (here, here, here, here and here) but this blog post assumes you have a validated “app idea” and you want to know how to develop yourself. If you do not want to develop an app yourself you may want to speak with Blue Chilli.

Find a good mentor.


True App Development Quotes

  • Finding development information is easy, following a plan is hard.
  • Aim for progress and not perfection.
  • Learn one thing at a time (Multitasking can kill your brain).
  • Fail fast and fail early and get feedback as early as possible from customers.
  • 10 engaged customers are better than 10,000 disengaged users.

And a bit of humour before we start.

(click for larger image)

Here is a funny video on startup/entrepreneur life/lingo


This is a good funny, open and honest video about programming on YouTube.

Follow Seth F Samuel on twitter here.

Don’t be afraid to learn from others before you develop

My fav tips from over 200 failed startups (from https://www.cbinsights.com/blog/startup-failure-post-mortem/ )

  • Simpler websites shouldn’t take more than 2-3 months.You can always iterate and extrapolate later. Wet your feet asap
  • As products became more and more complex, the performance degrades. Speed is a feature for all web apps. You can spend hundreds of hours trying to speed of the app with little success. Benchmarking tools incorporated into the development cycle from the beginning is a good idea
  • Outsource or buy in talent if you don’t know something (e.g marketing). Time is money.
  • Make an environment where you will be productive. Working from home can be convenient, but often times will be much less productive than a separate space. Also its a good idea to have separate spaces so you’ll have some work/life balance.
  • Not giving enough time to stress and load testing or leaving it until the last minute is something startups are known for — especially true of small teams — but it means things tend to get pretty tricky at scale, particularly if you start adding a user every four seconds.
  • It’s possible to make a little money from a lot of people, or a lot of money from a few people. Making a little money from a few people doesn’t add up. If you’re not selling something, you better have a LOT of eyeballs. We didn’t.
  • We received conflicting advice from lots of smart people about which is more important. We focused on engagement, which we improved by orders of magnitude. No one cared. Lesson learned: Growth is the only thing that matters if you are building a social network. Period. Engagement is great but you aren’t even going to get the meeting unless your top-line numbers reach a certain threshold (which is different for seed vs. series A vs. selling advertising).
  • We most definitely committed the all-too-common sin of premature scaling. Driven by the desire to hit significant numbers to prove the road for future fundraising and encouraged by our great initial traction in the student market, we embarked on significant work developing paid marketing channels and distribution channels that we could use to demonstrate scalable customer acquisition. This all fell flat due to our lack of product/market fit in the new markets, distracted significantly from product work to fix the fit (double fail) and cost a whole bunch of our runway.
  • If you’re bootstrapping, cashflow is king. If you want to possibly build a product while your revenue is coming from other sources, you have to get those sources stable before you can focus on the product.
  • Don’t multiply big numbers. Multiply $30 times 1.000 clients times 24 months. WOW, we will be rich! Oh, silly you, you have no idea how hard it is to get 1.000 clients paying anything monthly for 24 months. Here is my advice: get your first client. Then get your first 10. Then get more and more. Until you have your first 10 clients, you have proved nothing, only that you can multiply numbers.
  • Customers pay for information, not raw data. Customers are willing to pay a lot more for information and most are not interested in data. Your service should make your customers look intelligent in front of their stakeholders. Follow up with inactive users. This is especially true when your service does not give intermediate values to your users. Our system should have been smarter about checking up on our users at various stages.
  • Do not launch a startup if you do not have enough funding for multiple iterations. The chances of getting it right the first time are about the equivalent of winning the lotto.

Here are my tips on staying on track developing apps. What is the difference between a website, app, API, web app, hybrid app and software (my blog post here)?

I have seen quite a few projects fail because:

  • The wrong technology was mandated.
  • The software was not documented (by the developers).
  • The software was shelved because new developers hated it or did not want to support it.

Project Roles (hats)

It is important to understand the roles in a project (project management methodology aside) and know when you are being a “decision maker” or a “technical developer”. A project usually has these roles.

  • Sponsor/owner (usually fund the project and have the final say).
  • Executive/Team leader/scrum master (manage day to day operations, people, tasks and resources).
  • Team members (UI, UX, Marketers, Developers (DevOps, Web, Design etc) are usually the doers.
  • Stakeholders (people who are impacted (operations, owners, Helpdesk)).
  • Subject Matter Experts (people who should guide the work and not be ignored).
  • Testers (people who test the product and give feedback).

It can be hard as a developer to switch hats in a one person team.

How do you develop and gain knowledge

First, document what you need to develop (what problem are you solving and what value will your idea bring). Does this solution exist already? Don’t solve a problem that already exists.

Developing software is not hard, you just need to be logical, research, be patient and follow a plan. The hardest part can be gluing components together.

I like to think of developing software like making a car if you need 4 wheels do you have 4 wheels? If you want to build it yourself and save some money can you make wheels (make rubber strips with steel reinforced/vulcanized rubber, make alloys and add bearings and have them pass regulations) or should you buy wheels (some things are cheaper to make than other things)? Developing software can be easy if you know what your are doing and have the experience and are aware of the costs and risks.  Developing software can lead you down a rabbit hole of endless research, development, and testing if you don’t know what you are doing.

Examples 1:

I “need a webpage”:

Example 2:

I “need an iPhone and Android app”:

Research: You will need to learn iOS and Android programming and you may need a server or two to hold the apps data, webpage and API. You will also need to setup and secure the servers or choose to install a database or go with a “database as a service” like cloud.mongodb.com or google firebase.

Money can buy anything (but will it be flexible/cheap enough), time can build anything (but will it be secure enough).

Developing software can be easy if you know what your are doing and have the experience and are aware of the costs and risks but developing software can lead you down a rabbit hole of endless research, development and testing if you don’t know what you are doing.

Almost all systems will need a central database to store all data, you can choose a traditional relational SQL database or a newer NoSQL database. MySQL is a good/cheap relational SQL database and MongoDB is a good NoSQL database. You will need to decide on how your app talks to the database (directly or via an API (protected by OAuth or limited access tokens)).  It is a bad idea to open a database directly to the world with no security. Sites like www.shodan.io will automatically scan the Internet looking for open databases or systems and report this as an insecure site to anyone. It is in your interest to develop secure systems in all stages of development.

CRUD (Create, Read, Update and Delete) is a common group of database tasks that you can do to prove you can read, write, update and delete from a database. While performing CRUD operations is a good to benchmark to also see how fast the database it.  if a database is the slowest link then you can use memory to cache database values (read my guide here). Caching can turn a cheap server into a faster server. Learning by doing can quickly build skills so “research”, “do” and “learn”.

Most solutions will need a website (and a web server). Here is a good article comparing Apache and Nginx (the leading open source web servers).

Stacks and Technology – There are loads of development environments (stacks), frameworks and technologies that you can choose. Frameworks supposedly make things easier and faster but frameworks and technologies change (See 2016 frameworks to learn guide and 2017 frameworks to learn guide) frequently (and can be abandoned). Frameworks supposedly make things easier and faster but be careful most frameworks run 30% slower than raw server side and client code. I’d recommend you learn a few technologies like NGINX, NodeJS, PHP and MySQL and move up from there.

The Mean Stack is a  popular web development platform (MEAN = MongoDB, ExpressJS, Angular and NodeJS.).

Apps can be developed for Apple platforms by signing up here (about $150 AUD a year) and using the XCode IDE. Apps can be developed for the Android Platform by using Android Studio (for about $20 (one off fee)). Microsoft has a developer portal for the Windows Platform. Google also has an online scalable database as a service called Firebase. If you look hard enough you will find a service for everything but connecting those services can be timely, costly or make security and a scalable solution impossible so beware of using as-a-service platforms. I used the Corona SDK to develop an app but abandoned the platform due to changes in the vendor’s communication and enforced policies.

If you are not sure don’t be afraid of ask for help on Twitter.

Twitter is awesome for finding experts

Recent twitter replies to a problem I had.

Learning about new Technology and Stacks

To build knowledge you need to learn stuff, build stuff, test (benchmark), get feedback and build more stuff. I like to learn about new technology and stacks by watching Udemy courses and they have a huge list of development courses (Web Development, Mobile Apps, Programming Languages, Game Development, Databases,  Software Testing,  Software Engineering etc).

I am currently watching a Practical iOS 11 course by Stephen DeStefano on Udemy to learn about unreleased/upcoming features on the Apple iPhone (learning about XCode 9, Swift 4, Whats new in iOS 11, Drag and drop, PDF and ARKit etc).

Udemy is awesome (Udemy often have courses for $15).

If you want to learn HTML go to https://www.w3schools.com/.

https://devslopes.com/have a number or development related courses and an active community of developers in a chat system.

You can also do formal study via an education provider (e.g. Bachelor of computer sciences at UNE or Certificate IV in programming or Diploma in Software Development at TAFE).

I would recommend you use Twitter and follow keywords (hashtags) around key topics (e.g #www, #css, #sql, #nosql, #nginx, #mongodb, #ios, #apple, #android, #swift, #objectivec, #java, #kotlin) and identify users to follow. Twitter is great for picking up new information.

I follow the following developers on YouTube (TheSwiftGuy, AppleProgrammer, AwesomeTuts, LetsBuildThatApp, CodingTech etc)

Companies like https://www.civo.com/ offer developer friendly features with hosting, https://www.pebbled.io/ offer to develop for you and https://serverpilot.io/ help you spin up software on hosting providers.

What To Develop

First, you need to break down what you need. (e.g ” I want an app for iOS and Android in 5 months that does XYZ. The app must be secure and be fast. Users must be able to register an account and update their profile”).

Choosing how high to ensure you development project scales depends on your peak expected/active concurrent users (ratio of paying and free users). You can develop your app to scale very high but this may cost more money initially, it can be bad to pay to ensure scalability early. As long as you have a good product and robust networking/retry routines and UI you don’t need to scale high early.

Once you know what you need you can search the open source community for code that you can use. I use Alamofire for iOS network requests, SwiftyJSON for processing JSON data and other open source software. The only downside of using open source software is it may be abandoned by the creators and break in the future. Saving you time early may cost you time later.

Then you can break down what you don’t want. (e.g “I don’t want a web app or a windows phone or windows desktop app”). From here you will have a list of what you need and what you can avoid.

You will also need to choose a project management methodology (I have blogged about this here). Having a list of action item’s and a plan and you can work through developing your app.

While you are researching it is a good idea to develop smaller fun projects to refine your skills.  There are a number of System Development Life Cycles (SDLC’s) but don’t worry if you get stuck, seek advice or move on. It is a  good idea to get users beta testing your app early and seek feedback. Apple has the TestFlight app where you can send beta versions of apps to best testers. Here is a good guide on Android beta testing.

If you are unsure about certain user interface options or features divide your beta testers and perform A/B or split testing to determine the most popular user interfaces. Capturing user data and logs can also help with debugging and user usage actions.

Practice

Develop smaller proof of concept apps in new technologies or frameworks and you will build your knowledge and uncover limitations in certain frameworks and how to move forward with confidence. It is advisable to save your source code for later use and to share with others.

I have shared quite a bit of code at https://simon.fearby.com/blog/ that I refer to from time to time. I should have shared this on GitHub but I know Google will find this if people want it.

Get as much feedback as you can on what you do and choose (don’t trust the first blog post you read (me included)).

Most companies offer Webinars on their products. I like the NGINX webinars. Tutorialspoint have courses on development topics. Sitepoint is a  good development site that offers free books, courses, and articles. What are API’s information by Programmable web.

You may want to document your application flow to better understand how the user interface works.

Useful Tools

Balsamic Mockups and Blueprint are handy for mocking up applications.

C9.io is a great web-based IDE that can connect to a VM on AWS or Digital Ocean.  I have a guide here on connecting Cloud 9 to an AWS VM here.

I use the Sublime Text 3 text editor when editing websites locally.

(image courtesy of https://www.sublimetext.com/ )

I use the Mac Paw app to help test API’s I develop locally.

(image courtesy of https://paw.cloud )

Snippets is a great application for the Mac for storing code snippets.

I use the Cornerstone Subversion app for backing up my code on my Mac.

Webservers: https://www.iis.net/IIS Webserver, NGINX Webserver, Apache Webserver.

NodeJS programming manual and tutorials.

I use Little Snitch (guide here) for simulating network down in app development.

I use the Forklift file manager on OSX.

Databases: SQL tutorials, NoSQL Tutorials, MySQL documentation.

Siege is a command line HTTP load testing tool.

http://loader.io/ is a nice web-based benchmarking tool.

Bootstrap is an essential mobile responsive framework.

Atlassian Jira is an essential project tracking tool. More on Agile Epics v Stories v Tasks on the Atlassian community website here. I have a post on developing software and staying on track here using Jira.

Jsfiddle is a good site that allows you to share code you are working on or having trouble with.

Dribbble is a “show and tell” site for designers and creatives.

Stackoverflow is the go-to place to ask for help.

Things I care about during development phases.

  • Scalability
  • Flexibility
  • Risk
  • Cost
  • Speed

Concentrating too much on one facet can risk exposing other facets. Good programmers can recommend a deliver a solution that can be strong in all areas ( I hate developing apps that are slow but secure or scalable and complex).

Platforms

You can signup for online servers like Azure, AWS (my guide here) or you can use a cheaper CPanel based hosting. Read my guide on the costs of running a cloud-based service.

Use my link to get a free Digital Ocean server for two months by using this link. Read my blog post here to help setup you VM. You can always use Ubuntu on your local machine to use Ubuntu (read my guide here). Don’t forget to use a GIT code repository like GitHub or Bitbucket.

Locally you can install Ubuntu (developers edition) and have a similar environment as cloud platforms.

Lessons Learned

  • Deploy servers close to the customers (Digital Ocean is too far away to scale in Australia).
  • Accessibility and testing (make things accessible from the start).
  • Backup regularly (Use GIT, backup your server and use Rsync to copy files to remote servers and use services like backblaze.com to backup your machine).
  • Transportability of technology (Use open technology and don’t lock yours into to one platform or service).
  • Cost (expensive and convenient solutions may be costly).
  • Buy in themes and solutions (wrapbootstrap.com).
  • Do improve what you have done (make things better over time). Thing progress and not perfection.

There is no shortage of online comments bagging certain frameworks or platforms so look for trends and success stories and don’t go with the first framework you find. Try candidate frameworks and services and make up your own mind.

A good plan, violently executed now, is better than a perfect plan next week. – General George S. Patton

Costs

Sometimes cost is not the deciding factor (read my blog post on Alibaba cloud). You should estimate your apps costs per 1000 users. What do light v heavy users cost you? I have a blog post on the approx cost of cloud services.  I started researching a scalable NoSQL platform on IBM Cloudant and it was going to cost $4,000 USD a month and integrating my own App logic and security was hard. I ended up testing MongoDB Cloud where I can scale to three servers for $80 a month but for now, I am developing my current project on my own AWS server with MongoDB instance. Read my blog post here on setting up MongoDB and read my blog post on the best MongoDB GUI.

Here is a great infographic for viewing what’s involved in mobile app development.

You can choose a number of tools or technologies to achieve your goals, for me it is doing it economically, securely and in a scalable way that has predictable costs. It is quite easy to develop something that is costly, won’t scale or not secure or flexible. Don’t get locked into expensive technologies. For example, AWS has a user pays Node JS service called Lambada where you get Million of free hits a month and then you get charged $0.0000002 per request thereafter. This sounds good but I prefer fixed pricing/DIY servers better as it allows me to build my own logic into apps (this is more important than scalability).

Using open source software of off the shelf solutions may speed things up initially? Will It slow you down later though? Ensure free solutions are complete and supported and Ensure frameworks are helping. Do you need one server or multiple servers (guide on setting up a distributed MySQL environment )? You can read about my scalability on a budget journey here. You can speed up a server in two ways Scale Up (Add more Mhz or CPU cores) or scale out (add more servers).

Start small and use free frameworks and platforms but have a tested scale up plan, I researched cheap Digital Ocean servers and moved to AWS to improve latency and tested MongoDB on Digital Ocean and AWS but have a plan to scale up to cloud.mongodb.com if need be.

Outsource (contractors) 

Remember outsourcing work tasks (or complete outsourcing of development) can buy you time and or deliver software faster. Outsourcing can also introduce risks and be expensive. Ask for examples of previous work and get raw numbers on costs (now and in the future) and concurrent users that a particular bit of outsourcing work will achieve.

If you are looking to outsource work do look at work that the person or company has done before (if is fast, compliant, mobile scalable, secure, robust, backup up, do you have rights to edit/own and own the IP etc). I’d be cautious of companies who say they can do everything and don’t show live demos.

Also, beware of restrictions on your code set by the contractors. Can they do everything you need (compare with your list of Moscow must haves)? Sometimes contractors only code or do what they are comfortable with that can impact your deliverables.

Do use a private Git repository (that you own) like GitHub or BitBucket to secure your code and use software like Trello or Atlassian JIRA to track your project. Insist the contractors use your repository to retain control.

You can always sell equity in your idea to an investor and get feedback/development from companies like Bluechilli.

Monetization and data

Do have multiple monetization streams (initial app purchase cost, in-app purchase, subscriptions, in-app credit, advertising, selling code/components etc). Monthly revenue over yearly subscription works best to ensure cash flow.

Capture usage data and determine trends around successful engagement, Improve what works. Use A/B testing to roll our new features.

I like Backblaze post on getting your first 1,000 customers.

Maintenance, support risk and benefits

Building your own service can be cheaper but also riskier, if you fail to secure an app you are in trouble if you cannot scale you are in trouble. If you don’t update your server when vulnerabilities come out you are in trouble. Also, Google on monetization strategies. Apple apps do appear to deliver more profits over Android. Developers often joke “Apple devices offer 90% of the profits and 10% of the problems and Android apps offer 90% of the problems and 10% of the profits”.

Also, Apple users tend to update to the latest operating system sooner where Android devices are rather fragmented.

Do inform you users with self-service status pages and informative error messages and don’t annoy users.

Use Free Trials and Credit

Most vendors have free trials so use them

https://aws.amazon.com/free/AWS have 12 month free tiers.

Use this link to get two months free with Digital Ocean.

Microsoft Azure also give away free credit.

Google cloud also have free credit.

Don’t be afraid to ask.

MongoDB Cloud also gives away free credit if you ask.

Security

Sites like Shodan.io will quickly reveal weaknesses in your server (and services), this will help you build robust solutions from the start before hackers find them. Read https://www.owasp.org/index.php/Main_Page to know h0w to develop secure websites. Listen to the SecurityNow podcast to learn how the technology works and is broken. Following TroyHunt is recommended to keep up to date with security in general. @0xDUDE is a good ethical hacker to follow to stay up-to date on security exploits also @GDI_FDN is a good non-profit organization that helps defend sites that use open source software.

White hack hackers exist but so do black hat ones.

Read the Open Web Application Security site here. Read my guide on setting up public key pinning in security certificates here.

I use the ASafaWeb site to test your sites from common ASP security flaws. If you have a secure certificate on your site you will need to ensure the certificate is secure and up to date with the SSL Labs SSL Test site.

Once your websites IP address is known (get it from SSL Labs) run a scan over your site with https://www.shodan.io/ to find open ports or security weaknesses.

Shodan.io allows you and others to see public information about your server and services. You can read about well-known internet ports here.

Anyone can find your server if you are running older (or current) web servers and or services.

It is a  good idea to follow security researchers like Steve Gibson and Troy Hunt and stay up to date with live exploits. http://blog.talosintelligence.com is also a good site for reading technical breakdowns of exploits.

Networking

Do share and talk about what you do with other developers. You can learn a lot from other developers and this can save you loads of time and mistakes. True developers love talking about their code and solutions.

Decision Making

Quite a lot of time can be spent on deciding on what technology or platform to use, I decide by factoring in cost, risk and security over flexibility, support and scalability. If I need flexibility, lower support or scalability then I’ll choose a different technology/platform. Generally, technology can help with support. Scalable solutions need effort from start to finish (it is quite easy to slow down any technology or service).

Don’t be afraid to admit you have chosen the wrong technology or platform. It is far easier to research and move on than live with poor technology.

If you have chosen the wrong technology and stick with it, you (and others) will loath working with it (impacting productivity/velocity).  Do you spend time swapping technology or platforms now or be less productive later?

Intellectual property and Trademarks

Ensure you search international trademarks for your app terms before you start using them. The Australian ATO has a good Australian business name checker here.

https://namechk.com/ is also a good place to search for your app ideas name before you buy or register any social media accounts.

Using https://namechk.com/ you can see “mystartupidea” name is mostly free.

And the name “microsoft’ is mostly taken.

Seek advice from a start-up experts from https://www.bluechilli.com/ like Alan Jones.

See my guide on how to get useful feedback for your ideas here.

Tips

  1. Use Git Source Control systems like GitHub or Bitbucket from the start and offsite backup your server and environments frequently. Digital Ocean charges 20% of your servers costs to back it up. AWS has multiple backup offerings.
  2. Start small and scale up when needed.
  3. Do lots of research and test different platforms, frameworks, and technologies and you will know what you should choose to develop with.

(Image above found at http://startupquotes.startupvitamins.com/ Follow Startup Vitamins on Twitter here.).

 

You will know when you are a developer when you have gained knowledge and experience and can automatically avoid technologies that will not fit a  solution.

Share

Don’t be afraid to share what you know (read my blog post on this here). Sharing allows you to solidify your knowledge and get new information. Shane Bishop from EWWW Image Optimizer  WordPress plugin wrote Setting up a fast distributed MySQL environment with SSL for us. If you have something to share on here please let me know here on twitter.

It’s never too late to do

One final tip is knowledge is not everything, planning and research is key, a mind that can’t develop may be better than a mind that can because they have no experience (or baggage) and may find faster ways to do things. Thanks to http://zachvo.com/ for teaching me this during a recent WordPress re-deployment. Sometimes the simplest solution is.

Donate and make this blog better




Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.

DRAFT: 1.85 added white hack info (Saturday 22nd July 6:01pm AEST).

Alibaba Cloud how good is it?

by 1 Comment

I recently clicked an Alibaba Cloud add Banner on Facebook that said Alibaba Cloud offer a cloud server (with 40gb SSD, 1x CPU, 1GB Ram and 1TB traffic for $30 a year). I am familiar with Alibaba (Chinese AWS equivalent) and the owner Jack Ma (and his mindset). The thing that got me interested was hosting was available in Sydney (Something that Digital Ocean does not offer).  This hosting deal seems too good to be true. Digital Ocean has similar hosting plans but half the storage for $5 a month (but are not hosted in Australia). I asked Alibaba on Twitter (@alibaba_cloud) some questions last Sunday and got an answer on Friday evening over 5 days later.

Advertisement:



If You sign up for Alibaba Cloud here you will get $300 of New User Free Credit to use in 60 days.

or

Easily deploy an SSD cloud server on Digital Ocean in 55 seconds. Sign up using my link and receive $10 in credit (2 free months): https://www.digitalocean.com

Read my other guides

Alibaba Cloud

Alibaba seems to have similar products as Amazon Web Services.
alibaba001

After you click Create Free Account you will need to enter a credit or debit card.
alibaba002

You need to verify an email address and mobile number by verifying random tokens (that you will receive).
alibaba003

An email verify token will look like this.
alibaba004

When tour account is created you will be sent an email that states.

“Dear username,

Congratulations for successfully registering an Alibaba Cloud account! Your account ID is:____@ _____.___. Please remember to change your password on a regular basis and keep all account and password information secure.

Please remember to change your password on a regular basis and keep all account and password information secure.
Starting from today, you have 60 days to try and test Alibaba Cloud products with $300 New User Free Credit! The Free Credit will be automatically delivered to your account once you have successfully added a verified payment method. Follow this link to add your payment method and necessary information.”
alibaba005

You will need to add a billing address.
alibaba006

You will need to add a payment method.
alibaba007

Once the payment information is added you will need o verify this. Alibaba says they will deposit a  verification code and a set amount into your bank account that you will need to obtain to verify your account (fair enough).

alibaba008

The problem is I had two deposits and two codes that both did not work and the format of the deposit descriptions was not as expected (I was expecting “INTL* #####” and received “######  INTL*”, I tried both of these codes and neither worked).
alibaba009 error - multiple

The codes would not work.
alibaba010

Error: “The verification code is incorrect. You have 4 attempts left to verify.”.

alibaba011_verify_bank_failed

After 4 attempts I was blocked.
alibaba012_verify_bank_failed

I now sought to report this problem by logging a support ticket? In the Alibaba support area, it appeared I had a “Basic” account that may not have support rights as I could see a “Purchase Support Plan” link?
alibaba013_trying_to_report_error

I found the create support ticket link and proceeded to attach the support related screenshots. Apparently tiff files (Mac default screenshots format are not allowed).
alibaba014_unsupported_attahment_format

I proceeded to convert my screenshots to png files and attach then.
alibaba015_reported_verification_error

The support ticket was logged.
alibaba016_waiting_for_response

I tried again to add and verify my payment method.
alibaba017_readded_payment_card

Again I received two invalid verification deposits and codes.
alibaba018_multiple_verify_tokens_again_and_none_at_012c

Alibaba billing has a few FAQ articles that may or may not hep me

Why have I received two charges to my account?

Alibaba sent both a micocharge and an authorization hold to your card during signup. The authorization hold verifies that your card details are valid and that you do not have a zero balance. The authorization hold is always 1.00 USD. The microcharge is used to verify that you are the owner of the card, and can be any amount less than 1.00 USD. Both charges will be returned to you within 24 hours.

What if my card statement is not in US Dollars?

Users outside of the United States may not be able see the original USD amount of the microcharge in their bank statement because they are billed in local currency. In this case, you may opt to use the accompanying 6-digit transaction code displayed in the transaction description in order to complete account verification.

Alas I am still stuck verifying my payment method.
alibaba019

Waiting for support

A full business day later and I am still waiting for feedback on the ticket.

alibaba021

Support Response

Support said I need to NOT enter the transaction ID and enter the amount instead.

alibaba022

I tried that with no luck 🙁  I have removed m payment details an will stick with Digital Ocean.

alibaba023

It appears Alibaba still have issues with multiple bank verification ID’s being sent. I have deleted my payment car from Alibaba and will stick with Digital Ocean and AWS.  I guess I can try and pay Alibaba for more money help, but no.

Alibaba Support >If you want higher level support, refer to Alibaba Cloud Support Plans page https://intl.aliyun.com/support/after-sales

Do they realise I am not a US citizen and entering the amount will not work?

Alibaba Support (Official)

Support Engineer : Dear customer, we apologize for the bad experience you have suffered in micro charge verification, we have engaged the product and development team to make the feature enhancement.

Some of the bank may do not display the USD and random code in the billing statement, we are contacting the bank for further investigation. Currently, you have deleted the adding in the console, would you please try to add the card again to get the authorization & micro charge?

If you have problem in verification, please provide the screenshot of both authorization hold & micro charge in billing statement, we can bypass the micro charge for you after evaluation.

Feel free to contact us if there are any other questions, thanks for your support.

Alibaba Pros

  • Australian Hosting Locations
  • Robust Signup Verifications (email, phone and bank), so much so I am not verified.
  • Cheap (if you can get in)???

Alibaba Cons

  • Broken Signup
  • I can’t create a VM due to signup verification
  • No Authenticator Two-Factor Authentication at login.
  • Alibaba Cloud iOS app is not available in English
  • Support is lacking.

Alibaba Cloud iOS App



Conclusion

I don’t know how good Alibaba Cloud is as it appears the signup is broken for Australian customers. For now, stick with Digital Ocean and AWS.

I have received an email from Alibaba saying they need more information.

“> Please provide more information so that we can deal with the problem.
> You can log on Aliyun.com or click here to submit feedback.”  I tried Alibaba suggestion of entering the bank account verification account but that did not work. Alibaba needs better support.  Do they realise I am not a US citizen and entering the amount will not work?

If You sign up for Alibaba Cloud here you will get $300 of New User Free Credit.

or

Easily deploy an SSD cloud server on @DigitalOcean in 55 seconds. Sign up using my link and receive $10 in credit (2 free months): https://www.digitalocean.com

Alternatives

Digital Ocean, LiNode or AWS

Based on comments on the Alibaba Cloud Facebook ad.

https://www.scaleway.com/pricing/

http://smlurl.ml/vultr

Read my other guides

Donate and make this blog better





Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.

v1.6 Added support info.

Digital disruption or digital tinkering

by Leave a Comment

The biggest buzzwords used by prime ministers, presidents or management these days has been “Innovation” and “digital disruption”.  As a developer or manager do you understand what goes into a new digital customer focused service like an API or data driven portal? How well is your business or products doing in the age of innovation and digital disruption? Do you listen to what your customers want or need?

Advertisement:



When to Pivot

There comes a time when businesses realize they need to pivot in order to stay viable.

  • People don’t rent VHS movies they download movies from the Internet
  • Printing photographs, who does that anymore?
  • People learn from videos on YouTube, Khan Academy for free or pay for courses from, Pluralsight.com, Coursea.org, Udemy.com of Linda.com.
  • Information is wanted 24/7 and a call to customer service if information cannot be sourced online.

kodak-bankruptcy

Image source.

Chances are 90% of your customers are using mobile or tablet devices on any given day.  If you are not interacting with your customers via personalized/mobile technology prepare to be overtaken as a business.

Pivoting may require you to admit you are behind the eight ball and take a risk and setup a new customer focused web portal, API, app or services.  Make sure you know what you need before the lure of services, buzzwords and  “shiny object syndrome” from innovation blog posts and consultants take hold.

Advocates and blockers of change

Creating change in an organization that bean counts every dollar, exterminates all risk and ignore ideas is a hard sell. How do you get support from this with power and endless rolls of red tape?

Bad reasons for saying no to innovation:

  • You can’t create a mobile app to help customers because the use of our logo won’t be approved.
  • Don’t focus on customer-focused automation, analytics and innovation because internal manual processes need attention first.
  • Possible changes in 2 years outside of our control will possibly impact anything we create.
  • Third eye lids.

Management’s support of experimentation and change is key to innovation. HARVARD BUSINESS REVIEW have a great post on this: The Why, What, and How of Management Innovation.

Does your organization value innovation? This is possibly the best video that describes how the best businesses focus on innovation and take risks. Simon Sinek: How great leaders inspire action.

Here is a great post on How To Identify The Most Dangerous Person In Your Company who blocks innovation and change.

Also a few videos on getting staff on board and motivation and productivity.

Project Perspectives
consultant_001

Project Focus

  • Focus on customer requirements and what you need to be doing and ignore the tech frameworks/language/features/services.
  • Focus on customer requirements and what you need to be doing and ignore the tech frameworks/language/features/services.
  • Focus on customer requirements and what you need to be doing and ignore the tech frameworks/language/features/services.

I said that three times (because it is important).

Before you begin coding, learn from those who have failed

Here are some of the best tips I have collected from start-ups who have failed.

  • We didn’t spend enough time talking with customers and we’re rolling out features that I thought were great, but we didn’t gather enough input from clients. We didn’t realize it until it was too late. It’s easy to get tricked into thinking your thing is cool. You have to pay attention to your customers and adapt to their needs.
  • The cloud is great. Outsourcing is great. Unreliable services aren’t. The bottom line is that no one cares about your data more than you do – there is no replacement for a robust due diligence process and robust thought about avoiding reliance on any one vendor.
  • Your heart doesn’t get satisfied with any levels of development. Ignore your heart. Listen to your brain.
  • You can always iterate and extrapolate later. Wet your feet asap.
  • As the product became more and more complex, the performance degraded. In my mind, speed is a feature for all web apps so this was unacceptable, especially since it was used to run live, public websites. We spent hundreds of hours trying to speed up the app with little success. This taught me that we needed to having benchmarking tools incorporated into the development cycle from the beginning due to the nature of our product.
  • It’s not about good ideas or bad ideas: it’s about ideas that make people talk. Make some aspect of your product easy and fun to talk about, and make it unique.
  • We really didn’t test the initial product enough. The team pulled the trigger on its initial launches without a significant beta period and without spending a lot of time running QA, scenario testing, task-based testing and the like. When v1.0 launched, glitches and bugs quickly began rearing their head (as they always do), making for delays and laggy user experiences aplenty — something we even mentioned in our early coverage.
  • Not giving enough time to stress and load testing or leaving it until the last minute is something startups are known for — especially true of small teams — but it means things tend to get pretty tricky at scale, particularly if you start adding a user every four seconds.
  • It’s possible to make a little money from a lot of people, or a lot of money from a few people. Making a little money from a few people doesn’t add up. If you’re not selling something, you better have a LOT of eyeballs. We didn’t.
  • We received conflicting advice from lots of smart people about which is more important. We focused on engagement, which we improved by orders of magnitude. No one cared. Lesson learned: Growth is the only thing that matters if you are building a social network. Period. Engagement is great but you aren’t even going to get the meeting unless your top-line numbers reach a certain threshold (which is different for seed vs. series A vs. selling advertising).
  • Our biggest self-realization was that we were not users of our own product. We didn’t obsess over it and we didn’t love it. We loved the idea of it. That hurt.
  • Do not launch a startup if you do not have enough funding for multiple iterations. The chances of getting it right the first time are about the equivalent of winning the lotto.
  • It may seem surprising that a seemingly successful product could fail, but it happens all the time. Although we arguably found product/market fit, we couldn’t quite crack the business side of things. Building any business is hard, but building a business with a single app offering and half of your runway is especially hard.

Buzzwords

The Innovation landscape is full of buzzwords, here are just a few you will need to know.

  • API – Application Program Interface is a method that uses web address ( http://www.server.com/api/important/action/) to accept requests and deliver results. Learn more about API’s here http://www.programmableweb.com/category/all/apis
  • AR – Augmented reality is where you use a screen on a mobile, tablet or PC to overlay 3D or geospatial information.
  • Big Data – Is about taking a wider view of your business data to find insights and to predict and improve products and services.
  • BYOD – Bring your own device.
  • BYOC – Bring your own cloud.
  • Caching – Using software to deliver data from memory rather than from slower database each time.
  • Cloud – Someone else’s computer that you run software or services on.
  • CouchDB – An Apache designed Key/Value NoSQL JSON store database that focuses on eventual replication.
  • DaaS – Desktop as a service
  • DbaaS – Database as a service (hardware and database software maintained by others but your data).
  • DBMS – Database Management System – the GUI
  • HPC – High-Performance Computing.
  • IaaS – Cloud-based Servers and infrastructure (Google Cloud, Amazon AWS, Digital Ocean and Vultr and Rackspace).
  • IDaaS – Third Party Authorisation management
  • IOPS – Operations per Second –What limitations are on the interface or software in question.
  • IoT – Internet of things are small devices that can display, sense or update information (internet connected fridge or a button that orders more toilet paper.
  • iPaaS– integration Platform as a Service (software to integrate multiple XaaS)
  • JSON – A better CSV file (read more here)
  • MaaS – Monitoring as a Service (e.g Keymetrics.io)
  • CaaS – Communication as a service (e.g http://www.twillio.com)
  • Micro-services – an existing service that is managed by another vendor (e.g Notifications, login management, email or storage), usually charged by usage.
  • MongoDB – Another Key/Value NoSQL JSON Database that has upfront Replication
  • NoSQL – A No SQL database that stores data in JSON documents instead of normalised related tables.
  • PaaS – A larger stack of SaaS that you can customise from vendors Azure (Active Directory, Compute, Storage Blobs etc), AWS (SQS, RDS, Alasticache, Elastic File System, ), Google Cloud (Compute Engine, App Engine, Data-store ), Rackspace etc.
  • Rate Limiting – Ability to track and limit a user’s request to an API.
  • SaaS – A smaller software component that you can use or integrate (Google Apps, CiscoWebEx, GoTo
  • Scalable – the ability to have a website or service handle thousands to millions of hits and have baked in a way to handle exponential growth.
  • Meeting).
  • Scale Up – Increase the CPU speed and thus workload
  • Scale Out – Adding more servers and distributing the load instead of making servers faster.
  • SQL – A traditional relational database query language.
  • VR – Virtual Reality is where you totally immerse yourself in a 3D world with a head mounted display.
  • XaaS – Anything as a service.

External or Online Advice

A consultant once joked to our team that their main job was to “Con” and “Insult” you ( CONinSULTant ).  Their main job is to promote what they know/sell and sow seeds of doubt about what you do. Having said that please take my advice with a grain of salt (I am just relaying what I know/prefer).

Consultants need to rapidly convert you to their way of thinking (and services), consultants gloss over what they don’t know and leave you down a happy part solution nirvana (often ignoring your legacy apps or processes, any roadblocks are relished as an opportunity for more money making).  This is great if you have endless buckets of money and want to rewrite things over and over.

Having consultants design as develop a solution is not all bad but that would make his developer-focused blog post boring.

Microsoft IIS, Apache, NGINX, Lighthttpd are all good web servers but each has a different memory footprint, performance, and features when delivering static v dynamic content and each platform has maximum concurrent users that they can handle a second for a given server configuration.

You don’t need expensive solutions, read this blog post on “How I built an app with 500,000 users in 5 days on a $100 server”

Snip: I assume my apps will be successful. There’s no point in building an app assuming it won’t be successful. I would not be able to sleep if my app gains traction and then dies due to bad tech. I bake minimum viable scalability principles into my app. It’s the difference between happiness and total panic. It’s what I think should be part of an app MVP (Minimum Viable Product).

Blind googling to find the best platform can be misleading as it is hard to compare apples to apples. Take your time and write some code and evaluate for yourself.

Once you start worrying about scalability you start to plan for multiple servers, load balancing, replication and caching be prepared to open your wallet.

I prefer the free NGINX and if I need more grunt down the track I can move to the NGINX Plus as it has loads of advanced scalability and caching options  https://www.nginx.com/products/.
Alternatively, you can use XaaS for everything and have other people worry about the uptime/scaling and data storage but I find that it is inevitable you will need the flexibility of a self-managed server and FULL control of the core processes.

Golden rule = prove it is cheaper/faster/more reliable and don’t just trust someone. 

Common PaaS, SaaS and Self-Managed Server Vendors

Amazon AWS and Azure are the go to cloud vendors who offer robust and flexible offerings.

Azure: https://azure.microsoft.com/en-us/

Amazon AWS: https://aws.amazon.com/

Google cloud have many cloud offerings but product selection is hard. Prices are high and Google tend to kill off products that don’t make money (e.g Google Gears etc).

Google Cloud:

https://cloud.google.com/

Simple Self Managed Servers

If you want a server in the cloud on the cheap Linode and Digital Ocean have you covered.

High-End Corporate vendors

Other vendors

Moving to Cloud Pro’s

  • Lowers Risk
  • Outsource talent
  • Scale to millions of users/hits
  • Pay for what you use
  • Granular access
  • Potential savings *
  • Lower risk *

Moving to Cloud Con’s

  • Usually billed in USD
  • Limited upload/downloads or API hits a day
  • Intentional tier pain points (Limited storage, hits, CPU, data transfers, Minimum servers).
  • Cheaper multi-tenant servers v expensive dedicated servers with dedicated support
  • Limited IOPS (g 30 API hits a second then $100 per additional 10 Req/sec)
  • XaaS Price changes
  • Not fully integrated (still need code)
  • Latency between Services.
  • Limited access for developers (not granular enough).
  • Security

Vendors can change their prices whenever they want, I had a cluster of MongoDB servers running on AWS (via http://www.mongodb.com/cloud/ ) and one day they said they needed to increase their prices because they underestimated the costs for the AWS servers. They gave me some credit but I was instantly paying more and was also tied to USD (not AUD). A fall in the Australian dollar will impact bills in a big way.

Vendor Uptime:

Not all vendors are stable, do your research on who are the most reliable: https://cloudharmony.com/status

Quick Status Pages of leading vendors.

Some vendors have patchy uptime

consultant_003

Management Software and Support:

Don’t lock in a vendor(s) until you have tested their services and management interfaces and can accurately forecast your future app costs.

I found that Digital Ocean was the simplest to get started, had capped prices and had the best documentation. However, Digital Ocean do not sell advanced services or advanced support and they did not have servers in Australia.

Google Cloud left a lot to be desired with product selection, setup and documentation. It did not take me long to realize I would be paying a lot more on Google platforms.

Azure was quite clean and crisp but lacked controls I was looking for. Azure is designed to be simple with a professional appearance (I found the default security was not high enough for me unmanaged Ubuntu Servers).  Azure was 4x the cost of Digital Ocean servers and 2x the cost of AWS.

AWS management interfaces were very confusing at first but support was not far away online.  AWS seemed to have the most accurate cost estimators and developer tools to make it my default choice.

Free Trials

When searching for a cloud provider to test look for free trials and have a play before you decide what is best.

https://aws.amazon.com/free/ – 12 Month free trial.

https://azure.microsoft.com/en-us/free/ –  $200 credit.

Digital Ocean 2 moths free for new customers.

Cloudant offered $50 free a month for a single multi-tenant NoSQL database but after as IBM acquisition the costs seem steep (Financing is available through so it must be expensive). I walked away from IBM because it was going to cost me $4,000 a month for 1 dedicated Cloudant CouchDB Node.

Costs

It is hard to forecast your costs if you do not know what components you will use, what the CPU activity will be and what data will be delivered.

Google and AWS have a confusing mix of base rates, CPU credits, and data costs. You can boost your credits and usage but it will cost you compared to a flat rate server cost.

Digital Ocean and Linode offer great low rates for unmanaged servers and reasonable extra charges other vendors will scalp from the get go but lack the global presence.

Azure is a tad more expensive than AWS and a lot higher than Digital Ocean

At some point you need to spin up some servers and play around and if you need to change to another vendor.  I was tempted by IBM Cloud Ant CouchDB DBaaS but it would have been $4000 USD a month. (it did come with 24/7 techs that monitored the service for me).

Databases

Relational databases like MySQL and SQL Server are solid choices but replication can be tricky. See my guide here.

  • NoSQL database are easier to scale up and out but more care has to be given to the software controlling the data and collisions, Relational databases are harder to scale but are by designed to enforce referential integrity.

Design what you need and then chose a Relational, NoSQL or Mix of databases.  A good API will join a mix of databases but deliver the best of both worlds.

E.g Geographic data may best be served from MongoDB but related customer data from MySQL or MS SQL Server

Database cost will also impact your database decisions. E.g Why setup a SQL Server when a MySQL will do, why set up a Mongo DB cluster when a single MongoDB instance will do.

Also when you scale out the database capabilities vary.

  • Availability – Each client can always read and write data.
  • Consistency – All clients have the same view of the data
  • Partition Tolerance – The System works well despite physical network partitions.

nosql-triangle

Database decisions will impact the code and complexity of your application.

Website and API Endpoint

The website will be the glue that sticks all the pieces together.  An API on a web server ( e.g  https://www.myserver.com/api/v1/do/domething/important ) may trigger these actions.

  1. Check the request origin (Ip ban) – Check IP cache or request new IP lookup
  2. Validate SSL status.
  3. Check the users login tokens (are they logged in) – log output
  4. Check a database (MYSQL)
  5. Check for permissions – is this action allowed to happen?
  6. Check for rate limiting – had a threshold been exceeded.
  7. Check another database (MongoDB)
  8. Prepare data
  9. Resolve the API request – return the data.

A Web server then becomes very important as it is managing lot. If you decided to use a remote “as a service”  ID management API or application endpoint would each of the steps happen in a reasonable time-frame.  StormPath may be great service for IP auth but I had issues with reliability early on and costs werte unpredictable, Google firebase are great at Application endpoints but they can be expensive.

Carefully evaluate the pro’s and cons of going DIY/self-managed versus a mix of “as a service” and full “as a service”.

I find that NGINX and NodeJS is the perfect balance between cost, flexibility, and scalability and risk [ link to my scalability guide ] NodeJS is great for integrating MySQL, API or MongoDB calls into the back end in a non-blocking way.  NodeJS can easily integrate caching and connection pooling to enhance the throughput.

Mulesoft is a good (but expensive) API development suite https://www.mulesoft.com/platform/api

Location, Latency and Local Networks.

You will want to try and keep all of your servers and services as close as possible, don’t spin up a digital ocean server in Singapore if your customers are in Australia (the NETFLIX effect will see latency drop off a cliff at night). Also having a database on one vendor and a web server on another vendor may add extra latency issues, try and use the same vendor in the same data center.

Don’t forget SSL will add about 40ms to any request locally (and up to 200ms for overseas serves), that does impact maximum concurrent users (but you need strong SSL).

 

Also, remember the servers may have performance limitations (maximum IOPS ) sometimes you need to pay for higher IOPS or performance to get better throughput.

Security

Ensure that everything is secure, logged and you have some sort of IP banning or rate limiting and session tokens/expiry and or auto log out.

Your servers need to be patched and potential exploits monitored, don’t delay updating software like MySQL and OpenSSL when exploits are known.

Consider getting advice from a company like https://www.whitehack.com.au/ where they can review your code and perform penetration testing.

You may want to limit the work you do on authorization management and get a third party to do https://www.okta.com/ or http://www.stormpath.com can help here.

You will certainly need to implement two-factor authentication,  OAuth 2, session tokens, forward security, rate limiting, IP logging, polymorphic data return via API.  Security is a big one.

Here is a benchmark for an API hit overseas with and without SSL

 

consultant_002-1

Moving my Digital Ocean Server from Singapore to AWS in Australia dropped my API requests to under 200ms (SSL, complete authorization, logging and payload delivery).

Monitoring and Benchmarking

Monitoring your website’s health (CPU, RAM and Memory) along with software and database monitoring is very important to maintain a service.

https://keymetrics.io/ is a great NodeJS service and API monitoring application.

consultant_004

PM2 is a great node module that integrated Key metrics with NodeJS.

Siege is a good command line benchmark took, check out my guide here.

http://www.loader.io is a great service for hitting your website from across the world.

End to End Analytics

You should be capturing analytics from end to end (failed logins, invalid packets, user usage etc).  Caching content and blocking bad uses can them be implemented to improve performance.

 

Developer access

All platforms have varied access to allow developers in to change things.  I prefer the awesome http://www.c9.io for connecting to my servers.

If you go with high-level SaaS (Microsft CRM, Sitecore CRM etc) you may be locked into outdated software that is hard for developers to modify and support.

Don’t forget your customers.

At this point, you will have a million thoughts on possible solutions and problems but don’t forget to concentrate on what you are developing and it is viable.  Do you have validated customer needs and will you be working to solve those problems?

Project Pre-Mortem

Don’t be afraid to research what could go wrong, are you about to spend money on adding another layer of software to improve something but not solve the problem at hand?

It is a good idea to quickly guess what could go wrong before deciding on a way forward.

  • Server scalability
  • Features not polished
  • Does not meet customer needs
  • Monetization Issues
  • Unknown usage costs
  • Bad advice from consultants
  • Vendors collapsing or being bought out.

Long game

Make sure you choose a vendor that won’t go broke?  Smaller vendors like Parse were gobbled up by Facebook and Facebook closed their doors leaving customers in the lurch.  Even C9.io has been purchased by AWS and their future is uncertain.  Will Linode and Digital Ocean be able to compete against AWS and Azure? Don’t lock yourself into one solution and always have a backup plan.

Do

  • Do know what your goal is.
  • Make a start.
  • Iterate in public.
  • Test everything.

Don’t

  • Don’t trust what you have been told.
  • Don’t develop without a goal.
  • Don’t be attracted to buzzwords, new tech and shiny objects.

Good luck and happy coding.

 

Donate and make this blog better





Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.

Edit v1.11

What is you backup plan in 2016

by Leave a Comment

What would you do right now if the computer or device you were reading this on died? Do you have adequate backups and do you keep your most important files safe?

Advertisement:



I wrote and article back in 2013 about good backup plans and I used to spend 3 hours every month burning backup DVD’s and adding them to an indexed catalog.  I did not trust Apple iCloud to backup my photos and have had my fair share of lost photos. I came up with a plan for managing and sorting photos from multiple digital devices that put me in charge of copying photos from devices, backing up high res photos and shrinking photos for local use.

Fast forward 3 years and I am now capturing insane amounts of digital photos and videos and I am no longer backing up every month due to the massive amounts of data and now what seems like small capacity of DVD’s.  I am instead throwing all of my new files into an “Incoming” folder to be processed at a later date. This was fine as I assumed the data would be safe.

Murphy’s Law: Anything that can go wrong will go wrong.

On a recent hot summers day, I was performing maintenance on my primary computer (removing dust from the processor fan) and one of the hard drives dropped 30 centimeters to the ground.  I thought that all would be fine and the data would be safe as the hard drive was a good quality one.  Nope, the drive is dead, corrupted and fails to show files.  The drive spins up ok and does not sound like anything is broken inside. That drive had my last 3 years of photos on it with games, movies, music etc.  I don’t think it had much of value as my Incoming folder was on another drive but I can’t be sure.

How can I prevent this in future?

This happened because I failed to allocate time to sort and backup files and I assumed that Murphy’s Law would not apply to me..

Traditional golden rules of backup:

  1. Backup to Three different locations.
  2. Two of the media need to be different.
  3. One of the locations need to be offsite.

Simon’s new rules of backup:

  1. Backup to Four different locations (g DropBox, Time Machine, Backblaze, Google Drive, NAS).
  2. Three of the backup media need to be different (g Cloud, DVD, NAS).
  3. Two of the locations need to be offsite (g Dropbox, Backblaze/CrashPlan).
  4. One of the backups need to be automated (g Dropbox, BackBlaze, Time Capsule, CrashPlan).
  5. Zero worries

Why so many backups you may ask? Each backup method is best for different scenarios. Dropbox is good for recently changing files, Cloud backups are good for everything and local backups are good for when cloud providers stuff up or to cover major infrastructure damage/failure overseas.

The ultimate local backup solution is a Drobo 5N 5 NAS from https://www.pccasegear.com/ . Simply insert hard drives (up to 5 3.5″/64TB) plug it into a router or switch and you are good to go.  Read more here.

Simple Cloud Backup Options:

Dropbox is the simplest and safest method of syncing and storing files offsite.  You can store 2GB of files for free or 1TB (10,000 GB) for $12.99 a month).   Dropbox also works on mobile and tablet device too so it is a no-brainer.  Encryption options are not baked in so it is not as secure.

Use this link ( https://db.tt/a4KSuEHO ) to signup at Dropbox and get a bonus 500MB (2,500MB total) for free.

Backblaze.com is my favourite cloud storage provider for Windows of Mac that offers encrypted backups. Use this link to get a month free https://secure.backblaze.com/r/017nsm. It costs $5 a month and there is no limits on the data you can store. My laptop has 90GB of data stored in backblaze and it is all encrypted. Back blaze care about backup so much they have open sourced their backup enclosure designs: https://www.backblaze.com/blog/backblaze-storage-pod-4/

Use this link to get a months free cloud backup to give it a try:  https://secure.backblaze.com/r/017nsm

Crashplan.com is also a good cloud backup for Windows and Mac and offers encrypted backups. For under $6 a month you can backup one computer to the cloud (and store files on a local hard drive). I moved from Crashplan to backblaze as Crashplan did not play well with my Apple Mac a few years back.

Crashplan store: https://store.code42.com/store/

 

Advanced Cloud Backup:

If you do not trust cloud providers to store your data and you want to have offsite backups on the internet you can buy your own server online and manage your own files encryption and sync methods.

Digital Ocean allow you to create and store your own files in the cloud from $5 a month. Use this link and get a free server for 1 x month with 20GB storage.

https://www.digitalocean.com/?refcode=99a5082b6de5

 

Manual Backup

Performing a manual backup by plugging in a removable hard drive feels good but chances are you will not do it often enough.  Most of the time simply highlighting all files and copy pasting it to the new drive fails to work because files are often in use or access is denied to them by the operating system. Manually copying all files every month talks time and copying many smaller files over and over again will increase the wear on your drive and fragment your drive. Tip: The crashplan software above allows you to sync to local drives at no cost and it will copy only what has changed: http://support.code42.com/CrashPlan/4/Backup/Backing_Up_External_Hard_Drives

Manual backups are ok but they are only as good as the data you copy and the frequency you do it.

Network Attached Storage (NAS)

The main downside of cloud backup is it can take weeks if not months to get a full backup into the cloud. That’s where NAS comes into save the day.

A NAS is a mini computer that sits on your network (wired or wireless) and allows you to store files quickly with or without redundancy.

The ultimate local backup solution is a Drobo 5N 5 NAS from https://www.pccasegear.com/ . Simply insert hard drives (up to 5 3.5″/64TB) plug it into a router or switch and you are good to go.  Read more here.

You can choose a NAS with different drive capacities and you can allocate more space to redundancy (referred to as RAID read about raid here https://www.cloudwards.net/what-is-raid/ ).  You would want to at least mirror data (RAID 1) so if one drive fails you have a copy of files on the other disk.

 

Windows Advice:

Windows 7 had a nice Backup feature built in but Microsoft has changed that in Windows 8.1 and 10: http://www.apple.com/au/shop/product/ME182X/A/airport-time-capsule-3tb?fnode=5f

I feel Windows approach to automatically create backups of files on the C drive is a bad idea and copies of hundreds of thousands of files only increased the space ( http://www.apple.com/au/shop/product/ME182X/A/airport-time-capsule-3tb?fnode=5f ) used and wears out the drive.

 

Mac Advice:

Apple OSX comes with a handy feature called Time Machine, if you plug in a large capacity drive OSX will ask if it can use the drive to store backups of your computer.  Time Machine does a great job of keeping multiple versions of files and automatically phase out older versions when space on the backup drive gets low. The only catch is you may need to plug in your backup drive every few days/weeks to let Time Capsule do it’s thing.

Apple sell a wireless Time Capsule for those that do not want to plug in a drive to backup.  Time capsules can be a bit tricky to setup if your home or office has a wireless network already but once it is setup it is rock solid.

http://www.apple.com/au/shop/product/ME182X/A/airport-time-capsule-3tb?fnode=5f

 

Technical Bonus:

Don’t assume new storage devices are more reliable, if anything the cut-throat nature of the PC industry is driving down reliability and quality.  If someone is telling you otherwise they are lying or are trying to sell you something.

Most larger capacity drives (over 1TB) store data on spinning metal platters and the quality and reliability varies between brand and models.  Backblaze has a great write-up on hard drive reliability (based on experience of tens of thousands of drives): https://www.backblaze.com/blog/hard-drive-reliability-stats-for-q2-2015/

Ever increasing data capacity of devices is delivering devices that are more fragile and ones that dedicate less physical space to actually store every bit of data on your computer. Most Solid State drives store data in shared cells ( MLC ) and data is more prone to being corrupted due to manufacturers increasing consumers risk to stay profitable.

My recovery tool of choice for any dead or deleted SD card, USB key or physical drive is http://www.diydatarecovery.nl as it ignores the file application table and looks at the sectors and tried to build files back from there. The only catch is recovery scans take days and you need a new/empty drive that is bigger than the drive you are scanning to restore files to.

So by now you have:

  • Signed up for 2GB free Dropbox using this link to get a bonus 500MB
  • Signed up to a free BackBlaze account to get a free months trial using this link
  • Reviewed your local backup options?

Good luck and happy backing up.

Donate and make this blog better





Ask a question or recommend an article

Your Name (required)

Your Email (required)

Your Question

Please leave this field empty.