IoT, Code, Security and Server Stuff

Views are my own and not my employer's.

Personal Development Blog...

Coding for fun since 1996, Learn by doing and sharing.

Buy a domain name, then create your own server (get $25 free credit)

View all of my posts.

Installing MongoDB onto a Digital Ocean Ubuntu 14.04 server

Ok, I want to develop and launch mobile apps and was planning on using MySQL as the database for storing the bulk of the data. After much research, I was concerned that MySQL would not scale if app demands spiked and or offer advanced features (like GeoQueries, partitioning, replication, scalability via thread pools etc) without moving to the more advanced (non-free) MySQL versions.

Advertisement:



There are loads of free ways you can speed up a relational database by using indexes, optimising table structures, caching content in memory (OPCache for PHP) and using MySQL connection pools in PHP and running on dedicated (non-shared ) servers.  I moved away from a multi-tenant CPanel based domain and looked into purchasing an Amazon EC2 server instance and associated products like AWS SQS but was blown away at the upfront costs and hidden charges. Google and others cloud server providers were not making my decision any easier.

I ended up purchasing unmanaged servers(1x CPU, 512MB RAM, 20Gb SSD and 1,000 GB Transfers) from http://www.digitalocean.com (see my guide on how to set one up here) for $5 a month. Digital Ocean allows you to scale a server up to a server as high as 20 CPU’s, 64Gb RAM, 540Gb SSD and 9TB transfers for $640 a month. Amazon would possibly charge you $10,000 a month.  The good thing about Digital Ocean is they have awesome how-to guides.

NoSQL v Relational Database

I wanted to get away from using a relational database and use a distributed NoSQL database. Dataconomy.com have a good read on SQL, NoSQL and NewSQL databases here.

A relational database may be a good fit for some cases ( if normalisedd well ) but in my case, a document based NoSQL database would be perfect. If you are not sure what a relational or NoSQL database is read this and this.

Pros and Cons of NoSQL databases

When you are choosing a distributed NoSQL database you can only choose two of the following attributes.

  • Consistency (C) – all nodes see the same data at the same time.
  • Availability (A) – every request receives a response about whether it succeeded or failed.
  • Partition tolerance (P) – the system continues to operate despite arbitrary partitioning due to network failures.

cap

First I looked into CouchDB from http://www.cloudant.com , CouchDB was great and offered $50 USD of free credit a month before you get billed when using a slower multi-tenant server. $50 buys you approx 1,600,000 light reads or 300,000 heavy writes or 50GB storage a month, beyond that you would be charged for the number of API hits, size and throughput would be limited on multi-tenant servers.  My old write up on Cloudant here and here.

CouchDB on Cloudant allows read/writes via secure REST-based queries and had a multi-master replication method (where writes were fast but replication eventually syncs the data across nodes).  I really like Cloudant but it would have cost me $4,000 USD a month to move from a shared server to a dedicated (fully managed) server with a team of people monitoring my server’s performance 24/7. If money was no object I would use CouchDB on Cloudant.

MongoDB

The next option was MongoDB as it was similar to CouchDB but does not allow REST based access.  MongoDB 3.2 offers Geo based queries that Cloudant did not offer unless I went to a dedicated server for $4,000 a month. MongoDB has loads of supported drivers for most languages (except Lua) and can be installed on most platforms. Time to install it and see how it performs.

MongoDB slide deck for developers.

MongoDB Performance

MongoDB focuses on Consistency and Partition Tolerance over Availability. Having 3 servers in a cluster (replica set) does not mean you will get 3 times the read performance.  By default, the primary server on the replica set is queried unless you change some options at the expense of data consistency (same data everywhere).  If you want true scalability you need multiple shards (and replicas).  Darn.

Setting up a Digital Ocean Server

I installed MongoDB onto a Digital Ocean server (unmanaged Ubuntu VM). If you need your own server follow my guide here: https://fearby.com/article/the-quickest-way-to-setup-a-scalable-development-ide-and-web-server for $5 a month.

Installing MongoDB

Once you have a working Ubuntu Installation on Digital Ocean you can run these commands to install MongoDB 3.2.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8

Commands to prevent any updates to this version

I am happy to keep this MongoDB version to 3.2, the following commands will prevent MongoDB version updates.

echo "mongodb-org hold" | sudo dpkg --set-selections
echo "mongodb-org-server hold" | sudo dpkg --set-selections
echo "mongodb-org-shell hold" | sudo dpkg --set-selections
echo "mongodb-org-mongos hold" | sudo dpkg --set-selections
echo "mongodb-org-tools hold" | sudo dpkg --set-selections

Enable Public connections on port 27017

sudo nano /etc/mongod.conf

Now comment out this IP to allow remote connections.

# bind_ip = 127.0.0.1

Enabling Firewall Access (if you want to talk to a MongoDB database elsewhere)

iptables -A INPUT -s REMOTE_MONGO_IP -p tcp --destination-port 27017 -m state --state NEW,ESTABLISHED -j ACCEPT 
iptables -A OUTPUT -d REMOTE_MONGO_IP -p tcp --source-port 27017 -m state --state ESTABLISHED -j ACCEPT  

MongoDB should now be installed, you can start and stop the service by running one for the following commands.

Starting and stopping the MongoDB service

sudo service mongod start
sudo service mongod stop

Import Data

Now you can import data as required and start building your database.

/usr/bin/mongoimport --db mynewdatabase --collection locations --drop --file ./my_data_to_import.json

MongoDB components and shell

You can read here to learn more about the installed MongoDB components and the Mongo Shell here.

Happy Mongo’ing.

Helpful guides and sites that helped me.

Security

Secure your database (I used MongoChef, read my guide here).

Don’t forget to test your servers IP with shodan.io to ensure MongoDB is not open.

Donate and make this blog better





Ask a question or recommend an article
[contact-form-7 404 "Not Found"]