Ok, I want to develop and launch mobile apps and was planning on using MySQL as the database for storing the bulk of the data. After much research, I was concerned that MySQL would not scale if app demands spiked and or offer advanced features (like GeoQueries, partitioning, replication, scalability via thread pools etc) without moving to the more advanced (non-free) MySQL versions.
Advertisement:
There are loads of free ways you can speed up a relational database by using indexes, optimising table structures, caching content in memory (OPCache for PHP) and using MySQL connection pools in PHP and running on dedicated (non-shared ) servers. I moved away from a multi-tenant CPanel based domain and looked into purchasing an Amazon EC2 server instance and associated products like AWS SQS but was blown away at the upfront costs and hidden charges. Google and others cloud server providers were not making my decision any easier.
I ended up purchasing unmanaged servers(1x CPU, 512MB RAM, 20Gb SSD and 1,000 GB Transfers) from http://www.digitalocean.com (see my guide on how to set one up here) for $5 a month. Digital Ocean allows you to scale a server up to a server as high as 20 CPU’s, 64Gb RAM, 540Gb SSD and 9TB transfers for $640 a month. Amazon would possibly charge you $10,000 a month. The good thing about Digital Ocean is they have awesome how-to guides.
NoSQL v Relational Database
I wanted to get away from using a relational database and use a distributed NoSQL database. Dataconomy.com have a good read on SQL, NoSQL and NewSQL databases here.
A relational database may be a good fit for some cases ( if normalisedd well ) but in my case, a document based NoSQL database would be perfect. If you are not sure what a relational or NoSQL database is read this and this.
Pros and Cons of NoSQL databases
When you are choosing a distributed NoSQL database you can only choose two of the following attributes.
- Consistency (C) – all nodes see the same data at the same time.
- Availability (A) – every request receives a response about whether it succeeded or failed.
- Partition tolerance (P) – the system continues to operate despite arbitrary partitioning due to network failures.
First I looked into CouchDB from http://www.cloudant.com , CouchDB was great and offered $50 USD of free credit a month before you get billed when using a slower multi-tenant server. $50 buys you approx 1,600,000 light reads or 300,000 heavy writes or 50GB storage a month, beyond that you would be charged for the number of API hits, size and throughput would be limited on multi-tenant servers. My old write up on Cloudant here and here.
CouchDB on Cloudant allows read/writes via secure REST-based queries and had a multi-master replication method (where writes were fast but replication eventually syncs the data across nodes). I really like Cloudant but it would have cost me $4,000 USD a month to move from a shared server to a dedicated (fully managed) server with a team of people monitoring my server’s performance 24/7. If money was no object I would use CouchDB on Cloudant.
MongoDB
The next option was MongoDB as it was similar to CouchDB but does not allow REST based access. MongoDB 3.2 offers Geo based queries that Cloudant did not offer unless I went to a dedicated server for $4,000 a month. MongoDB has loads of supported drivers for most languages (except Lua) and can be installed on most platforms. Time to install it and see how it performs.
MongoDB slide deck for developers.
MongoDB Performance
MongoDB focuses on Consistency and Partition Tolerance over Availability. Having 3 servers in a cluster (replica set) does not mean you will get 3 times the read performance. By default, the primary server on the replica set is queried unless you change some options at the expense of data consistency (same data everywhere). If you want true scalability you need multiple shards (and replicas). Darn.
Setting up a Digital Ocean Server
I installed MongoDB onto a Digital Ocean server (unmanaged Ubuntu VM). If you need your own server follow my guide here: https://fearby.com/article/the-quickest-way-to-setup-a-scalable-development-ide-and-web-server for $5 a month.
Installing MongoDB
Once you have a working Ubuntu Installation on Digital Ocean you can run these commands to install MongoDB 3.2.
1 2 3 4 5 6 | sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927 echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list sudo apt-get update sudo apt-get install -y mongodb-org sudo apt-get install -y mongodb-org=3.2.8 mongodb-org-server=3.2.8 mongodb-org-shell=3.2.8 mongodb-org-mongos=3.2.8 mongodb-org-tools=3.2.8 |
Commands to prevent any updates to this version
I am happy to keep this MongoDB version to 3.2, the following commands will prevent MongoDB version updates.
1 2 3 4 5 6 | echo "mongodb-org hold" | sudo dpkg --set-selections echo "mongodb-org-server hold" | sudo dpkg --set-selections echo "mongodb-org-shell hold" | sudo dpkg --set-selections echo "mongodb-org-mongos hold" | sudo dpkg --set-selections echo "mongodb-org-tools hold" | sudo dpkg --set-selections |
Enable Public connections on port 27017
1 | sudo nano /etc/mongod.conf |
Now comment out this IP to allow remote connections.
1 | # bind_ip <span class="token operator">=</span> 127.0.0.1 |
Enabling Firewall Access (if you want to talk to a MongoDB database elsewhere)
1 2 3 | iptables -A INPUT -s REMOTE_MONGO_IP -p tcp --destination-port 27017 -m state --state NEW,ESTABLISHED -j ACCEPT iptables -A OUTPUT -d REMOTE_MONGO_IP -p tcp --source-port 27017 -m state --state ESTABLISHED -j ACCEPT |
MongoDB should now be installed, you can start and stop the service by running one for the following commands.
Starting and stopping the MongoDB service
1 2 | sudo service mongod start sudo service mongod stop |
Import Data
Now you can import data as required and start building your database.
1 | /usr/bin/mongoimport --db mynewdatabase --collection locations --drop --file ./my_data_to_import.json |
MongoDB components and shell
You can read here to learn more about the installed MongoDB components and the Mongo Shell here.
Happy Mongo’ing.
Helpful guides and sites that helped me.
- https://docs.mongodb.com/master/tutorial/install-mongodb-on-ubuntu/?_ga=1.136227121.1408651081.1446847268
- https://www.mkyong.com/mongodb/mongodb-allow-remote-access/
Security
Secure your database (I used MongoChef, read my guide here).
Don’t forget to test your servers IP with shodan.io to ensure MongoDB is not open.
Donate and make this blog better
Ask a question or recommend an article