Working with CouchDB to Achieve Replication and Scaling

Posted by: Mahesh Paolini-Subramanya

We here at Aptela have a Hate-Hate relationship with data. On one hand, we hate the torrential, never-ending, fire hose-like deluge of data that is constantly cascading out of all of our telephony operations. On the other hand, we hate the configuration and state data associated with our users and clients. Why this level of hate? Well, primarily because we need to keep all of this information synchronized across all of our servers, all of the time.

Along with all of this, our business VoIP systems need to be massively yet inexpensively scalable. We handle millions of phone calls a day, with each and every one of these phone calls triggering complex phone applications and phone routing. Each of these calls generates a Call Detail Record (CDR), which needs to be processed for Billing, stored for Reporting, and readily available for the User's access - Users like to be able to see all this stuff in real-time after all!

Our entire server-infrastructure is (and needs to be!) cloud-based, i.e., highly distributed, reliable, scalable, location-independent, fine-grained, and with built-in coffee service. Take incoming calls for example. In our environment, they go to a randomly chosen server, which figures out what to do with the call based on the called number. Then the server waits until the Official Data Store figures out what to do with the call. The problem? We were spending all of our time figuring out how we could improve our databases to support our application, and not actually spending any time on improving our application!

Enter couchDB, a document-oriented database, which has worked like a charm for us. If anything, we have only begun to tap all the cool things that it does for us.

Remember the fire hose of data I mentioned at the beginning of this post? We now handle it by dumping it into local instances of couchDB on each of our telephony nodes. At this point, a couple of really neat things happen (ok, neat for me, probably not for you):

- billing information gets extracted from these CDRs, and replicated over into the billing system
- metadata associated with voicemails and recordings get replicated across to the other telephony nodes
- metadata associated with the calls get replicated over to the application nodes
- the CDRs themselves all end up getting replicated to the reporting servers, where all sorts of goofy reports can now get generated off of them.

The free-form nature of couchDB is tailor-made for reporting, and that alone makes it worth the price of admission. Come to think of it, that was pretty much what made us look at CouchDB in the first place! That said, once we started working with it, it became immediately and abundantly clear that this was the solution to all our data-managament and maintenance issues.
couchDB is written in erlang, and to paraphrase - We love erlang so much we wrote our entire application in it - which makes it trivially easy to integrate it into our application.

It also has an extremely easy to use REST API (Representational State Transfer), which makes integrating it into our back-office systems just about as trivial, something that is remarkably important as I've alluded to elsewhere.

Yes, I know, if we only did things some other way, we could have solved all our problems with <<"Pick Your SQL Database Here">>. However, sometimes you really are better off using the right tool for the job, and in our case, couchDB is that tool. Bravo Damien!

Want to learn more about couchDB? Check out the Couchio - Aptela case study, as well as these other great references:

http://jchrisa.net/drl/_design/sofa/_show/post/CouchDB-Google-Tech-Talk

http://news.cnet.com/8301-13846_3-20006527-62.html

http://planet.couchdb.org/