Bits & Bytes #3 - How I learned to stop worrying and love Back Office Systems

Posted by: Mahesh Paolini-Subramanya

Today, we are going to take a side-trip - a little one mind you, but still highly relevant. We are going to talk about that oh-so exciting topic Integrating With Back-Office Systems.  Relevant? you think to yourself.  Exciting? you exclaim, just before your eyes glaze over and/or you decide to see whats new at Slashdot  And, to you, I respond, Yup!  Exciting *and* Relevant!
If you look at the vast majority of applications that are developed, or more to the point, not bought off-the-shelf, the Productization Sequence tends to be somewhat like the following (at least, in most people's minds)

1. Inception of a Great Idea.

  Bob:  We'll let people (Urp) design and buy their toasters online!
  Jane: Brilliant!  Gimme another Beer!

2. Development of the Product

  Jane:  Look!  I made it so that friends can help you design your toaster!
  Bob:  And i put a multi-touch screen on it!

3. Launch

  Bob:  Its launched! The sales are ringing up already!
  Jane:  We need to hire more salespeople!  And a V.P. of Marketing!

4. Profit

  Jane:  <Rolling in Money>
  Bob:   <Also rolling>

Ok, maybe not exactly like this, but you get the drift.  Do you, however, notice anything missing in the above sequence? To be precise, where are all the pesky details involving Accounting, Billing, Commissions, Shipping, Supply Chains, etc., etc.?  Yes, you might consider them dross, and yes one can work miracles with the help of Fedex and SalesForce.com, but in the end, your oh-so-precious jewel of an application needs to interact Consistently and Coherently with all that dross if you have any hope of ever Rolling In Money.
Face it, the reality is that come the launch the back-office systems of Toasters Within Reach will consist of a mish-mosh of emails, database twiddling, phone calls, perl_scripts, and various other hacks that are done as necessary so that the show can go on.  e.g.

• Running out of fiddly-bits?  Quick!  Call the supplier to order
  
• Customer wants the bottom half of his toaster sent to Waukegan? Go to  into the database, and change the order manually!
• Need to get your accounting data into quickbooks?  Run the upload script every night at 8:00pm!

Well, you get the point.  Trust me, this sounds terribly familiar to a lot of people out there...

The unfortunate reality is that Bob and Jane are never going to get to the Profit stage unless they get their Back-Office act together.  It doesn't have to look pretty (unless they are planning on selling it too, but that is a separate story), but it has to work Consistently and Coherently.  This means GUIs for their database tools, real supply-chain management systems, billing and accounting integrated into their workflow, etc. etc.
Wait, You think this sounds boring?  Not true!  Given the nature of the world, it is almost certain that each and every one of these tools, systems, processs, etc. uses a different supplier (e.g. Fedex vs. NetApps), different protocols (e.g SOAP vs. HTTP/REST, vs. Some BIzarre API), different languages (Java?  W*Star? C++?), different formats (XML?  JSON? SDF?).  Imagine an orchestra, where the horn section is from Beijing, the violin section from Cincinnati, the pianist from Frankfurt, and you the conductor, have never seen any of them before.  Oh heck, to make this even more realistic, the flautist has never seen a flute before, and the trumpet section never showed up.  Now, really, where exactly are you going to find a challenge like this? 
    To bring this all back to the realm of Architecture and Technology, Bob and Jane would be very well served to have at least thought about this before they started on their quest to Roll In Money.  We at Aptela  in fact did think about this.  Then again, to tell you truth, when we started in 1999 this was pretty much the last thing on our minds.  However, we learned, and learned fast, and have gotten quite good at interfacing with our providers, suppliers, and back-office systems.
    How exactly do we go about conducting this huge - and potentially cacophonic - orchestra?  Well, we have decided to go down the Enterprise System Bus route.  Before your eyes start glazing over again, let me assure you that an Enterprise System Bus is not all that complex a creature, and the specific type we have chosen is really quite straightforward.  When any of our back-office systems talk (send messages) to each other, there are basically three things that need to happen

1. Each message needs to get to the correct destination.  We can't have orders falling through the cracks.  This is Guaranteed Transport 
2. If one of the destinations goes away, the message will still be there waiting for it when it comes back.  This is Guaranteed Delivery
3. Each destination needs to understand the message, regardless of the sender.  This is a Standardized Format

We use an open-source system called ActiveMQ ( to provide the first two, viz. Guaranteed Transport and Delivery (amongst a whole host of other things. But it does these two extraordinarily well).  We use a protocol called STOMP to provide the third, viz. Standardized Format.  
Think of STOMP as the Universal Language that every member of the orchestra speaks (sheet music perhaps?), and ActiveMQ as the - astoundingly patient - gofer who carries all the pieces of sheet music around, ensuring that none of it ever gets lost or dropped.

So, how does STOMP and ActiveMQ play with CouchDB and Erlang?  Stay tuned for further episodes of Bits & Bytes...