In the past couple weeks I’ve been writing a few different modules and publishing them to npm. When I’m writing new code to solve a problem I often end up coding together some functionality and releasing a singular module however this time around I tried to do things a little differently. Instead I tried to sit down and break out each individual piece of my solution (similar to what’s often referred to as the substack pattern). What pieces could I work on separately that would have value and merit on their own? The result was 3 individual modules with each one being a composition of the last. Breaking out my solution like this did a few things that I think are big wins when programming:

1. I made my problem’s surface area smaller by identifying tinier problems (and easier wins)
2. I made useful pieces of my solution available for others to compose into other potential solutions
3. My github activity bar is looking awesome

##The Problem The problem I set out to solve was how to keep dat (a project with the goal of streamlining data transfer between any and all data stores or file formats) in sync with MongoDB. Doing a single import had been solved by mongs but there was no existing way to keep that import up to date with changes in Mongo. Having identified my problem I broke it out into smaller problems.

##Oplog Docs The first tinier problem was translating Mongo’s oplog documents into actions that can be performed on any object. Mongo utilizes the oplog to translate changes in the primary to its secondaries keeping them in sync. The syntax of these JSON documents is little peculiar at first but they provide information about document updates, inserts, as well as deletes. Essentially everything you need to make the same changes to a document that lives outside of Mongo as well. I built oplog-transform to take oplog documents, turn them into functional expressions, and apply those changes to any JSON object you want.

##Tailing the Oplog Now that we can translate oplog docs we just need to hook up to the oplog and tail it to perform these changes in real-time. I created oplog-transform-tail to do just that. It’s a simple composition of the great mongo-oplog lib as well as oplog-transform. We’re already reaping the benefits of breaking these pieces out by being able to build them up in new useful ways.

##Wrapping up Finally we can solve the overarching problem we set out for in the beginning, syncing dat with MongoDB, and the result is dat-oplog. Dat has hooks you can plug into with your own functionality and dat-oplog makes use of listen. Now using oplog-transform-tail we can push updates to dat every time a document changes in Mongo.

The code needed to create dat-oplog was relatively small since we can leverage the previous modules. Each module being responsible for its own functionality keeps the individual code bases small which can reduce bugs and increase transparency into what is happening. Building a solution this way has a lot of advantages and I think it’s a worthwhile exercise when tackling any large problem.