One rockin' vendor's journey to SOA

Songkick, an online service that tracks bands and concert dates, spent the last two years service-orienting its systems and offerings, and recently documented its journey online. 

Songkick works by indexing many different ticket vendors, venue websites, and local newspapers to create a massive database of upcoming concerts happening around the world. "It’s our mission to have every show happening anywhere, from your friend’s show at your local bar, right up to Lady Gaga tour dates in Tokyo," the vendor says. The site has also accumulated more than one million past concerts stretching back to the 1960s.

Songkick, created in 2007, is written on the open-source Rails language. But, as described in the Songkick team's first installment on the SOA journey, the Rails app was one big hairball consisting of "a database, a website, message queues, background processing, file storage, daily tasks like email notifications, and so on, it was all one big project."
The challenge was to make changes, everything needed to be changed together. "If you changed our background job processor you had to bump a submodule in the Rails app, run all the tests and deploy the entire thing to all the machines," says James, one of the Songkick developers that lead the SOA effort. "Oh and did I mention the build took two hours spread across a handful of Jenkins (then Hudson) build slaves? It’s a wonder we ever shipped anything. Time for some house-cleaning."

Where did Songkick start with the project? By rewriting the Rails app itself, as described by Sabrina, another Songkickster. The goal was to make it "a simpler web app," she says. "This would also give us the opportunity to understand the types of resources and API endpoints needed, so the services could later be built based on how they were used by clients. Another benefit of starting with the Rails app itself, instead of the services, was that we would have the immediate benefits of a simpler, decoupled web app."
She described some of the details of rewrite:

"The plan was for an 'inside-out rewrite,' that is, we didn’t start a new project from scratch. Instead, we went template after template on Songkick’s website and re-wrote it end to end, from the models and controller to the views, CSS and JavaScript. This way, our code was continuously integrated, which meant the benefits and flaws of our design were seen as soon as a template was done, instead of emerging with a completely new project months later. The drawback of this approach is that it takes a lot of effort to work with evolving code. However, I think that this is an important skill for us to learn as developers. We started crossing the SOA chasm by creating application-specific client model' classes that wrapped ActiveRecord models, and 'service' classes that would call the respective methods on those models, decoupling the domain model from the presentation layer."

The rewrite took about 10 weeks. "During this period, we also rewrote our front end code to have an architecture that mirrors more closely the pages and visual components used on the website," Sabrina adds.

The resulting effort generated a number of orthogonal web services, the developers report, including the following:

• Event-listings handles data relating to concerts, artists, venues, and so on
• Accounts handles users’ account data and authentication
• Taste-imports processes sets of artists uploaded by various sources of user taste data
• Caltrak handles users’ taste data and calendar generation
• Attendance stores concerts users have said they are going to
• Media handles and stores file uploads – photos, videos and the like
• Recommendations determines sets of similar artists

In the end, the Songkicksters report, "each application is free to tweak how it interacts with the service APIs, without affecting any other application, and this is a big win for us. It means no change to one application can have side effects or block work on another application, and we have’t actually found ourselves reinventing substantial pieces of logic since that’s all hidden behind the HTTP APIs."

The Songkick team also has advice for those contemplating SOA. For instance, keep the effort small and incremental -- "favor small components: components and libraries with focused responsibilities that you can easily reuse. Encouraging this style of development means you need to make it easy to develop, integrate, and deploy many small components rather than one big ball. The easier this is, the more likely people are to create such libraries."

In addition, they add, "remember there’s no ‘one right way’ to do things. You have to make trade-offs all the time, and the textbook engineering answer doesn’t always give your team the greatest velocity. Examine why you’re making each change, focus on long-term productivity, and you won’t go far wrong."