What’s a Nice XPer Doing in a Company Like This? – InApps 2022

Most people know me as someone who is into extreme programming (XP) and the things associated with XP: object-oriented programming, test-driven development, test data management and effective build systems. For some, the link between the work we do at Container Solutions and XP may seem tenuous or nonexistent. However, look closer and you’ll see the link between the two is strong.

Extreme Programming is all about shortening feedback loops. Here is the classic onion diagram:

Don Wells, 2001, via wikipedia.

Feedback loops are important in XP because the method tries to help development teams be more responsive to their customers’ needs. This is why, although object-oriented (or component-oriented) development is not explicitly a part of XP, it is important because well designed objects can be quickly recombined to service a new customer requirement. This is why refactoring and test-driven development are an essential part of XP.

Refactoring and Test-Driven Development

Refactoring is about improving code without changing its functionality. This is dangerous to do without a suite of tests, which prove that the code behaves in the same way.

Test-driven development (TDD) is about design; by writing tests first, we are forced to think about our interfaces and the interactions between our objects. TDD with mock objects, as popularized by Steve Freeman and Nat Pryce, is a particularly good method for growing object-oriented systems.

A great side effect of TDD is that you are left with a suite a decent tests, which means you can refactor your code more or less in safety. (Once the tests are written, the test suite is a sort of safety net, a collection of checks to make sure everything is more or less working as it was before any refactoring started.)

Testing Infrastructure

As well as using TDD, many of us used to write tests for infrastructure. For example, I used to write tests that pinged all the databases, messaging systems, made sure the payment system was there, etc. We did this for two reasons. Firstly, because if something went wrong, the test suite would point to where the problem was; the tests were used for diagnosis purposes. Secondly, we did this because we often had no control over the infrastructure. The tests helped us to pass information onto the operations teams that looked after the staging, pre-production and production environments.

The Cloud

Once clouds started to become popular, we were forced to think about how we could deploy to the cloud without destroying the tight feedback loops we had become used to. In other words, we had to think carefully about how to deploy, test and refactor our cloud-based systems. This wasn’t exactly trivial. For example, how could we launch a virtual machine on the cloud and then update it in light of new user requirements? How could we recreate the cloud locally in order to iron out any infrastructure problems?

Mesos

As cloud technologies evolved, so, too, did the range of solutions we could build. Apache Mesos burst into our lives about a year ago. The tool allows us to utilize our hardware more effectively. For our customers who have large numbers of computing assets, this is important because using Apache Mesos can literally save them millions of euros. (A more detailed explanation of the benefits of Mesos can be found on the Container Solutions’ website in an article written by Phil Winder.)

For us, then, once Mesos arrived, a new question arose: how can we develop containerized applications and frameworks for Mesos whilst still carrying out the core XP practices of TDD and continuous integration?

Enter Mini-Mesos

Containerized solutions provide two key services. On the one hand, they enable the smooth deployment and management of applications in production. On the other, they provide application development. We therefore have to come up with tools that help with both application development and infrastructure design.

One way to quickly iron out operational problems is to reproduce your full production environment on your laptop. Docker Compose took a small step towards this. Unfortunately, it doesn’t have any support for Mesos. This meant we had to set up our Mesos cluster, deploy to it, then manually run our tests. This could take minutes, and for more complex frameworks, tens of minutes. Thus, the feedback loops provided by XP were utterly destroyed. We realized we’d have to create our own test tool for building Mesos frameworks. We called this tool Mini-Mesos.

Mini-Mesos has three key features:

• It allows developers to start a Mesos cluster, complete with any frameworks they are developing, on their local machines.
• It has an API for programmatically manipulating the cluster. We call this the “Container API.”
• It has an API for querying the state of the cluster. We call this the “Cluster State API.” This API lets us get the state of the cluster in order to run automated tests.

Because of these features, we are able to build our containerized applications and our Mesos frameworks test first, which improves the design and allows us to perform quick refactorings. Because we can instantiate the cluster, run the test suite and destroy it in a few seconds, we have a fast development cycle; i.e., we don’t have to deploy to a real cloud, or check our code in, before verifying our most recent changes.

In short, Mini-Mesos allows us to develop applications and frameworks using our favorite development method: Extreme Programming.

Conclusion

When I was a student in 1998, I messed about with Java’s reflection libraries and created a test runner that looked for methods that started with “test.” Later, I discovered JUnit, and maybe I smiled that I had stumbled across a good idea.

Back then, we needed to build our own tools to in order to do XP in practice. It’s the same now; we have to build the tools that will allow us to do XP against the cloud. Mini-Mesos is a tool that allows us to do that. It is one of a new breed that is now on a list, which almost certainly includes Apache’s Mesos, Hashicorp’s Terraform, various CI tools and of course, JUnit. Together, these new tools and old classics are forming the bedrock of a new SDK for the development of distributed systems.

For examples of how Mini-Mesos can be used, see the Mesos/ElasticSearch project on Git. More over at Mini-Mesos.

Cisco and Docker are sponsors of InApps.

Feature image: “OMG!” by Lili Vieira de Carvalho is licensed under CC BY-ND 2.0.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.