Airbnb’s 10 Takeaways from Moving to Microservices – InApps 2022

At the recent FutureStack 2017 in New York, Melanie Cebula, Software Engineer at Airbnb, talked about the journey her company has been on and brought 10 takeaways from their experience.

The home-sharing service operates in 65,000 cities located in 191 countries. This summer saw their four millionth listing. This summer alone, 45 million guests will stay in an Airbnb listing, making a total of 200 million guest stays since 2008.

In order to make this possible, they deploy 3500 microservices per week, with a total of 75 thousand production deploys per year. The company now has 900 engineers, and are hiring.

“We basically want to have a hundred percent availability and we also want a hundred percent productivity,” said Cebula. “A hundred percent availability means that our site is completely available for our guests, hosts and employees. And a hundred percent productivity means that everyone in product can easily create, change and release features.”

Here’s what she’s learned on her journey from monolith to microservices:

1. Monolith first

Don’t start with a microservices. They still have a monolithic Ruby on Rails application that is maintained in MonoRail. It worked for a very long time and allowed them to focus on infrastructure. Starting out with microservices is pre-optimization, she said, and not necessary.

But when your company grows, it’s important to create a plan to start investing in microservices. For Airbnb, the tipping point was at 500,000 lines of code and growing fast. They realized that doubling the lines of code every year wasn’t going to be scale.

“It’s important to have that insight pretty early on and start building the components required at that point,” said Cebula.

2. DevOps culture

The concurrent creation of a DevOps culture “was really huge,” she said. “You want each engineer to be prepared to support their service because right now we have a ratio of three engineers per microservice and it’s going to continue to grow.” For Airbnb, the rise of the DevOps culture came from their SysOps group.

The SysOps team has three main responsibilities in the event of an incident. They are the first to triage the problem, coordinate the response, and communicate the response to the appropriate

internal service owners, product teams, infrastructure experts, and to third-party service owners.

Over time, Cebula said, the SysOps group has “rather unexpectedly became a huge lever for building a DevOps culture at a historically product-driven company. This becomes really important later on when you’re trying to scale mega services.”

The SysOps training is open to all engineers, who can then volunteer to be on call. They’ve opened the training up to all engineers, and it’s become quite popular, according to Cebula, with over 33% of engineers have attended to date. Attendees learn about infrastructure, current architecture and how it has evolved over time, how to set-up an on-call for their team if they’re interested, how to be an effective on-call. They also have a hands-on lab session where everyone gets paged. According to her, the engineers see this as a highlight.

“We’ve cultivated a culture where every engineer can individually build, test and release their software frequently and lively,” she said.

3. Configuration as Code

Developers aren’t great at trying to keep track of all these tools, said Cebula. But developers do understand code.

So configuration of code became really important. Having your configuration as code makes it very easy reason about, she explained. “What configuration is currently in production, whether a configuration has been rolled back, making changes to configuration, it’s all right there for them.”

4. Monitoring and Alerting

Every service should use a standardized tool like New Relic, she said. This will allow developers to monitor and alert the same way, across applications.

Also important is giving the engineers the ability to write metrics alongside feature code, making it easy for developers to write their own business metrics. Alerts are configured as code, said Cebula. They have their own Domain Specific Language (DSL) for writing alerts based on these metrics. Creating a new alert is as simple as opening a pull request.

Whenever possible, she said, they give developers things for free, like bundling statistics into a pull request.

5. Continuous Delivery

Scaling deployment for a bunch of engineers is very hard, said Cebula. But if you’ve gotten this right, every new service that’s created will have the same process and the same functionality as a monolith. The keys for success are standardization and automation.

6. Automate, Automate, Automate

Manual steps are very difficult and error-prone, she said. So automate everything you can.

As you’re breaking out these services, she suggested, “make sure you’re starting to think about what they all have in common, what should they all have, can we build our configurations for effect.”

Applications’ code is also very important. The code lives inside the repository for the feed to the code. It is automatically tested and validated as part of the continuous integration process. Basically, it’s its own little build and is deployed and consumed as an artifact. They’ve standardized formats and defaults. “So the easy thing is the right thing for developers,” she said.

And just a reminder, no snowflakes.

7. Your First Service = Bad

This will be a long and hard process, she said, pointing out that Netflix took ten years to fully transition from monolith to microservices. It’s likely your first service is going to be bad, she said. It’s very challenging and you have to remind yourself that everything’s going to be okay.

8. Services Own Their Data

They have “Democratic Deploys,” Cebula said. All devs are expected own their own features from implementation to production. Each dev is expected to deploy their changes, monitor their changes, rollback, abort, or revert if necessary, from development to production.”

9. Break Apart the Monolith

When once you’ve been creating microservices for a while you’ll see an increase in speed and automation. “It’s going to get awesome,” she said. That’s about the time you realize you still have millions of lines of code stuck in your monolith. In order to move the code, you’re going to have to work with infrastructure teams who’ve been building out microservices and bring in product teams to start breaking apart the monolith. Bringing in the product engineers is key. “You can show them these really awesome graphs about how they can just deploy whatever they want and be really fast and it will be awesome,” she said. But you have to get buy-in across the board and really invest in the transition.

10. Standardize Service Creation

This is a working in progress at Airbnb. The goal is to make all configuration is easily accessible so devs can create a service with one pull request. They create the repo and create the configuration at the same time.

The ideal thing is that you also only need to define what is separate about your service.

Feature image via Pixabay.