Meteor Galaxy Containerizes JavaScript Apps for Full-stack Management – InApps Technology 2022

After evaluating orchestration technologies including Kubernetes, Mesos and Docker Swarm, the Meteor Development Group decided to partner with Amazon to build out Meteor Galaxy, its cloud platform for operating and managing JavaScript applications.

Using a container-based architecture, Galaxy is built atop Amazon Elastic Compute Cloud (Amazon EC2) and Amazon EC2 Container Service (Amazon ECS).

“If your goal is to make building applications as easy as possible, the effort can’t stop at the water’s edge when you’ve written the app, it has to include the challenge of running the application in a modern cloud environment. That’s quite difficult for the modern development team,” said Matt DeBergalis, Meteor development group co-founder.

In building Galaxy, the company dealt with a range of issues including managing large numbers of containers, providing high availability to customers, scaling and providing responsiveness to development teams.

It required a runtime in the cloud that manages connected clients.

“We had very specific custom requirements around runtime scheduling,” DeBergalis explained. “We used ECS Scheduler APIs to build a smart application container instance scheduler, and swapped out the default ECS scheduler for our custom scheduler. We also optimized Galaxy to solve for the ‘noisy neighbor’ problem of multi-tenant platforms where certain apps consumed a disproportionate amount of computing resources compared to peer apps. This required monitoring application resource utilization, analyzing trends, and then defining container configurations that met the needs of all our customers.”

The challenge of containers goes beyond running a single process inside a container, he said.

“Where it really gets interesting is when you want to manage a large number of them in a production setting. The dirty secret of containerization is that for production use at scale, it’s still very, very early. When you actually get down the brass tacks of what it takes to run thousands of containers in a reliable setting on cloud infrastructure, you run into all these challenges that surround the containers themselves. How do you reliably restart them in the face of system failure? How do you monitor such a large number at the same time in an automated way? How do you collect performance metrics and logging information on containers? That was the big stumbling block among our own developers,” he said.

“Running a container in production at scale — that was a big ask,” DeBergalis said.

Developers want to be building software; they don’t want to be building the devOps around that software so they can run it. “We found the advantage of container orchestration-as-a-service is that it’s a simpler and faster way to solve a lot of the challenges you run into. Some of them are just maddening,”  DeBergalis said.

He says the integration points worked out with ECS helped the company get Galaxy to market faster.

Meteor, which bills itself as a full-stack solution using JavaScript, is designed to facilitate applications that partly run on the mobile device or the browser and partly runs in the cloud as a set of Node.js processes. Those parts talk to each other through a web socket. Data is pushed from the server to the device to update the screen while people are using the app.

Galaxy is focused on running Meteor apps at scale in the cloud.

“There are a few parts to that, but the biggest is you want to securely run inside of individual containers as many copies as necessary and give people an easy way to scale that number up or down, then you want to able to manage that network connection for each active user – each open browser tab or mobile device that’s using the app – down to a particular container,” DeBergalis said.

“So a lot of technology in Galaxy is about how do we select the best back-end container for each incoming user, what’s the right load balance when you have web sockets that represent a session for a user that could be an hour or even a day – it’s very different from a traditional HTTP app. And how do we keep track of all those clients?”

Meteor allows you to push updated code into the user’s device, which requires careful coordination between the server and the client, and Galaxy allows that to happen so that the user doesn’t notice an interruption, he said.

One early user of the framework has been Ikea.

“When they sit down with their partner to build that app, they don’t want to figure out for each of those questions which piece of technology they should use. JavaScript is moving so quickly today, it’s so fragmented, taking on the burden of deciding on a stack, assembling a stack of components to do all this stuff ends up being more work than writing the software,” DeBergalis said.

“It’s different from the Java ecosystem or .NET ecosystem where you have those conventions, a standard way of doing things. So with Meteor, when you sit down to write that application on day one, you’re actually writing code rather than deciding which version of which view layer you should attach to which version of which web socket library and on and on and on. That what we mean by ‘full stack’: It addresses all parts of the application,” DeBergalis said.

Changing course

Launched in 2012, Meteor has raised $31.2 million in three rounds from investors including Andreessen Horowitz and Matrix Partners.

Founders Matt DeBergalis, Geoff Schmidt and Nick Martin originally set out to create a travel recommendation site. As part of the startup incubator Y Combinator, they built the original version of Meteor as its basis and found the other teams struggling to also build that basis for their apps. Then they realized their framework was more valuable than the travel app.

Meteor doesn’t require developers to know myriad languages such as Ruby or Python, just common JavaScript.

The company released version 1.0 of its eponymous platform in late 2014 and in version 1.1 added support for Windows and MongoDB 3.0. Meteor supports OS X, Windows, and Linux. Version 1.2 made ECMAScript 2015 its official language and added support for the Angular and React view engines alongside traditional Blaze templates, as well as additional support for mobile application development. In December, it announced a 1.3 release for the Angular package.

Independent JavaScript developer Ryan Burgett wrote in a blog post that Meteor has already solved problems that developers have to re-solve every day.

“Meteor has its problems, but for many applications – even in Meteor’s current state – there is nothing to compare to it. You don’t need to worry about HTTP calls, network or CRUD abstractions, you don’t need to worry about caching data or manually setting intervals. Those are all taken care of for me!” he wrote.

Sacha Grief, co-author of “Discover Meteor” and involved with other Meteor projects, laid out his views of Meteor’s problems, including that although it makes getting started easy, developers soon hit a wall of complexity. He believes the company’s all-in-one approach has been a liability.

“The recent Blaze/React debate clearly illustrates this gap: as one part of the community is pulling Meteor towards React, Webpack, and the rest of the JavaScript ecosystem, another is despairing the loss of the very things that were making Meteor worth using in the first place,” he wrote.

He’s urging Meteor to embrace React, citing Meteor’s strength as a framework for both the server and client side.

Asked about Grief’s rationale, DeBergalis said: “In its earliest release, the Meteor framework was an all-or-nothing stack. At the same time, the JavaScript ecosystem has changed the last 18 months.”

“To remain a relevant JavaScript app platform, Meteor has grown to support other popular frameworks and tools (e.g. Angular, React, npm). At the same time, Meteor will continue to provide official guidance on how to build apps, for example, Meteor Guide, support earlier technologies, such as Blaze view layer, and also involve the community,” DeBergalis said.

Feature Image: “Galaxy NGC 60” by Raven Vasquez, licensed under CC BY-SA 2.0.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.