Facebook Yarn Unravels Dependency Management Issues for JavaScript Packages – InApps 2022

Facebook has released as open source a new JavaScript package manager the company developed in conjunction with Google, Exponent and Tilde, called Yarn. Yarn is designed to be a faster, more reliable and more secure npm client that retains access to the vast npm (Node Package Manager) registry while solving issues that arose when Facebook, among other companies, attempted to scale npm to keep pace with exponential workforce and codebase growth.

Facebook has used JavaScript packages and Node’s npm client for years, and many of the company’s projects — like React — depend on code from the npm registry. Facebook’s software engineers, however, found they were increasingly spending time building individual workarounds to issues as they popped up — workarounds that, in turn, often created new issues.

The most frequent and problematic of these issues included a lack of consistency across different machines and users when installing dependencies, and the increasing amount of time required to pull dependencies in. There were also security concerns regarding the automatic execution of code from certain npm dependencies.

“Yarn guarantees that the exact same set of dependencies are downloaded on every install, reducing the potential for conflicts across different machines.”

“It’s no secret that major frameworks like Ember, Angular and React stress the limits of the official npm client,” wrote Ember.js co-creator Yehuda Katz, in a blog post about the release. Katz came on board in the later stages of Yarn’s development as an active contributor to the project along with “a couple of dozen other people from a bunch of companies.”

Facebook was far from the only company experiencing npm growing pains, and the original Yarn.js (not to be confused with YARN, the popular Hadoop scheduler) development team reached out to other engineers in the community.

Exponent, Google, and Tilde all contributed brainpower to build out the Yarn client to work seamlessly on every major JS framework, as well as be extensible to other languages.

Yarn is designed to outright replace the npm client in existing workflows while remaining fully compatible with the npm registry. Yarn’s creators say it has the same feature set — while operating faster, more securely, and more reliably — is also set up as a fully open source community project with its own GitHub organization, using the same governance model that has been effective for Rust and Ember.

According to a Facebook blog post:

• “Yarn is ultra fast. After installing a project for the first time, Yarn allows developers to add and update dependencies in just seconds. Projects of any scale will see the benefits of Yarn; the larger the code base, the larger the benefit. In our tests, we saw improvements of 10x on average.

• Yarn guarantees that the exact same set of dependencies are downloaded on every install, reducing the potential for conflicts across different machines. With Yarn, engineers working at any scale still have access to the hundreds of thousands of packages in the npm registry that they need for their projects, with the additional benefits of moving fast and having better control over the way code is executed.”

This “guarantee” is based on rethinking part of the Node ecosystem. The npm client functions non-deterministically, meaning that dependencies go into a given node_modules directory in the order they’re installed by a given user — thus, the structure of a node_modules directory can differ from one person to another, potentially introducing maddeningly obscure “well it works on my machine” bugs.

To resolve versioning issues, Yarn uses lockfiles and a deterministic install algorithm. This locks the installed dependencies to a specific version and ensures that every install produces the same exact file structure in node_modules across all instances. Further, the written lockfile is concise, to ensure that review is simple, and uses ordered keys to ensure that changes are easily made.

“Yarn enables engineers to move faster and with confidence when using shared code so they can focus on what matters — building new products and features,” wrote Facebook engineers Sebastian McKenzie, Christoph Pojer and James Kyle, in the blog post.

Does this mean that npm is broken?  Far from it, say those involved most closely with Yarn’s development.

“For all of the complaints people have about the official client, it does a whole lot that people rely on, and the npm team has done a lot to improve it over the years. I genuinely respect their work, and believe that the hard work associated with maintaining a project the size and scope of the npm CLI client is vastly underappreciated, “ Yehuda Katz wrote.

After all, npm is the most popular JavaScript package manager. It’s a gateway to more than 300,000 packages in the npm registry, which sees up to 5 billion downloads every month and is used by more than 5 million engineers.

Those not working at the extreme scale of Facebook or Google are unlikely to experience the issues that drove the creation of Yarn as an alternative. Really it’s a “more the merrier” situation, say those most involved in the Node.js universe.

“JavaScript and Node.js have grown tremendously over the last few years,” said Mikeal Rogers, community manager of the Node.js Foundation. “As enterprises increasingly leverage Node.js to scale and meet the needs of today’s mobile and device-driven world, the demands on the package ecosystem continue to grow as well. It’s great to see continued investment in Node.js and the Node.js Ecosystem.”

Still, others are happy for an alternative to npm.

“This is a huge leap forward for the JavaScript community — probably more than many people will realize right away,” wrote one commenter on Hacker News, who likened Yarn to Rust’s package manager Cargo.