What Linux Distributions Can Teach about Rolling Releases – InApps 2022

Programming practices happen fast in open source — often, it seems, faster than in programming elsewhere. The freedom from economic motive and the number of alternative projects make open source a testing ground for almost any alternative that can be imagined.

Nowhere is that more obvious than in how software is released, especially in Linux distributions, where the traditional point releases have been replaced in some cases with rolling releases, a less formal strategy that focuses on the release of individual packages or features more than the whole.

Starting with the release of Gentoo Linux in 2000, rolling releases have been adapted by other Linux distributions, the most prominent being Arch Linux and openSUSE. The collective lessons demonstrated by these distributions are worth studying for any software project.

In Linux distributions, rolling releases have become the alternative to point releases. As you probably know, in a point release — also called a standard or general release — all components are released at the same time. Major releases are given a whole number (2.0), and numbers after a decimal point denote a minor release (2.3). Typically, a point release has a rough alpha release, followed by a more feature-complete beta. Then, a feature freeze is declared, and a release candidate is issued, followed by the official release.

In some development projects, such as the Ubuntu or Fedora Linux distributions, point releases occur on a regular basis, usually every six months, although some projects, such as Debian, the interval can vary and be as long as several years.

By contrast, in a rolling release, each package is released when a new version is tested and ready. Sometimes, core features such as an operating system’s core, remain as a point release for the sake of stability, and a few projects, such as openSUSE maintain both rolling and point releases. Others, like Arch Linux, are entirely rolling releases.

The Need for Speed

Why consider a rolling release? In the case of free software, the answer is simply that the demand exists. For example, openSUSE chair Richard Brown explained that the Tumbleweed rolling release of openSUSE began because kernel developer Greg Kroah-Hartman “wanted it as his own distribution” and the release attracted other developers.

More often, however, the main reason for a rolling release is that some users want the newest upstream releases as soon as possible. “That especially applies to complex projects like GNOME or Python, which can be tricky to use alongside what distros provide,” said Bartłomiej Piotrowski, an Arch Linux developer. Piotrowski also noted that there is “less maintenance burden, as it is enough to upgrade the system once in a week or two, as opposed to a potentially troublesome huge upgrade every six months or two years,” which can often mean the loss of customized settings as features evolve and change.

In comparison, Brown notes, a point release often has trouble coordinating with upstream projects, because the upstream projects usually have their own release schedules. Efforts have occasionally been made to eliminate this lack of coordination, such as Mark Shuttleworth’s call in 2008 for upstream projects to match their schedules to downstream ones like Linux distributions, but none of these efforts have ever succeeded.

This situation, Brown said, means that development projects frequently have “to make hard decisions about keeping one [dependency] back in order to facilitate others.” As a result, users of point releases may either have to wait months or even years to have the newest features or, at best, rely on updates and third party sources whose security and stability are less thorough that their other software.

For private users, such situations are the reason for impatience, especially users of Linux and other open source software. However, as openSUSE’s Douglas DeMaio pointed out, “developers need their system to be as close as possible to relevant upstream [projects].” Too often, with a point release, fulfilling that need can “come at a cost of stability.”

In addition, DeMaio says, commercial developers may find that the difference in upstream and downstream release schedules can seriously affect their time for market. At a time when an increasing number of products depend on open source projects, this concern is increasingly important.

Rolling releases are not the only solution for these demands, of course. So-called universal package systems such as Flatpak and Snaps are being developed to help solve exactly this lack of coordination. However, these alternatives are still in development, and, according to Brown, may require both a shift of testing to upstream projects and greater standardization between projects. Moreover, rolling releases have the advantage of being already in use, with a number of models for those wishing to implement their own.

Security Potholes for Rolling Releases

Yet if rolling releases satisfy developers and advanced users, they may not be for everyone. For system administrators running networks and servers, or average users whose main interest is productivity, new releases are less important than security and stability — and the traditional wisdom is that point releases are best at providing both.

Rob Szumski, Tectonic product manager at CoreOS, defends the traditional perspective by saying that point releases “power reproducible infrastructure.”

For example, using CoreOS’s own products as an example, Szumski says, “a user can deploy Container Linux v1451.2.0 on one hundred machines and be confident that each machine will boot in the same manner, and versions of the kernel, init systems, and container runtime will be compatible with each other.”

Because of point releases, Container Linux has “a smaller testing matrix, which is easier to automate and allows for a holistic environment to test against,” Szumski said. Infrastructure management tools like Chef or Puppet can increase this control by helping to manage upgrades and patches.

However, not everyone agrees with the traditional wisdom. For example, Piotrowski suggests that the idea that point releases are more secure is a “quite often a repeated mistake. In fact, the older a ‘stable’ distribution is, the more work it takes to backport any security or regular fixes from new project releases, as the code can change a lot from the freeze point. Rolling distributions just update the package to the latest release or easily apply a patch from Git; usuall, they are faster than traditionally release ones.”

Still, even if Piotrowski is correct, rolling releases still require safeguards. At the 2016 openSUSE conference, Brown gave a presentation in which he acknowledged that, with their emphasis on individual packages, a Linux distribution as a whole can be neglected, since packages not only need to work, but to be compatible with other packages as well.

As an axiom for rolling releases, Brown offers, “In order to move anything quickly, you need to move everything quickly.” He also cautions against what he calls “passive testing:” leaving a package in a testing branch of a version repository, then releasing it if no problems are voiced.

Just as a point releases uses bugfixes and patches, so openSUSE’s Tumbleweed uses a number of safeguards. To start with, part of the process of testing Tumbleweed is automated with openQA and the Open Build Release. Just as importantly, Tumbleweed defaults to Btrfs, a file system that supports snapshots and rollbacks to earlier versions of an installation, so that users can back out of problem upgrades and continue using their systems in the way they are used to. Recently, Tumbleweed has also included for security Position Independent Executables (PIE), which loads executables at random memory addresses, making them much harder to crack.

Such safeguards can have their own costs. Brown points out, for instance, that too many snapshots can add to the size of online repositories, and that, at a certain point, supporting old snapshots might limit the ability to install new packages.

Still, except perhaps for their novelty, these safeguards seem no more of a concern than the security updates and backporting of new software that occurs in a distribution like Debian or Ubuntu.

Choosing a Rolling or a Point Release

More than anything, what the examples of Linux distributions shows is that rolling releases can be a useful alternative to point releases — but they are not for every user. Regardless of the release strategy used, testing and updating still needs to be done somewhere along the line. The difference is when they are done, the tactics for coordinating a distribution as a whole, and the intended users.

A single Linux distribution can often choose one release method over another because the alternative is supplied by a related one. For instance, the long intervals between Debian releases is compensated for the fact that derivative point releases like Ubuntu occur regularly and more often, and rolling releases like Siduction are also available.

In other software projects, a choice may be needed between a point and a rolling release, especially if the project is large. Yet even when the main strategy is chosen, other tactics are likely to be necessary to obtain some benefits of the other strategy, whether backports for point releases, or rollbacks for rolling releases. Such lessons should apply to other software as much as they do to Linux distributions and free software.

CoreOS is a sponsor of InApps.

Feature image via Pixabay.