What to Expect (and Not Expect) from Linux Universal Packages – InApps Technology 2022

Last June, Canonical Software launched the development of Snap packages, which are intended to work on any Linux distribution. A week later, Red Hat announced its own version of universal packages, called Flatpak. In the months since, both Flatpak and Snap have been promoted as the solutions to the problems with traditional package formats. However, the solutions are not as complete as advertised and call for a rearrangement of the responsibility for security for which free software is simply not prepared.

Traditionally, Linux has relied on two main package formats, .deb (Debian) and .rpm (Red Hat). Although structured differently, both .deb and .rpm packages consist of upstream software customized for a particular distribution, plus the scripts to install any dependencies — that is, required libraries, utilities, and other packages — not already installed on the system. This arrangement restricts the required hard drive space to a minimum, and both formats have served Linux well for almost two decades.

However, in recent years, these traditional formats have come under increasing criticism. “Linux users are increasingly expecting more of a consumer experience for application — much like the app experience on a smartphone, for example,” says Thibaut Rouffineau, Canonical’s head of Internet of Things marketing. Rouffineau criticizes the distributions that use .deb or .rpm for having “strict and complicated” requirements that slow the introduction of new versions and complicate the support of Linux by requiring a separate version of the package for almost every distribution.

By having packages created by upstream developers and including all dependencies as part of each package, Snap and Flatpak are intended to simplify and speed up the delivery of the latest software to users, while giving upstream developers more immediate feedback.

Containers have no mechanism for detecting the need for updates, or for automatically updating, and can, therefore, be a source of vulnerabilities in themselves.

Alexander Larsson of the Red Hat desktop group, and the original developer of Flatpak, notes that universal packages can add another level of security by installing in containers that isolate them from the rest of the system. “Having a way to separate security domains in a single desktop makes it possible to protect against [intrusions and loss of privacy] in a much better way,” Larsson said.

The fact that these advantages have been widely announced and stated on Snap’s and Flatpak’s home pages obscures for many the reality that these benefits are still in development, and sometimes more hopeful than realistic. For example, Larsson mentions that containers are optional for Flatpak, and that the level of sandboxing varies, so that “few Linux apps are possible to fully contain at this moment.” Moreover, both Rouffineau and Larsson see traditional and universal packages as co-existing, rather than one eventually replacing the other.

Yet, even so, universal packages are being closely watched by distributions. OpenSUSE Chairman Richard Brown, for instance, speculates in a recent FOSSDEM presentation that universal packages could reduce the amount of work required to maintain distributions. For this reason, despite the exaggerated casual claims sometimes heard, universal formats could still be a game-changer for Linux packaging.

The Limits of Containers

Observers have widely noted that, with Snap developed by Canonical, which builds the popular Debian derivative Ubuntu and Flatpak by Red Hat, the end result could simply be another version of the division between the .deb and .rpm formats. However, this observation offers nothing more useful than irony. Other considerations are far more important.

To start with, containers are not the universal security solution that is sometimes assumed. In April 2017, programmers at North Carolina State University released a study of security vulnerabilities on Docker Hub. In 356,218 community-contributed images and 3,802 official images, the study identified an average of 153 vulnerabilities in the community images, and 76 in official images, most of them of high severity. In another study, the writers of Banyan Blog found that over thirty percent of official images in Docker Hub contained High Priority security vulnerabilities.

In other words, while containers may provide security against intrusions, nothing protects users against the actual contents of containers. As Debian developer John Goerzen notes containers have no mechanism for detecting the need for updates, or for automatically updating, and can, therefore, be a source of vulnerabilities in themselves.

As pointed out by industry observer Adrian Coyler, who summarized the North Carolina State study, a large part of the problem seems to be that for convenience new packages often borrow dependencies from older ones, making the perpetuation of vulnerabilities all too possible. In fact, even a containerized package that is up to date when created may later prove to have vulnerabilities yet continue to be used. With the introduction of automatic tools such as the Docker Security Scanning Service, these problems may be mitigated, but, even so, much still depends upon the conscientiousness of a package’s maintainer.

Policy vs. Technology

All of which brings up another point: although technical communities instinctively search for solutions in new technology, as long-time Debian maintainer Josh Triplett observes, “technology doesn’t replace the need for policy.”

Asked on a mailing list to compare the .deb format with universal packages, Triplett does not defend the technical format at all. “The .deb format in isolation just represents an archive of files; what makes one into a real Debian package is Debian Policy, he says. “Debian without the .deb format would still be Debian; Debian without Debian Policy would just be SourceForge or Rpmfind” — that is, collections of packages and source code with no overall quality control.

Triplett is referring to the Debian Policy Guideline, a lengthy document that details everything that a package for the Debian distribution must and must not contain, and how it must interact with other packages. Where files should be placed, how logs should act, the formats for xservers and terminals, how to add a menu item to a desktop environment — all this and more is specified in Debian Policy.

Before a Debian package can move from the Unstable repository to Testing and Stable, it has been repeatedly scrutinized for its conformity to Debian Policy. It is this scrutiny, Triplett is stating, that makes Debian the standard it has become in Linux and the source of two-thirds of all active distributions, especially those concerned with security.

Nor does the policy end with conformity to Debian policy. As John Goerzen blogs:

If you run a Debian system, the combination of unattended-updates, needrestart, debsecan, and debian-security-support will help one keep a Debian system secure and verify it is. […] Debian’s security team generally backports fixes rather than just say “here’s the new version”, making it very safe to automatically apply patches. As long as I use what’s in Debian stable, all layers mentioned above [everything] will be protected using this scheme.

So far, neither Snap nor Flatpak has a policy anywhere near as complete as Debian’s. Nor, at this stage of development, should anyone expect it to. Yet in the end, whether Flatpak and Snap can deliver all they promise will be dependent on the policies for building packages, not just on their technical features.

Upstream vs. Distros

Other questions that arise are: who is going to enforce quality control on universal package? And, closely related, at what point in the release process will enforcement take place?

As Triplett and Goerzen make clear, traditionally, quality control takes place at the distribution level. In many cases, upstream developers traditionally provide only source code, leaving packaging entirely up to the distributions. At the most, they provide packages of varying quality for a few popular distributions — usually, Ubuntu.

This arrangement is why complaints about the difficulty of writing applications when so many Linux distributions exist are invalid — under the current arrangement, upstream developers are not responsible for tailoring their code to the distributions; the distributions do that for themselves. For example, when Steam was ported to Linux, the original development work was developed for Ubuntu, and within weeks other distributions had adapted the Ubuntu version for their own distributions.

Yet if universal packages are released without the intervention of the distro maintainers, the implication is that upstream developers will be testing packages and enforcing policy themselves. As Richard Brown said in his FOSSDEM presentation, “With these tools, Flatpak and Snappy, the only real solution is to start thinking like a distribution. You’re going to have to deal with the same responsibilities that we deal with in the distribution world.” Automatic testing may simplify the process, as it does for openSUSE, yet at some point, it still needs to be done.

Moreover, Brown warns, all distributions are not structured in the same way. They place files in different locations and sometimes use different applications to coordinate their versions of Linux. “You’re still going to have to develop and test everything on every possible distro,” Brown says. “Unless you bundle every dependency for everything you need anyway […] You’re going to end up crying.”

Brown concludes that, as the experience of AppImage, an earlier effort at universal packages shows, the idea of universal packages that can run on any Linux distribution, is currently impossible. The only way that he sees that universal packages will be able to fill their promise is if some standard like the Linux Standards Base is used on every distribution — and that is an effort that has failed in the past.

Brown is interested in the promise of speedy releases promised by universal packages. However, based on openSUSE’s experience with its Tumbleweed release, Brown suggests that rolling releases — continual updates rather a complete formal release of the entire distribution — may be a more practical solution for rapidly delivery of new releases than universal packages. Otherwise, the shift of package preparation to upstream developers strikes him as an additional responsibility that few are prepared to shoulder. Nor is the shifting of responsibility necessary, since distribution-based developers are already acting in that role, and have the required experience to do it well.

Once the Dust Settles

Because of these considerations, those who expect universal packages to replace the traditional formats are almost certain to be disappointed. Nor, to be fair, do those most involved in the development of universal packages have any such expectations.

“We see a co-existence between Snap and conventional package management systems,” Rouffineau said. “For example, deb packages offer flexibility while Snaps offer ease of use. Conventional packages are the best solution for building operating systems where a small group of people are managing a large number of deeply interconnected packages. On the other hand, Snaps are a better solution for distributing independent, stand-alone applications — whether that be on desktops or headless servers/devices.”

Similarly, Larsson said, “Flatpak is really only for desktop applications, and is not a good fit for the core operating system itself. My hope is that the separation of applications from the core OS leads to a model where the distributions can focus on improving the core, rather than competing in in how many different applications they can package.”

As universal packages become more widely used, their uses will gradually become more obvious. Already, distributions are hedging their bets by preparing releases for all formats. OpenSUSE, for example, currently supports builds for Snap, AppImage, Docker, although not for Flatpak, which Adrian Schröter, openSUSE’s build infrastructure manager describes as still having “unsolved problems of reproducible builds.”

Meanwhile, while universal packages are still being developed, knowing what they can and what they cannot do is important to counter the rising of unrealistic expectations.

Red Hat is a sponsor of InApps Technology.

Feature image via Pixabay.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.