The Good, the Bad and the Ugly – InApps 2022

A few weeks ago, Canonical announced cross distribution support for Snap application delivery mechanism, one that could support mobile, desktop and server-based applications. The announcement ruffled some feathers within the Linux community, who saw scant evidence of any other Linux distro supporting Snaps.

Snap is Canonical’s attempt to refine the app packaging and delivery mechanism on the Linux platform. The origin of Snap finds its roots in Click, a solution that Canonical created for its mobile platform back in 2014, to handle the complexity of delivering apps to Ubuntu phone and tablet users, an ecosystem far different from desktops and servers. Additionally, mobile also needed to confine apps and sandbox them, so that they would not compromise the entire platform.

Click enabled developers to bundle all dependencies into a single app package, so users and developers didn’t have to worry about conflicting systems and app libraries. It allowed developers to use the latest libraries in their apps, to offer new features as they were decoupled from the libraries installed on the system.

Beyond Mobile to Servers and IoT

Click evolved into Snap and Canonical extended its potential use to the server, cloud, Internet of Things (IoT) and desktop spaces. With the release of Ubuntu 16.04 in April, Canonical officially brought Snap to the Ubuntu desktop.

Canonical is also developing Snappy Ubuntu Core, a tiny Linux-based operating system for large-scale cloud container deployments, IoT devices and much more, and anything that needs transitional updates and tiny footprint. Snappy Ubuntu Core works more or less like Google’s Chrome OS and CoreOS where it offers transitional image based updates for the system and apps that can be rolled back. Simultaneously, it offers the confinement that is known in the container world.

Snap is the soul of that technology.

The Linux Mess of App Delivery

The existing app delivery model on Linux has many flaws. The Linux world is mostly divided into two camps of app packaging: rpm and deb. Also, there are different versions of the same distributions that use different libraries, so it becomes quite challenging for app developers to package apps for Linux.

As a result, app developers can’t take advantage of the latest libraries and offer more features to their users if the distribution is still using older libraries.

In most cases, open source app developers don’t package their apps for distributions. Popular or long-established apps are usually packaged by the distro developers. Newer or less popular ones are either packaged by the upstream for each distro they want to support, or not offered as a distro package at all. When a new version of an app is released, these developers test the app and then make it available for the distro. As you can see, instead of the app developer just writing one app for the entire platform, which in this case is Linux, there are many people bringing that app to users.

It adds more complexity.

There are cases where Windows and MacOS get the latest versions immediately, as apps are packaged by primary developers, whereas Linux distros can take a while. That’s not all; some distros get them fast, and some have to wait for weeks or months.

In the case of commercial apps like Skype, Dropbox, Chrome, the onus is on the primary developer to package apps and offer binaries for different distributions. Many apps don’t work on older, yet supported, versions of many distributions.

In a word, it’s a mess.

Even the creator of Linux, Linus Torvalds, is not happy with the existing app delivery model on Linux. He doesn’t offer binaries of his apps for the Linux platform citing this mess. Dirk Hohndel, the maintainer of Subsurface (a project started by Linus), explained, “The current situation with dozens of distributions, each with different rules, each with different versions of different libraries, some with certain libraries missing, each with different packaging tools and packaging formats … that basically tells app developers ‘go away, focus on platforms that care about applications.’”

“Application packaging is hugely important, but so is the provenance and maintenance of all the pieces involved,” agreed Gunnar Hellekson, director of product management for Red Hat Enterprise Linux.

Snap was designed to bring some order to this state of affairs. But whether it will be adopted is another question entirely.

Hellekson argued that tools like Snap don’t “magically solve interoperability problems between Linux distributions, and actually shift more responsibility for upkeep and security to the application vendors. In this way, they could make developing on Linux even harder despite the many benefits to the users.”

“Snap’s advantages directly relate to its disadvantages … in essence, Snap is the equivalent of static linking — it is convenient on the one hand, [but] creates duplication and challenges with respect to security (and security patching) on the other hand,” said Gerald Pfeifer, SUSE vice president of products and technology programs.

However, not everyone is of the same opinion. Brandon Philips, co-founder and chief technology officer of CoreOS, told us that containers are the future of apps on Linux. But Docker containers can’t be used everywhere. They have their limitations. What we need is a container-like solution, something like Snap.

During another interview, leading Linux kernel developer Greg Kroah-Hartman told us that Android has been doing containers “for your applications for eight or nine years now. Every application runs in its own namespace. It can’t see outside of that. That is the way you have to do it. That is the future of Linux and the way systems work.” He said that Snap is the same concept, same idea.

Snap Goes Cross Platform

The success of Snap very much depends on its adoption by the larger open source community. What good is an app format if no one is using it? As an Ubuntu-only technology, which may not even get officially supported by its derivatives like Linux Mint, KDE neon or elementary OS, Snap could face a not so successful career.

When Canonical announced Snap, it claimed in a press conference that it has been working with various distributions to add support for Snap.

Canonical said that Snaps are natively available for Arch, Debian, Fedora, Kubuntu, Lubuntu, Ubuntu GNOME, Ubuntu Kylin, Ubuntu MATE, Ubuntu Unity, and Xubuntu. Canonical is also working on validating Snaps on CentOS, Elementary, Gentoo, Mint, OpenSUSE, OpenWrt and Red Hat Enterprise Linux.

Canonical and SUSE are currently not engaged in the development around collaboration on Snaps, and in fact, it appears as though this was not the result of open community design and development,” Gerald Pfeifer, SUSE.

Canonical’s announcement that they worked with other distributions attracted some criticism from some members of the open source community. Fedora developers didn’t take kindly to this announcement and criticized Canonical stating that there was no collaboration whatsoever between the two projects.

When we followed up with Canonical founder Mark Shuttleworth, to ask about these charges, he responded, “We did brief with the Red Hat desktop team and are always open to working with all Linux distributions to enable Snaps.”

This turned out to be news for Red Hat, however, which shepherds the Fedora distribution.

“While we work with Canonical in several communities, we were as surprised as everyone else when they announced our involvement on this particular initiative. We are open to more discussion on this, and to see how the Linux community overall wants to work together,” Hellekson said.

SUSE officials also downplayed any collaboration. “Canonical and SUSE are currently not engaged in the development around collaboration on Snaps, and in fact, it appears as though this was not the result of open community design and development,” Pfeifer said.

What seemed to have happened is that Canonical worked with some developers, perhaps only a handful, of other projects and they assisted in packaging Snaps for other distributions.

Canonical is Not Perfect

Part of the pushback that Canonical may have gotten may be a byproduct of how it does business.

For one, other Linux companies, such as Red Hat and SUSE, maintain a very clear separation of powers when it comes to Fedora and openSUSE. Fedora and openSUSE are fairly independent, community-driven projects where decisions are made by the community members and not the sponsoring companies.

In contrast, Ubuntu is a Canonical product that also happens to be a community project. Due to that contrary to openSUSE and Fedora, Canonical has direct control over Ubuntu. We already learned in the case of ownCloud; it’s really hard to maintain a balance between the needs of the community and the company; it often ends up in a disaster like ownCloud.

The question arises of how one ‘works’ with a project?

Also, many feel there is a lack of transparency with Ubuntu. At times many leading Ubuntu contributors have expressed frustration as Canonical kept major announcements around Ubuntu under wraps, hidden even from core members. Ubuntu members had no clue about Ubuntu for Windows or the ill-fated Edge project.

“We don’t hold things back,” Red Hat CEO Jim Whitehurst responded to a question about that company’s approach to open source.  “When we build things, we don’t hide it from people to do a big reveal and say ‘here is the code.’ We don’t do that. We work in communities from day one, all the time. I do think we try to be really good participants, and I think that comes through.”

On top of these differences, Canonical is also known for making some missteps when it comes to releasing information. This certainly seems to be the case for announcing cross-distribution support for Snap; the wording that irked Fedora developers was “Developers from multiple Linux distributions.” To be clear, Canonical never explicitly stated in its press release that it worked with Fedora. And based on the information we collected, there was no “official” engagement with Linux distros such as Fedora, Red Hat, SUSE or others.

That said, the question arises of how one ‘works’ with a project? If a developer working on an app collaborates with an Ubuntu developer, a Fedora developer and an openSUSE developer to bring that app to those platforms, would it be wrong to say the developer worked with multiple distributions to bring the app to them? Or does the dev need official engagement with project leaders of these distros to be able to make such a statement? It’s a distributed, open source community, after all.

Whither Snap?

The bigger question is, will Snap be officially adopted by other distros? That’s where things get a bit complicated. It becomes a technological and political issue.

First of all, Snap is not the only solution around. There are many projects that tried to offer distro agnostic packaging ecosystems for the Linux ecosystems. But it can be difficult to find consensus within the Linux distro world. There is AppImage and Flatpak, which offer suitable alternatives to Snap; these projects have been around much longer than Snap.

So why didn’t Canonical adopt these solutions instead of starting Click or Snap?

“We looked at it, and found that AppImage works perfectly for ‘leaf-node’ desktop applications,” said Shuttleworth. A leaf-node application is one that doesn’t need to integrate with any other app. Snap was designed more to package apps with dependencies and related applications. “Snaps have a range of capabilities for describing those connections — interfaces, plugs and slots, as well as shared files and so on.”

In addition to that, Shuttleworth told us that Snap was designed for higher levels of security compared to AppImage, partly because the team working on the mobile phone was focused on security and efficiency, and partly because Snap uses newer kernel capabilities.

Flatpak supposedly fell victim to the rivalry between Canonical and Red Hat as it was being developed by Fedora people. When I asked, Shuttleworth dismissed it as a project being developed by a single developer.

That explains why Canonical went ahead with Snap.

Now, will other distros adopt it? Fedora developers have already declared that they won’t use Snap. A Fedora developer wrote on the mailing list, “We, of course, have zero plans to adopt Snappy in Fedora, and in fact, multiple Fedora developers are working on a competing solution, Flatpak.”

Red Hat previously embraced a Canonical project for system initialization called Upstart. Hellekson said, “These are very early days for all of these packaging efforts, and we look forward to working with all the related communities to make life as easy as possible for Linux users.”

Beyond the rivalry between Red Hat and Canonical, there are some technical issues which may discourage other projects from collaborating on Snap. One such reason is that developers have to sign a CLA (Contributor License Agreement) if they contribute to the Snap project. Michael Hall, a Canonical developer, said that a “CLA is needed when a developer wants to contribute to the Snap project itself. There is no CLA required to create or distribute Snaps of applications.”

CLAs allow Canonical to change the license of the project. That’s beneficial to Canonical, as they can relicense a project, allowing it to be used with non-free or incompatible licenses. Developers usually don’t like CLAs.

Frank Karlitschek of Nextcloud told us that it is terminating CLAs from Nextcloud, as he felt CLAs don’t have a very good reputation within the open source community.

When we asked Red Hat if CLAs would dampen collaboration around Snap, Hellekson said, “Contributor License Agreements do make it difficult for us to participate. We believe straightforward, open source licensing is a better, more inclusive approach.”

The response from SUSE was much the same. Pfeifer said, “SUSE engineers and staff continue to contribute to a plethora of open source projects, both on company time and our own, and where necessary we have signed CLAs. However, wherever and whenever we initiate or influence projects, we do try to stay clear of CLAs. We generally do not like CLAs that assign a copyright that enables organizations to re-license open source code using a non-open source license.”

When we asked Shuttleworth if he would consider dropping CLAs from Snap to gain broader support from other distros, he started off by explaining why CLAs are good, but then also said that “if that would make a difference, then yes, we would find a way to drop it.”

And Then There was One?

Over time, maturity, popularity and technical superiority of one over the other may leave us with one solution. We have seen that before with Upstart and systemd, where systemd emerged as a superior technology and became the default on all three leading Linux distros: Ubuntu, RHEL and SUSE.

In either case, both technologies are in very early stages of development, and it’s hard to say which project will succeed. However, it’s pretty clear that in the container-centric world, Linux does need a new app delivery mechanism.

Many companies and communities have expressed support for solutions like Snap. Samsung and Dell joined Canonical in the press call and stated the need of such a solution. On the desktop end, projects like LibreOffice and Firefox have already expressed support for Snap.

Time will tell whether Snap will be adopted outside of Ubuntu or not. Whatever the future holds for Snap, we should give Canonical due credit for pushing the envelope, forcing the rest of the Linux community to finally focus on solving a problem that we either had become so comfortable with or, perhaps, never had the motivation to solve.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.