Builder: add platform-agnostic support for dependency packages

[alexlarsson]

From @pulb on July 13, 2016 17:6

Currently Snappy has a tremendous advantage over Flatpak: it supports creating .snap bundles with included dependencies pulled from .deb packages. Look at the endless dependency list of the VLC snappy build file for example. A developer can create a .snap bundle with many dependencies like this in a...snap, whereas he has to collect all dependency source tarballs and automake build settings in order to build an equivalent Flatpak bundle. Honestly, if I were an unbiased developer, I'd rather setup an Ubuntu VM and quickly build a bundle for my project with Snappy (snapcraft) before going through the hell of finding dependency sources, propper build settings and fixing compilation errors.

So I'd propose to incorporate a generic package plugin interface which can be implemented distro specific (i.e. with backends for apt, dnf, pacman and so on).

Copied from original issue: flatpak/flatpak#183

[alexlarsson]

From @matthiasclasen on July 13, 2016 17:27

See #87 which I think is a similar request. I don't think building a PackageKit-like package system abstraction is a great idea, but some support for packages as modules is certainly in scope.

[alexlarsson]

From @pulb on July 13, 2016 17:32

Ah, I didn't see that one, thanks. Yes, it's similar, but please don't make it rpm-only.

[alexlarsson]

From @pulb on July 13, 2016 17:48

IMHO this should be on top of the priority list - convenient features like this look like a very convincing selling point for developers currently comparing new distrubution solutions.

[alexlarsson]

You can't just take pre-existing builds and put them in flatpak and expect them to work though. Flatpak apps and bundled libraries need to be built in a special prefix, and linked against the libraries in the runtime. You can't just pick the ubuntu packages as-is (and it only works in snap because snap enforces that you use the base libraries from ubuntu core).

I.e. its not really possible to make "platform-agnostic" package support.

[alexlarsson]

From @pulb on August 22, 2016 15:46

Well, it does work with packages that don't make use of a hardcoded prefix. For libs and runtimes (like Mono) that's the case very often. So launching a distros package manager (e.g. pacman -S mylib --root ~/builder/myapp/libs ...), adding /app/libs to LD_LIBRARY_PATH and just building the app itself with the /app prefix should be sufficient in many cases.

[alexlarsson]

Yes, this is how the rpm support referenced to above is meant to work. I.e. its for specialized rpms, and will not just work with any random, non-relocatable package.

[amtlib-dot-dll]

Question: Why snappy can use pre-compiled .deb packages while we cannot?

I guess that snappy uses something debootstrap-ped as their base image, just like the Ubuntu installer, and everything can be managed by apt and dpkg

Our base image uses something called yocto

I don't know much about that, but there is no rpm or dpkg inside it. Package managers are missing (except pkg-config)

Can we just write some "converters" that turns SRPMs or deb source files into shared modules or something better? No one wants to maintain another Linux distribution, if we maintain those packages by ourselves. The effort to maintain such a package system is exactly the same as maintaining a distro

That's also why I open flathub/flathub#308

[TingPing]

Why snappy can use pre-compiled .deb packages while we cannot?

Because the snapcraft tool simply extracts Ubuntu 16.04 packages into a directory. Thus all .deb files ABI compatible with Ubuntu 16.04 will work.

The Flatpak runtimes are not based upon Ubuntu 16.04.

Can we just write some "converters" that turns SRPMs or deb source files into shared modules or something better?

It is possible to do (though can't be guaranteed to always work).

[amtlib-dot-dll]

Thanks for your reply

The Flatpak runtimes are not based upon Ubuntu 16.04.

So, why not, or why not base our runtimes on other distros like Fedora, Alpine, CentOS or Debian?

[alexlarsson]

Its very much possible to build flatpaks from debs or rpms. See e.g. https://git.collabora.com/cgit/user/smcv/flatdeb.git/tree/README for a debian version or the work fedora atomic workstation is doing for an rpm based one.

The main issue here is that the app will rely on a runtime based on the dependencies that the .deb you're using have, because the debs only work with those exact dependencies (since there is no ABI compat between distros or distro versions). So you will need one runtime per distro and per version of that distro.

Its doable, and people are doing it, but long term it does not imho lead to a good position where you have 25 distros installed on each users machine just so that some developer didn't have to rebuild a dependency.

[alexlarsson]

Or alternatively, we can go the snappy route and pick a distro and distro version and declare (like canonical) that everyone must use that base. I don't think this is a good idea either.

[amtlib-dot-dll]

I thought that this is a RedHat-sponsored project and it is Fedora or RHEL oriented 🤔

[TingPing]

It isn't Fedora or RHEL oriented.

[alexlarsson]

It is open source / free software oriented.

[alexlarsson]

I’m not sure what you mean by that? Is flatpak more open source if we forbid packaging non-free software with it?

We want the core software to be flexible and not decide what you should run. On a higher layer we might recommend a particular runtime or ship software built a specific way or from a specific repo. But that should not keep you from being able to do something else if you want.

[amtlib-dot-dll]

Sorry, I led this conversation to a wrong topic

[gasinvein]

Any update on this?
I totally agree with @pulb saying

this should be on top of the priority list - convenient features like this look like a very convincing selling point for developers

[alexlarsson]

@gasinvein Its not that easy though. Its like saying you should be able to use debian packages in fedora, because that would be very convenient for fedora users. Its not wrong, it would be, but its not actually possible. If you have any specific ideas on how things could be easier, but it has to be doable.

[alexlarsson]

Eh, if you have any specific ideas, those would be welcome.

[gasinvein]

As far as I understand, Flatpak isn't tied to flatpak-builder and it's completely possible to create ubuntu-based runtime (with debootstrap or so) and put ubuntu's packages into app budle built on top of that runtime.
The only design-related obstacle I see is the prefix, which should somehow be altered in packages.
Apart from that, we only need some tools to make building package-based app/runtime easier.

[alexlarsson]

Yes, that is possible, and e.g. fedora is planning to do that.

However, anyone doing this is now essentially maintaining a runtime, and I wouldn't call that "easy". You went from "I have to specify how to build some dependencies" to "I maintain a distribution fork".

[gasinvein]

Well, maintaining such runtime doesn't look harder than maintaining a regular container for me. I we have a tool to automatically pull listed packages from a repository and install them, isn't it easy?

[alexlarsson]

That's just creating it. Maintaining means regularly updating it when there are bugfixes and security updates, accepting and handling bug reports, testing to ensure new versions keep apps working. It also means distributing the runtime to users, and maintaining the servers and networks and whatnot for that.

Fedora will do this for the fedora runtime, and the end result will be that it will be easy to use fedora packages to create flatpaks. This is a large investment by the fedora community though, and is not "easy".

[gasinvein]

I think if we are just pulling binary packages from a repository, we only have to watch for ABI changes. It shouldn't be similar to maintaining a full-featured distribution fork.
How does Canonical do that for core snap?

[alexlarsson]

They do the work.

[alexlarsson]

And, its not an automatic thing. There are a lot of smalle ways a core snap or a flatpak runtime differs from the host install, and when such cause problems they pay people do debug, update, test and distribute the fix.

You're of course free to set up some automatic scripts to try to do this, i'm just saying that it will not be anywhere near trivial.

[gasinvein]

set up some automatic scripts to try to do this

The whole issue is about such scripts, isn't it?
I just say that installing binary packages into app/runtime should be easier, maybe directly supported by flatpak-builder.
Personally I see this as a manifest syntax extension, something like this:

id: some.package.based.App
packages:
  repository:
    format: deb
    url: http://archive.ubuntu.com/ubuntu
    dist: xenial
  stage-packages:
    - libfoo0
    - libbar1
  build-packages:
    - libfoo-dev
    - libbar-dev
modules:
  - name: the-app:
    ...

[matthiasclasen]

the fedora flatpak build support is in place now. Might be worthwhile looking at that for inspiration

[mildred]

Another way to help with dependencies would be to have a common catalog of libraries as suggested in #103

if we do that however we become a distribution. We must take care of software updates, including security updates, ... Probably the modules should be associated with a license so we can check for licensing issues too. It's not easy to do it at all.

[christian-rauch]

Could flatpak-builder integrate support for homebrew to fetch source scripts and pre-compiled binaries? This would dramatically improve the reusability of software packages and reduce compilation time and thus a more efficient use of the build servers.

[cjao]

Now Red Hat has its own "LTS" runtime based on RHEL 8. That runtime will be maintained for 10 years and should be ABI compatible with other RPMs in the Red Hat 8 universe. Would it make sense to a) integrate that runtime into the Fedora flatpak build system, which would allow developers to easily bundle existing RPMs, or b) revisit extending flatpak-builder to support RPM modules as suggested earlier in this thread? The status quo for flatpak currently requires each third-party application developer to duplicate the work of package maintainers for each dependency and for each dependency's dependency.