flatpak-builder: Allow using a local checkout as the source of a git repo

[alexlarsson]

From @alexlarsson on May 25, 2016 15:17

If you're checking in a builder manifest into the git repo of your app and using flatpak to build it, it doesn't necessarily make a lot of sense to download the git repo again. We should have some way to support using the existing repo, maybe even the current outstanding changes to it.

Not sure exactly how this would work though. Its complicated both how to reference to the current git repo, and how to reuse the current checkout without old builds affecting the new build.

Copied from original issue: flatpak/flatpak#42

[alexlarsson]

From @mwleeds on July 19, 2016 21:15

This feature would be very helpful in getting working flatpak support into GNOME Builder. I can work on it if no one else is. The first step would be to allow a path to be specified for git sources like it is for archive sources, right?

[alexlarsson]

From @muelli on August 4, 2016 19:52

FWIW: I've tried to use smth like

        "sources": [
            {
                "type": "git"
                , "url": "."

which seems to work. Sort of. I then can't convince my flatpak-builder 0.6.7 that things have changed, e.g. it'll always tell me that it has a cache hit instead of picking up new commits. Using an absolute file:// URI works better. I need, however, to create commits in order for flatpak to pick it up.

[alexlarsson]

From @cmollekopf on August 4, 2017 20:36

This would indeed be great. The crude hack that I'm using right now is:

• Define the git repo with a file url.
• Replace the corresponding directory in .flatpak-builder/git/ with a symlink to the local git repostiories .git directory
• Add a second source line of type patch with a static location
• Build using a wrapper script that always first dumps "git diff" into the patchfile

That works, but it's not pretty.

[refi64]

Ok, so I'm going to propose something a bit different to solve this. The standing issues with simply adding local checkouts and factoring in uncommitted changes are:

• It requires changing your manifest back and forth.
• You won't be able to use local source directories for anything other than Git.

I'd like to propose a more generalized solution: local overrides.

Local overrides would simply be this:

# in sources 
type: git 
# ...
local-override: '.'

Now, if the user runs flatpak-builder --use-local-overrides ..., the local override directories will be overlay mounted under a tmpfs inside the build directory.

Combine that with a --reuse-local-override-builds option to avoid clean building each time, and you've got a solution that satisfies both the general case of wanting to temporarily try local directories and the case of IDEs that build your code inside Flatpaks.

[loonycyborg]

Yeah, a way to parametrize flatpak-builder builds from outside would be really nice.

[sudden6]

I could use something like that too. To do automatic PR testing via Travis CI I would like to use the official json file, but with the master git commit and not the tag and commit from the file.

Is there another possibility I'm missing to achieve this, besides using sed to change that?

[bossjones]

Hi everyone, first of all thanks for working on flatpak-builder. I'm in a similar situations to several people in here, namely @sudden6 's situation where I want to support PR testing via Travis CI, and just in my generic development environment. I see some interesting mentions to some hacks to get by at the moment. Pseudocode that @alexlarsson shared(#13 (comment)) seems simple enough to implement, but i'm curious if anyone has any decent code samples that they're already using ( if it is shareable ). If not, I might make a dummy repo that illustrates this problem and a scripted solution to get around it( I'll post it in this git issue as well so others can find it ). Thanks in advance for reading.

[alexlarsson]

The latest flatpak-builder also supports source type of "dir", which just means it copies a locally available directory. This is often a pretty good match for CI, which typically has the CI itself check out the commit.
Then you can just say type=dir, path=".", or something like that.

[sudden6]

@alexlarsson Thank you very much, I've implemented your solution here and it seems to work like a charm. I think this can be closed now.

[bossjones]

That solution works for me as well, thanks for the tip! @alexlarsson ( And thanks for the further validation @sudden6 ! )

[sudden6]

I noticed a problem with my setup. Instead of taking the directory state as is, it checks out the version from the manifest, which is not what I wanted. I need flatpak to use the state of the directory to be able to test my changes.

[alexlarsson]

That seems weird. Sources of type "dir" just does a cp -r of the directory into the build-dir. There is nothing git specific in that code.

[sudden6]

It seems that maybe my original intent was not clear. I wanted to override the source from an existing manifest. @TingPing told me in IRC that this is not possible. I now resorted to patching the relevant part in the CI build see https://github.com/qTox/qTox/pull/5123/files for the exact implementation.