Problems building with local source (neither git_url nor path practical)

[mara004]

What happened?

Local source is not practically usable with conda, due to the following limitations of existing concepts:

• path: ../.. copies all files, not just those in revision control. This is problematic if there are big generated dirs that should be excluded, then copying the source would take eternally. (Deleting all unneeded dirs prior to install is not applicable.)
• git_url: ../.. does not include uncommitted changes (see the docs).
  This is unsuitable/confusing for development, where you want to modify the source and rerun to test changes without committing.
  Further, it is a problem if we want to include generated files for installation that are not under revision control.

We'd need a way to honor gitignore rules, while including uncommitted changes and having git describe info available.
Even better would be a way to use a setuptools MANIFEST.in file defining the files to copy into the build dir.

I can't see how to achieve this with conda-build ATM (if it were possible somehow, then the docs would be very obscure).

Conda Details

conda info

active environment : test_bundling
    active env location : /home/me/miniconda3/envs/test_bundling
            shell level : 3
       user config file : /home/me/.condarc
 populated config files : /home/me/miniconda3/.condarc
                          /home/me/.condarc
          conda version : 23.9.0
    conda-build version : 3.27.0
         python version : 3.11.5.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.36=0
                          __linux=6.5.7=0
                          __unix=0=0
       base environment : /home/me/miniconda3  (writable)
      conda av data dir : /home/me/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/bblanchon/linux-64
                          https://conda.anaconda.org/bblanchon/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/me/miniconda3/pkgs
                          /home/me/.conda/pkgs
       envs directories : /home/me/miniconda3/envs
                          /home/me/.conda/envs
               platform : linux-64
             user-agent : conda/23.9.0 requests/2.31.0 CPython/3.11.5 Linux/6.5.7-100.fc37.x86_64 fedora/37 glibc/2.36
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

conda config

==> /home/me/miniconda3/.condarc <==
channels:
  - bblanchon
  - defaults

==> /home/me/.condarc <==
auto_activate_base: False

[mara004]

I think I found a workaround with automatic temp commit on before conda build, but undoubtedly it's not nice.

[kenodegard]

@mara004 Unfortunately this is unlikely to change, there would be too much guesswork happening on conda-build's part (would it only include staged changes? would it include unstaged changes? what about untracked changes? would conda-build need to implement all of these flavors of cloning a git repo?) and each alternative would be surprising/unexpected to somebody else.

[mara004]

Hmm, thinking about this some more, I acknowledge this lies outside the regular use case, for all dynamic files should normally be generated within the conda-build call.¹
Further, once the initial recipe writing is done, conda packages would usually be built only in release workflows, where uncommitted changes are not a question. Local installation (let alone with uncommitted changes) seems to be fairly rare in practice.

I still find that including uncommitted changes would be better with development, and that conda is inflexible regarding file inclusion, but it is logical given the focus on feedstock-based instead of project-inline packaging.
What helped me understand this is to think of conda as an OS repo being more similar to Linux distro packaging than to PyPA packaging.

Footnotes

1. In our case that's different because of a dependency that would be difficult to maintain as conda package, but nevermind. I realize we seem to be doing something quite uncommon in our project... ↩