Asking for guidance for making EUPS relocatable

[airnandez]

Dear EUPS experts,

I'm working towards helping making the LSST stack relocatable. Since EUPS is an essential ingredient of the LSST stack, we need to make EUPS itself relocatable. Before I spend more time coding and testing I would like to get some feedback on how the relocation could be implemented. Here are my understanding and the foreseen ways of making progress.

Among other things, the setup scripts $EUPS_DIR/bin/setups.*sh initialize 2 environment variables: EUPS_DIR and EUPS_PATH. The value set to EUPS_DIR is the directory where EUPS is installed (option --prefix passed to configure). EUPS_PATH is set to the value of the option --with-eups passed to configure. So, currently, the value of those two variables are set at installation time, which make relocating EUPS difficult.

The purpose of the modification is to set those two environment variables with sensible values at execution time. The value EUPS_DIR can be set according to the directory where the scripts $EUPS_DIR/bin/setups.*sh are located and this can be obtained at execution time.

The value of EUPS_PATH cannot be automatically set to a sensible value at execution time. My idea is to modify each one of the shell scripts in $EUPS_DIR/bin/setups.*sh so that they accept an optional command line argument which value would be appended to the current value of the environment variable EUPS_PATH (if set), overwriting the default value. The default value for EUPS_PATH would be set to the value of --with-eups passed to configure at installation time. In this way, loadLSST.shwould setup EUPS by calling:

source $EUPS_DIR/bin/setups.sh $LSST_HOME

or, alternatively

EUPS_PATH=$LSST_HOME
source $EUPS_DIR/bin/setups.sh

These 2 proposed modifications would both keep backwards compatibility.

Since I'm not an expert in EUPS nor familiar with the various contexts it may be used, I very likely don't see scenarios where the proposed way of working could fail or be incompatible with the way EUPS is currently used. So your feedback on this would be very appreciated: if these proposed modifications are considered useful, I would proceed doing the coding and eventually sending a pull request for you to review.

[PaulPrice]

@CraigLoomis, EUPS guru, writes:

Why is EUPS_DIR set absolutely in setups.sh? That has always felt wrong to me (with relocation in mind).

Oh, Is LSST_ROOT even necessary, or should it come from a "site" or "project" or "lsst" eups product? [Which of eups & project is the chicken, which egg?]

And If we are asking about such matters, can we settle on a definitive scheme to place and find the active "eups" given one element of EUPS_PATH? For most users there is only one item in $EUPS_PATH, and the "source $path/setups.?sh" for that really should be both scrutable and not overly specific.
This may all be fixed in current LSST-land, but I have seen at least the following:

a. $ROOT/Linux64/eups/1.4.1/bin/setups.sh in .bash(rc|_profile). Horrible. The worst.
b. $ROOT/eups/ as a real directory, holding some unnamed eups version. I dislike this almost as much as a.
c. $ROOT/bin/{eups, setups.*} as real files. Dislike, for same reason.
d. $ROOT/eups symlink to eups list -c eups. I prefer this, especially if you also have a $ROOT/bin symlink.
e. $ROOT/eups/current, or $ROOT/current symlinks. I don't think these add anything.

I.e., I think the eups bootstrap&installation scripts should install eups itself as a proper eups product, then create some nice symlinks to publish to users.

[mjuric]

@airnandez : These seem reasonable to me.

The only thing I'd propose is not to do the command line parameter, but just use the environmental variable for EUPS_PATH. We already have precedent for deferring to the environment, though this may theoretically break someone's code if they rely on setups.sh overwriting whatever they had in EUPS_PATH.

[RobertLuptonTheGood]

The purpose of the modification is to set those two environment variables with sensible values at execution time. The value EUPS_DIR can be set according to the directory where the scripts $EUPS_DIR/bin/setups.*sh are located and this can be obtained at execution time.

I don't know how to get the value of the directory where a sourced script is found (bash uses $BASH_SOURCE, but that's an extension). What are you proposing?

I'm OK with changing all the setups scripts (or more precisely the script mksetup that writes them) to obey a pre-existing EUPS_DIR variable rather than setting it absolutely.

If you want to hack setting EUPS_DIR into the loadLSST.sh (and .csh and any others that people have written) that's OK with me; I never use it.

EUPS_PATH=$LSST_HOME
source $EUPS_DIR/bin/setups.sh

You need to do this in the opposite order, as setups.sh adds the configured path to your EUPS_PATH. That is:

source $EUPS_DIR/bin/setups.sh
EUPS_PATH=$LSST_HOME
(no need to export it).

As Mario notes, as corrected this is the correct answer for EUPS_PATH rather than adding a new command line option.

The solution that works out of the box, and how this is supposed to work, is:
source /some/eups/bin/setup.sh # bootstrap
setup -r /my/better/eups # or "setup eups", or "setup eups 1.5.2", or whatever
EUPS_PATH=...
This may not work for your use case.

                        R

[airnandez]

Thanks for your feedback. To get the path of the shell script within the script itself I use $BASH_SOURCE. This is indeed an extension implemented in BASH. My assumption is that in general, for interactive work, sh is actually bash nowadays in most platforms I work on.

Would this assumption hold for the contexts where EUPS is used? If so, we could go ahead and implement my proposed modifications. If not, EUPS' setups.sh could at least obey an existing value of EUPS_DIR and not overwrite it. The latter possibility would require modifying the LSST-specific scripts that bootstrap EUPS, but I think this is possible (I'm currently helping improving them).

By the way, I would also like to suggest that mksetup generates bootstrap scripts with explicit extensions, i.e. .sh, .bash, .zsh, .ksh, .dash. In my opinion, this makes clearer for the end-user which shell flavour those scripts are intended for. The contents of some of them may be identical or the files may be symbolic links, indeed, according to the shell compatibility guaranties.

[PaulPrice]

You cannot assume that sh is bash. On my Ubuntu machine:

price@neverland:~ $ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Feb 19  2014 /bin/sh -> dash*

See also https://wiki.ubuntu.com/DashAsBinSh

[RobertLuptonTheGood]

Thanks for your feedback. To get the path of the shell script within the script itself I use $BASH_SOURCE. This is indeed an extension implemented in BASH. My assumption is that in general, for interactive work, sh is actually bash nowadays in most platforms I work on.

No, it needs to work with a posix shell

Would this assumption hold for the contexts where EUPS is used? If so, we could go ahead and implement my proposed modifications. If not, EUPS'setups.sh could at least obey an existing value of EUPS_DIR and not overwrite it. The latter possibility would require modifying the LSST-specific scripts that bootstrap EUPS, but I think this is possible (I'm currently helping improving them).

I'm willing to accept a patch that does that.

By the way, I would also like to suggest that mksetup generates bootstrap scripts with explicit extensions, i.e. .sh, .bash, .zsh, .ksh, .dash. In my opinion, this makes clearer for the end-user which shell flavour those scripts are intended for. The contents of some of them may be identical or the files may be symbolic links, indeed, according to the shell compatibility guaranties.

It generates setups.sh (which should work for bash, sh, and ksh), setups.csh ([t]csh), and setups.zsh (zsh/dash, but I think I can make setups.sh work for them too when I fix the ksh problems).

I don't think we need distinguish beyond bourne-shell and csh-shell variants.

                        R

[airnandez]

I will work on a patch to mksetup so that that it generates bootstrap shell scripts (specifically setups.sh, setups.cshand setups.zsh) which obey to an pre-initialized EUPS_DIR. I understood that those shell scripts need to be POSIX-compliant so no BASH-specific features can be used.

I will send you a pull request then when my proposed modification is ready for reviewing.

[timj]

I know it's been two years, but did this get fixed all on its own?

[RobertLuptonTheGood]

Not that I know of