Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share basilisk's cache between the BBS and the SPB on the Linux builders #400

Open
hpages opened this issue Feb 8, 2024 · 8 comments
Open

Comments

@hpages
Copy link
Contributor

hpages commented Feb 8, 2024

So I'm trying to find a satisfying solution for sharing basilisk's cache between the biocbuild (BBS) and pkgbuild (SPB) users on nebbiolo1.

For this to work, basilisk's cache and any subfolder would need to be writable by both users. A good place for such shared cache is /var/cache/basilisk (/var/cache is kind of the standard system-level cache location on Linux). This can be controlled with the BASILISK_EXTERNAL_DIR environment variable.

Now it's easy to manually set the permissions on this folder so that both biocbuild and pkgbuild can write to it, this is a one time thing to do. But we also need to make sure that any subfolder that gets automatically created after that will be writable by both users. One way to achieve this maybe is by having biocbuild and pkgbuild belong to the same group (right now they don't), and by setting the umask value appropriately. I'm going to explore that approach a little.

@hpages
Copy link
Contributor Author

hpages commented Feb 8, 2024

This is the setup I came up with:

  1. Change the primary group of the pkgbuild user to make it the same as the primary group of the biocbuild user.

    Check their primary groups (gid) with id biocbuild and id pkgbuild.

    Assuming that biocbuild's primary group is biocbuild, make this the primary group of the pkgbuild user with:

    sudo usermod -g biocbuild pkgbuild
    

    Then check their primary groups again.

    An additional check is to check their effective group ids with id -g biocbuild and id -g pkgbuild. They should be the same.

  2. Create basilisk's cache folder in /var/cache/:

    cd /var/cache
    sudo mkdir basilisk
    sudo chown biocbuild:biocbuild basilisk
    sudo chmod 775 basilisk
    
    
  3. Basic testing. Log as biocbuild and do:

    cd /var/cache/basilisk
    mkdir titi
    

    The folder should be writable by any member of the group. Check this with:

    ls -ld titi
    

    This should display something like:

    drwxrwxr-x 2 biocbuild biocbuild 4096 Feb  8 01:04 titi
    

    In particular the flags must be drwxrwxr-x. If not, then set umask to 002 in ~/.profile, logout and login again for the change to take effect, and try to create another folder in /var/cache/basilisk/.

    Then log as pkgbuild and do:

    cd /var/cache/basilisk
    mkdir toto
    

    The folder should be writable by any member of the group:

    ls -ld toto
    

    This should display something like:

    drwxrwxr-x 2 pkgbuild biocbuild 4096 Feb  8 01:03 toto
    

    If not, then set umask to 002 in ~/.profile, logout and login again for the change to take effect, and try to create another folder in /var/cache/basilisk/.

    Finally clean /var/cache/basilisk/ with:

    cd /var/cache/basilisk
    rm -rf *
    
  4. Set BASILISK_EXTERNAL_DIR to /var/cache/basilisk in ~/.profile for biocbuild and pkgbuild by putting the following line at the end of each file:

    export BASILISK_EXTERNAL_DIR="/var/cache/basilisk"
    

    Do NOT put this in /etc/profile!

    IMPORTANT: This setting won't be effective on the SPB until the SPB server gets restarted (even if, in an interactive R session, basilisk.utils::getExternalDir() points to the new location).

  5. Next build run will repopulate /var/cache/basilisk/.

Let's see how things go in the next couple of days. If everything goes fine, then we'll be able to remove ~biocbuild/.cache/R/basilisk and ~pkgbuild/.cache/R/basilisk.

Checking again a couple of days later that ~biocbuild/.cache/R/basilisk and ~pkgbuild/.cache/R/basilisk have not been recreated will be an extra confirmation that the BBS and the SPB are now both using the new cache location. If we are curious, we could even look at the ownership of the stuff under /var/cache/basilisk, it should tell us who created what.

@jwokaty
Copy link
Collaborator

jwokaty commented Feb 8, 2024

I'm just noting here that we'll want to try setting BASILISK_EXTERNAL_DIR on merida1 next, where both biocbuild and pkgbuild are members of group staff.

@martin-g
Copy link
Contributor

martin-g commented Feb 8, 2024 via email

@hpages
Copy link
Contributor Author

hpages commented Feb 8, 2024

It is important that the group is the primary one for both users! Otherwise newly created subfolders won’t be shareable.

@martin-g Absolutely!

I'm just noting here that we'll want to try setting BASILISK_EXTERNAL_DIR on merida1 next

@jwokaty That would be the next step.

I'm running into some gotchas with the new setup on nebbiolo1. The -00LOCK files and other files (e.g. basilisk/1.15.4/0/pkgs/cache/47929eba.json) under /var/cache/basilisk are created with restrictive permissions that do not let the pkgbuild user touch them. So despite the umask setting, it seems that when these files get created, their perms get explicitly restricted.

I'll investigate more.

@hpages
Copy link
Contributor Author

hpages commented Feb 8, 2024

Temporary hack (from a sudo account):

cd /var/cache/basilisk
sudo chmod --quiet -Rc g+rw .
sudo find . -type d -exec chmod --quiet -c go+rx {} \;

With this I can finally run R CMD build and R CMD check on sketchR as the pkgbuild user.

Of course far from being satisfying because these commands would need to be re-run every time more stuff gets added to the cache.

@hpages
Copy link
Contributor Author

hpages commented Feb 8, 2024

Similar setup on merida1 (but using .bash_profile instead of ~/.profile, and replacing --quiet with -f in chmod).

With this new setup sketchR passes R CMD build and R CMD check in 9m32s and 4m20s, respectively. The daily builds were not running though so it might take a little bit longer when they are.

@jwokaty
Copy link
Collaborator

jwokaty commented Feb 8, 2024

I appreciate pointer; I did just get lucky. On Macs, you don't automatically get a group that is the same as your username, like Linux. If https://unix.stackexchange.com/questions/410367/how-to-get-the-primary-group-of-a-user#answer-410370 is correct, the primary group is staff.

merida1:~ biocbuild$ id -gn biocbuild
staff
merida1:~ biocbuild$ id -gn pkgbuild
staff

@hpages
Copy link
Contributor Author

hpages commented Feb 8, 2024

Yes, we are lucky that on merida1, unlike on nebbiolo1, biocbuild and pkgbuild already had the same primary group. That was one less thing to change 🍀 I don't think it would be too hard to change that if we had to though...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants