v1.4 canfar run

[martinkilbinger]

Summary

Major modifications for processing run on canfar of ShapePipe v1.4 catalogues.

Auxilliary scripts:

• Added bash scripts for job handling on new canfar science portal (curl based)
• New python script to create final catalogue as hdf5 file.

Main pipeline:

• New command line option-e ID to select exclusive image ID
• Created new docker image, to replace conda installation

Modules:

• Batch saving for ngmix; output catalogue can be written to disk in batches. In case the (typically very time-consuming)
  processing is interrupted, reprocessing does not have to start from the beginning. Note that in this case the random numbers
  do not match the previous object ID any more.
• Added check_existing_dir option to modules mask, ngmix. Processing of the corresponding module is skipped if
  expected output file exists in this directory.
• Added file tests to many modules, improved error messages
• make catalogue runner: spread model input now optional
• merged star cat output moved from HDU Merge pipeline template to master branch #2 to HDU test ci setup #1; added HDU config option for input

Reviewer Checklist

Reviewers should tick the following boxes before approving and merging the PR.

• The PR targets the develop branch
• The PR is assigned to the developer
• The PR has appropriate labels
• The PR is included in appropriate projects and/or milestones
• The PR includes a clear description of the proposed changes
• If the PR addresses an open issue the description includes "closes #"
• The code and documentation style match the current standards
• Documentation has been added/updated consistently with the code
• All CI tests are passing
• API docs have been built and checked at least once (if relevant)
• All changed files have been checked and comments provided to the developer
• All of the reviewer's comments have been satisfactorily addressed by the developer

[martinkilbinger]

Hi @martinkilbinger this is a really impressive amount of work, congrats! I took a look at your comments and the Docker-related files that I am already familiar with, and they all look good. Beyond this, is there anything in particular it would be helpful for me to take a close look at? And what needs to be done before this can be merged?

Don't think anything in particular needs to be checked. Maybe a bit more care should be taken for changes in the underlying pipeline compared to updates in the modules.

[cailmdaley]

I noticed that the black changes are still showing up in the diff, despite the fact that black was run on the main branch as well. I asked my assistant about this and they said:

Why This Happens:
1. Commit History: When you run black on the main branch, the changes are added as new commits, and these commits are not part of your fork’s branch. Therefore, the diff in the PR is calculated relative to the state of the main branch before the black reformatting was applied.
2. Git Diffing in PRs: The PR diff compares your fork’s branch to the base branch (main branch) as it existed at the time the PR was opened. If the main branch’s changes (e.g., from running black) are not integrated into your fork, the diff will still show your reformatting as new changes.

Suggestions to Resolve This:

1. Rebase Your Fork Onto the Updated Main Branch
   • Pull the latest changes from the main branch into your fork, including the black reformatting.
   • Rebase your fork onto the updated main branch. This aligns your fork’s branch with the current state of the main branch and removes any redundant diffs caused by black.

# Fetch the latest changes from the main repository
git fetch upstream
 
# Checkout your branch
git checkout your-feature-branch

# Rebase onto the updated main branch
git rebase upstream/main

Would you be interested in trying this on your fork? It'll be a lot easier to look at the real changes if it works!

[martinkilbinger]

I noticed that the black changes are still showing up in the diff, despite the fact that black was run on the main branch as well. I asked my assistant about this and they said:

Why This Happens:

1. Commit History: When you run black on the main branch, the changes are added as new commits, and these commits are not part of your fork’s branch. Therefore, the diff in the PR is calculated relative to the state of the main branch before the black reformatting was applied.

2. Git Diffing in PRs: The PR diff compares your fork’s branch to the base branch (main branch) as it existed at the time the PR was opened. If the main branch’s changes (e.g., from running black) are not integrated into your fork, the diff will still show your reformatting as new changes.
   Suggestions to Resolve This:

3. Rebase Your Fork Onto the Updated Main Branch
   • Pull the latest changes from the main branch into your fork, including the black reformatting.
   • Rebase your fork onto the updated main branch. This aligns your fork’s branch with the current state of the main branch and removes any redundant diffs caused by black.

# Fetch the latest changes from the main repository
git fetch upstream
 
# Checkout your branch
git checkout your-feature-branch

# Rebase onto the updated main branch
git rebase upstream/main

Would you be interested in trying this on your fork? It'll be a lot easier to look at the real changes if it works!

I started doing the rebase, but got a ton of conflicting files. I have to fix those by hand; it seems that the rebasing goes through every commit over the last two years or so. After half an hour of manually fixing (recurring) conflicts, I think I am close to giving up.

[martinkilbinger]

@cailmdaley I went through the pipeline and utility scripts. Nothing worrying seemed to be changed, most if not all of the diff was due to black. From my side we are ready to merge (and hope for the best!)

[cailmdaley]

Great, please go ahead and merge when you're ready!

[martinkilbinger]

Great, please go ahead and merge when you're ready!

OK let's do it!