how I set up an analysis repository building on a dataset (IN PROGRESS) #251

alecristia · 2021-07-07T10:58:40Z

alecristia
Jul 7, 2021
Maintainer

In many of our projects, we just want recording-level metrics -- eg how many vocalizations (in total or per hour) the child produced. In this S&T, I explain how I set up one such analysis repository. I'm drawing code from a private repo (LAAC-LSCP/el1000-metrics) -- so credit for that code goes to @lucasgautheron ; my own analysis will be public, but the dataset on which it is based is not (yet) public.

1. Set up a YODA repo

source ~/ChildProjectVenv/bin/activate
datalad create -c yoda solis_metrics
cd solis_metrics

In order to share with others & get feedback, I followed these instructions to publish to GIN, except that:

in the first step, I created the repo inside LAAC-LSCP rather than in my user (so that people in my team can edit)
I unchecked the box at the bottom so the repo is completely clean

I got the following messages, which are normal:

[INFO   ] Could not enable annex remote gin. This is expected if gin is a pure Git remote, or happens if it is not accessible. 
[WARNING] Could not detect whether gin carries an annex. If gin is a pure Git remote, this is expected.

Now that means that when I'm ready to share my progress, I do:

datalad save . -m "first commit" # don't forget the dot!
datalad push --to gin  # don't forget the "to" bit

Another time I wanted to share this with my colleagues at ACLEW, so I followed instead these instructions to publish to github. I had to generate a token in github here (which allows me to access the github API).

datalad create-sibling-github -s origin --access-protocol ssh aclew10kq1 --github-organization aclew --private --github-login alecristia

I got these messages:

You need to authenticate with 'alecristia@github' credentials. https://github.com/settings/tokens provides information on how to gain access
token:
[INFO ] Successfully obtained information about organization aclew using token ghp...
.: origin(-) [[email protected]:aclew/aclew10kq1.git (git)]
'[email protected]:aclew/aclew10kq1.git' configured as sibling 'origin' for Dataset(/Users/acristia/Documents/gitrepos/aclew10kq1)

Note that where it says token: the system waited until I copy-pasted my token.

2. Import analyses etc from a similar analysis repo

I first made a local copy of the similar analysis repo somewhere else (i.e., not in the repo I just created):

datalad install [email protected]:/EL1000/metrics.git
cd metrics

After poking around, I decided to:

copy over without making any changes: .gitattributes
adapt README: and Makefile

cp ../metrics/.gitattributes .
cp ../metrics/Makefile .
cd ../metrics
datalad get README.md  #note!
cd ../solis_metrics
cp ../metrics/README.md .

And then open in a text editor and make changes to makefile & readme.

3. Nest the dataset

I created folders for input and output:

mkdir output
mkdir input

I'll put it inside input, per YODA recommendations.

cd input
datalad install -r [email protected]:/LAAC-LSCP/solomon
cd solomon
datalad get annotations/*/converted/

4. Run analyses

As described in the README inside the repo you just created, you launch this from the top level of your analysis repo, with:

child-project metrics input/solomon/ solomon.csv aclew

I got a series of lines like:

A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
errors=errors,

Those are just to be ignored...

5. Run reliability/accuracy analyses TBD

6. Embedding these analyses in a paper

For this paper, we are working in a dropbox folder and it is not convenient to migrate to git. So I decide to
embed a clone of these analyses in my dropbox folder. This simply means installing the ANALYSIS datalad dataset;
notice I ALSO use the -r switch because I'm going to use directly the original dataset, specifically to be able
to extract child-level and recording-level metadata.

datalad install -r [email protected]:/LAAC-LSCP/solis_metrics.git
cd solis_metrics
datalad get solomon.csv

And done!

Now when I have an update in solis_metrics that I want to take into account, I can cd into it, and do git pull . But I can also stick to the current copy, so that my analyses are reproducible.

Troubleshooting

Permission denied

It may be that analyses get a permission error when I tried to write a file:

PermissionError: [Errno 13] Permission denied: 'some-filename'

To fix that I did: datalad unlock output

could not get content

When trying to get files I got:

[WARNING] could not get some content in /Users/acristia/Documents/gitrepos/solis_metrics/input/solomon/annotations/vtc/converted

As documented in #252, this was due to mistakenly installing using https, rather than ssh

the call to child-project didn't work

I doubt others will have this problem, but if you do see:

usage: child-project [-h]
{validate,import-annotations,merge-annotations,intersect-annotations,remove-annotations,rename-annotations,import-data,overview,compute-durations,process,sampler,zooniverse,eaf-builder,anonymize}
...
child-project: error: invalid choice: 'metrics' (choose from 'validate', 'import-annotations', 'merge-annotations', 'intersect-annotations', 'remove-annotations', 'rename-annotations', 'import-data', 'overview', 'compute-durations', 'process', 'sampler', 'zooniverse', 'eaf-builder', 'anonymize')

after your call child-project metrics input/ etc, that means child-project is outdated and doesn't have the metrics option. Fixed with:

pip3 install git+https://github.com/LAAC-LSCP/ChildProject.git --force-reinstall

alecristia · 2021-07-28T08:11:55Z

alecristia
Jul 28, 2021
Maintainer Author

instructions followed & one typo fixed.
step 4 ok but see https://gin.g-node.org/LAAC-LSCP/vanuatu/issues/5 (closed)
Section 5 is tbd -- to be tested in the future!

1 reply

alecristia Aug 17, 2021
Maintainer Author

section 6 tested on oberon. The first level was installed ok but I got a permission error for the embedded dataset:

(ChildProjectVenv) [acristia@oberon ~]$ datalad install -r [email protected]:/LAAC-LSCP/vanuatu_metrics.git
[INFO ] Scanning for unlocked files (this may take some time)
install(ok): /scratch1/home/acristia/vanuatu_metrics (dataset)
[INFO ] Installing Dataset(/scratch1/home/acristia/vanuatu_metrics) to get /scratch1/home/acristia/vanuatu_metrics recursively
[ERROR ] Failed to clone from any candidate source URL. Encountered errors per each url were:
| - [email protected]:/LAAC-LSCP/vanuatu_metrics.git/input/vanuatu
CommandError: 'git -c diff.ignoreSubmodules=none clone --progress [email protected]:/LAAC-LSCP/vanuatu_metrics.git/input/vanuatu /scratch1/home/acristia/vanuatu_metrics/input/vanuatu' failed with exitcode 128 [err: 'Cloning into '/scratch1/home/acristia/vanuatu_metrics/input/vanuatu'...
GIN: Repository does not exist or you do not have access
CommandError: 'ssh -o ControlPath=/scratch1/home/acristia/.cache/datalad/sockets/dcd7d980 -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '"'"'/LAAC-LSCP/vanuatu_metrics.git/input/vanuatu'"'"''' failed with exitcode 1
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.'] [install(/scratch1/home/acristia/vanuatu_metrics/input/vanuatu)]
install(error): /scratch1/home/acristia/vanuatu_metrics/input/vanuatu (dataset) [Failed to clone from any candidate source URL. Encountered errors per each url were:

[email protected]:/LAAC-LSCP/vanuatu_metrics.git/input/vanuatu
CommandError: 'git -c diff.ignoreSubmodules=none clone --progress [email protected]:/LAAC-LSCP/vanuatu_metrics.git/input/vanuatu /scratch1/home/acristia/vanuatu_metrics/input/vanuatu' failed with exitcode 128 [err: 'Cloning into '/scratch1/home/acristia/vanuatu_metrics/input/vanuatu'...
GIN: Repository does not exist or you do not have access
CommandError: 'ssh -o ControlPath=/scratch1/home/acristia/.cache/datalad/sockets/dcd7d980 -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '"'"'/LAAC-LSCP/vanuatu_metrics.git/input/vanuatu'"'"''' failed with exitcode 1
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.']]
action summary:
install (error: 1, ok: 1)

I just removed the empty embedded repository, and used datalad install xxx to install it by hand

lucasgautheron · 2021-08-17T07:31:43Z

lucasgautheron
Aug 17, 2021
Maintainer

The reason is that the .gitmodules file that contain information about the subdatasets in vanuatu_metrics is somehow pointing to solomon rather than vanuatu.
I guess it was copy-pasted.

I suggest to git rm -rf .gitmodules, git rm -rf input/vanuatu, and then re-install it (datalad install -d . [email protected]:/LAAC-LSCP/vanuatu.git input/vanuatu)

4 replies

alecristia Aug 17, 2021
Maintainer Author

ah yes, that's exactly it!

That's my step 2. So I shouldn't copy over gitmodules, right?

alecristia Aug 17, 2021
Maintainer Author

git rm -rf input/vanuatu yields

fatal: could not lookup name for submodule 'input/vanuatu'

rm -rf input/vanuatu/ yields

rm: cannot remove ‘input/vanuatu/.git/annex/objects/wz/xx/MD5E-s526236--05c25b470e916d3951b058fe7a6951b4.csv/MD5E-s526236--05c25b470e916d3951b058fe7a6951b4.csv’: Permission denied

datalad unlock input/vanuatu/.git yields

[ERROR ] 'NoneType' object has no attribute 'pathobj' [status.py:_yield_status:111] (AttributeError)

alecristia Aug 17, 2021
Maintainer Author

so I went ahead even if the removal had been unsuccessful, and did:
datalad install -d . [email protected]:/LAAC-LSCP/vanuatu.git input/vanuatu

It looks like it works, but input/vanuatu is empty:

[INFO ] Scanning for unlocked files (this may take some time)
install(ok): vanuatu (dataset)
add(ok): vanuatu (file)
add(ok): .gitmodules (file)
save(ok): . (dataset)
action summary:
add (ok: 2)
get (notneeded: 1)
install (ok: 1)
save (ok: 1)

ls input/

vanuatu

ls input/vanuatu/

(nothing is printed out)

ls

CHANGELOG.md code input Makefile README.md vanuatu vanuatu.csv

less .gitmodules

[submodule "vanuatu"]
path = vanuatu
url = [email protected]:/LAAC-LSCP/vanuatu.git
datalad-id = e6789925-03aa-490a-b832-db7dd64da7b5
datalad-url = [email protected]:/LAAC-LSCP/vanuatu.git

lucasgautheron Aug 17, 2021
Maintainer

My bad, it should have been datalad install -d . --source '[email protected]:/LAAC-LSCP/vanuatu.git' input/vanuatu... It's even more messed up now :)

I fixed it myself, so you can just re-install vanuatu_metrics to have a clean state.

For the record, here is how I did it:

Remove the submodule altogether according to these instructions: https://forum.freecodecamp.org/t/how-to-remove-a-submodule-in-git/13228
datalad install -d . --source '[email protected]:/LAAC-LSCP/vanuatu.git' input/vanuatu
datalad push

alecristia · 2023-06-16T08:07:00Z

alecristia
Jun 16, 2023
Maintainer Author

If needing your nested dataset to refer to a specific branch:

cd input/tsimane2017
git branch -v -a # see which remote branches are available
git switch -c anno2023 remotes/origin/anno2023 # to create a local version of this remote branch

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how I set up an analysis repository building on a dataset (IN PROGRESS) #251

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

how I set up an analysis repository building on a dataset (IN PROGRESS) #251

alecristia Jul 7, 2021 Maintainer

1. Set up a YODA repo

2. Import analyses etc from a similar analysis repo

3. Nest the dataset

4. Run analyses

5. Run reliability/accuracy analyses TBD

6. Embedding these analyses in a paper

Troubleshooting

Permission denied

could not get content

the call to child-project didn't work

Replies: 3 comments · 5 replies

alecristia Jul 28, 2021 Maintainer Author

alecristia Aug 17, 2021 Maintainer Author

lucasgautheron Aug 17, 2021 Maintainer

alecristia Aug 17, 2021 Maintainer Author

alecristia Aug 17, 2021 Maintainer Author

alecristia Aug 17, 2021 Maintainer Author

lucasgautheron Aug 17, 2021 Maintainer

alecristia Jun 16, 2023 Maintainer Author

alecristia
Jul 7, 2021
Maintainer

Replies: 3 comments 5 replies

alecristia
Jul 28, 2021
Maintainer Author

alecristia Aug 17, 2021
Maintainer Author

lucasgautheron
Aug 17, 2021
Maintainer

alecristia Aug 17, 2021
Maintainer Author

alecristia Aug 17, 2021
Maintainer Author

alecristia Aug 17, 2021
Maintainer Author

lucasgautheron Aug 17, 2021
Maintainer

alecristia
Jun 16, 2023
Maintainer Author