Skip to content

Commit

Permalink
Introduce project folder argument (#68)
Browse files Browse the repository at this point in the history 
Key Updates

    Project Folder and Paths
        Introduced a -p flag for specifying the project folder, and -d for data.
        Updated templates to use relative paths for outputs and scripts.
        Refactored path handling using a new ProjectPaths class for better consistency.

    Template and Workflow Improvements
        Removed the output folder flag for templates.
        Improved prior dataset handling, including renaming and pathing fixes.

    Documentation and Examples
        Added detailed examples and extra info to the README.
        Updated examples to reflect the new -p and -d flag and path handling.

    Workflow Enhancements
        Added steps to run scitree before and after jobs execution.
        Fixed issues with running jobs in workflows.

    General Refactoring
        Cleaned up file handlers and folder names for clarity.
        Standardized platform checks and snake_case naming conventions.

    Miscellaneous
        Added .venv to .gitignore.
        Improved code quality with ruff.
  • Loading branch information
@jteijema
jteijema authored Jan 7, 2025
1 parent 1a92ba5 commit 6a4cec4
Show file tree
Hide file tree
Showing 13 changed files with 468 additions and 261 deletions.
51 changes: 25 additions & 26 deletions .github/workflows/ci-workflow.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,33 +27,20 @@ jobs:
- name: Lint python with ruff
run: |
ruff check .
- name: Create directories using Python
run: python -c "import os; [os.makedirs(path, exist_ok=True) for path in ['./tmp/basic/data-test', './tmp/arfi/data', './tmp/prior/data', './tmp/multimodel/data', './tmp/scripts', './tmp/synergy/data']]"
- name: set up environment
run: |
cp .github/workflows/test_data/labels.csv ./tmp/basic/data-test/labels.csv
cp .github/workflows/test_data/labels.csv ./tmp/arfi/data/labels.csv
cp .github/workflows/test_data/labels.csv ./tmp/prior/data/labels.csv
cp .github/workflows/test_data/labels.csv ./tmp/prior/data/prior_labels.csv
cp .github/workflows/test_data/labels.csv ./tmp/multimodel/data/labels.csv
- name: Render makita templates
run: |
cd tmp/basic
asreview makita template basic --classifier nb --feature_extractor tfidf --query_strategy max --n_runs 1 -s data-test -o output-test --init_seed 1 --model_seed 2 --skip_wordclouds --overwrite --instances_per_query 2 --stop_if min --balance_strategy double | tee output.txt
grep -q "ERROR" output.txt && exit 1 || true
cd ../arfi
asreview makita template arfi | tee output.txt
grep -q "ERROR" output.txt && exit 1 || true
cd ../prior
asreview makita template prior | tee output.txt
grep -q "ERROR" output.txt && exit 1 || true
cd ../multimodel
asreview makita template multimodel | tee output.txt
grep -q "ERROR" output.txt && exit 1 || true
asreview makita template basic -p basic -d .github/workflows/test_data/ --classifier nb --feature_extractor tfidf --query_strategy max --n_runs 1 --init_seed 1 --model_seed 2 --skip_wordclouds --overwrite --instances_per_query 2 --stop_if min --balance_strategy double | tee output_basic.txt
grep -q "ERROR" output_basic.txt && exit 1 || true
asreview makita template arfi -p arfi -d .github/workflows/test_data/ | tee output_arfi.txt
grep -q "ERROR" output_arfi.txt && exit 1 || true
asreview makita template prior -p prior -d .github/workflows/test_data/ | tee output_prior.txt
grep -q "ERROR" output_prior.txt && exit 1 || true
asreview makita template multimodel -p multimodel -d .github/workflows/test_data/ | tee output_multimodel.txt
grep -q "ERROR" output_multimodel.txt && exit 1 || true
- name: Render makita scripts
run: |
asreview makita add-script --all -o ./tmp/scripts | tee output.txt
grep -q "ERROR" output.txt && exit 1 || true
asreview makita add-script --all -o ./tmp/scripts | tee output_scripts.txt
grep -q "ERROR" output_scripts.txt && exit 1 || true
- name: Run SciTree
if: ${{ matrix.os != 'windows-latest' }}
run: |
Expand All @@ -62,8 +49,20 @@ jobs:
- name: Execute basic template jobs file
if: ${{ matrix.os != 'windows-latest' }}
run: |
cd tmp/synergy
synergy_dataset get -d van_de_Schoot_2018 -o ./data -l
asreview makita template basic --instances_per_query 100 --skip_wordclouds --overwrite --n_runs 2
mkdir basic_run
cd basic_run
asreview makita template basic -d ../.github/workflows/test_data/ --instances_per_query 100 --skip_wordclouds --overwrite --n_runs 1
scitree
sh jobs.sh
scitree
- name: Upload output files
if: always()
uses: actions/upload-artifact@v3
with:
name: output-files
path: |
output_basic.txt
output_arfi.txt
output_prior.txt
output_multimodel.txt
output_scripts.txt
4 changes: 4 additions & 0 deletions .github/workflows/test_data/labels.csv
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
Unnamed: 0,abstract,alternate_title2,alternate_title3,file_attachments2,authors,id,issn,keywords,note,number,place_published,title,publication_year,publisher,secondary_authors,start_page,type_of_reference,label_included,url,volume
0,"BACKGROUND: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example.com,"['Marx, Karl', 'Lindgren, Astrid']",12345,1932-6208,"['Pippi', 'Nordwind', 'Piraten']",1008150341,3,United States,Title of reference,2014//,Fun Factory,"['Glattauer, Daniel']",e0815,JOUR,1,http://example_url.com,9
0,"BACKGROUND: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example.com,"['Marx, Karl', 'Lindgren, Astrid']",12345,1932-6208,"['Pippi', 'Nordwind', 'Piraten']",1008150341,3,United States,Title of reference,2014//,Fun Factory,"['Glattauer, Daniel']",e0815,JOUR,1,http://example_url.com,9
0,"BACKGROUND: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example.com,"['Marx, Karl', 'Lindgren, Astrid']",12345,1932-6208,"['Pippi', 'Nordwind', 'Piraten']",1008150341,3,United States,Title of reference,2014//,Fun Factory,"['Glattauer, Daniel']",e0815,JOUR,1,http://example_url.com,9
0,"BACKGROUND: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example.com,"['Marx, Karl', 'Lindgren, Astrid']",12345,1932-6208,"['Pippi', 'Nordwind', 'Piraten']",1008150341,3,United States,Title of reference,2014//,Fun Factory,"['Glattauer, Daniel']",e0815,JOUR,1,http://example_url.com,9
0,"BACKGROUND: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example.com,"['Marx, Karl', 'Lindgren, Astrid']",12345,1932-6208,"['Pippi', 'Nordwind', 'Piraten']",1008150341,3,United States,Title of reference,2014//,Fun Factory,"['Glattauer, Daniel']",e0815,JOUR,1,http://example_url.com,9
1,"BACKGROUND: Lorem dammed ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example2.com,"['Marxus, Karlus', 'Lindgren, Astrid']",12345,1732-4208,"['Pippi Langstrumpf', 'Nordwind', 'Piraten']",1228150341,3,Germany,The title of the reference,2006//,Dark Factory,"['Glattauer, Daniel']",e0815341,JOUR,0,http://example_url.com,6
1,"BACKGROUND: Lorem dammed ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example2.com,"['Marxus, Karlus', 'Lindgren, Astrid']",12345,1732-4208,"['Pippi Langstrumpf', 'Nordwind', 'Piraten']",1228150341,3,Germany,The title of the reference,2006//,Dark Factory,"['Glattauer, Daniel']",e0815341,JOUR,0,http://example_url.com,6
1,"BACKGROUND: Lorem dammed ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. RESULTS: Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. CONCLUSIONS: Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium.",lorem,Lorem,http://example2.com,"['Marxus, Karlus', 'Lindgren, Astrid']",12345,1732-4208,"['Pippi Langstrumpf', 'Nordwind', 'Piraten']",1228150341,3,Germany,The title of the reference,2006//,Dark Factory,"['Glattauer, Daniel']",e0815341,JOUR,0,http://example_url.com,6
Expand Down
Loading

0 comments on commit 6a4cec4

Please sign in to comment.