Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support __getitem__ with JobsCursor objects #1017

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

janbridley
Copy link

@janbridley janbridley commented Jan 28, 2025

Description

Implemented __getitem__ for JobsCursors, allow for indexing and slicing after calling project.find_jobs()

Motivation and Context

JobsCursors are iterable through the _JobsCursorIterator API, but slicing and indexing was not supported. This is atypical for Python types and requires users to unpack the cursor to index it.

This implementation returns a single Job if an integer index is requested, or a _JobsCursorIterator if a slice is requested. New tests have been added to cover the new functionality.

Checklist:

@janbridley janbridley added enhancement New feature or request python Pull requests that update Python code labels Jan 28, 2025
Copy link

codecov bot commented Jan 28, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.25%. Comparing base (82a9c69) to head (bd53150).
Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1017      +/-   ##
==========================================
+ Coverage   86.24%   86.25%   +0.01%     
==========================================
  Files          20       20              
  Lines        3503     3508       +5     
  Branches      662      663       +1     
==========================================
+ Hits         3021     3026       +5     
  Misses        331      331              
  Partials      151      151              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@janbridley
Copy link
Author

Note that docs are failing due to the ReadTheDocs configuration change - that will need to be addressed in a separate PR.

@janbridley janbridley changed the title Feature/jobscursor getitem Support __getitem__ with JobsCursor objects Jan 28, 2025
@janbridley janbridley marked this pull request as ready for review January 28, 2025 17:34
@janbridley janbridley requested review from a team as code owners January 28, 2025 17:34
@janbridley janbridley requested review from bdice and tommy-waltmann and removed request for a team January 28, 2025 17:34
@bdice
Copy link
Member

bdice commented Jan 28, 2025

This part of the code is usually benchmarked thoroughly before changes are accepted. @janbridley Would you be able to run the benchmark suite and share results? Here is a guide: https://github.com/glotzerlab/signac/blob/main/doc/support.rst#benchmarking

@janbridley
Copy link
Author

The asv suite does not seem to run either on my local machine (osx-arm) or a remote computer (Rocky Linux 8.10 x86). It could be a problem with micromamba, although I did try with a uv environment as well. If this is a known behavior please let me know - otherwise I can troubleshoot later this week.

· Creating environments
· Discovering benchmarks
·· Uninstalling from virtualenv-py3.12
·· Building 82a9c691 <main> for virtualenv-py3.12....
·· Installing 82a9c691 <main> into virtualenv-py3.12
·· Failed to build the project and import the benchmark suite.

@bdice
Copy link
Member

bdice commented Feb 4, 2025

@janbridley I was able to reproduce the failure you're seeing. I tried $ asv run main..feature/jobscursor-getitem -v but the verbose output didn't help. I think this is the same as airspeed-velocity/asv#1453.

@bdice
Copy link
Member

bdice commented Feb 4, 2025

@janbridley I found that using asv==0.6.1 works but newer versions do not work. I reported this upstream. airspeed-velocity/asv#1453 (comment)

@janbridley
Copy link
Author

ASV results. This always failed if a micromamba environment was active, but a clean asv==0.6.1 install with no mamba/conda/uv did work.

· Creating environments
· Discovering benchmarks
· Running 14 total benchmarks (2 commits * 1 environments * 7 benchmarks)
[ 0.00%] · For signac commit 79f9a157 <feature/jobscursor-getitem>:
[ 0.00%] ·· Benchmarking virtualenv-py3.12
[ 3.57%] ··· Running (benchmarks.ProjectBench.time_determine_len--)..
[10.71%] ··· Running (benchmarks.ProjectBench.time_iterate_load_sp--)..
[17.86%] ··· Running (benchmarks.ProjectRandomJobBench.time_search_lean_filter--)..
[25.00%] ··· Running (benchmarks.ProjectRandomJobBench.time_select_by_id--).
[28.57%] ··· benchmarks.ProjectBench.time_determine_len                      ok
[28.57%] ··· ====== ===================== =================== ================ =============== =============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0           302±7μs
              1000            10                   0                100               0         2.83±0.03ms
             ====== ===================== =================== ================ =============== =============

[32.14%] ··· benchmarks.ProjectBench.time_iterate                            ok
[32.14%] ··· ====== ===================== =================== ================ =============== ============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ------------
              100             10                   0                100               0         14.3±0.2ms
              1000            10                   0                100               0          140±3ms
             ====== ===================== =================== ================ =============== ============

[35.71%] ··· benchmarks.ProjectBench.time_iterate_load_sp                    ok
[35.71%] ··· ====== ===================== =================== ================ =============== ============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ------------
              100             10                   0                100               0          141±40ms
              1000            10                   0                100               0         1.39±0.01s
             ====== ===================== =================== ================ =============== ============

[39.29%] ··· ...hmarks.ProjectBench.time_iterate_single_pass                 ok
[39.29%] ··· ====== ===================== =================== ================ =============== =============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0         1.43±0.01ms
              1000            10                   0                100               0          14.5±0.3ms
             ====== ===================== =================== ================ =============== =============

[42.86%] ··· ...rojectRandomJobBench.time_search_lean_filter                 ok
[42.86%] ··· ====== ===================== =================== ================ =============== =============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0         1.71±0.05ms
              1000            10                   0                100               0           129±30ms
             ====== ===================== =================== ================ =============== =============

[46.43%] ··· ...rojectRandomJobBench.time_search_rich_filter                 ok
[46.43%] ··· ====== ===================== =================== ================ =============== ============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ------------
              100             10                   0                100               0         6.33±0.2ms
              1000            10                   0                100               0          161±20ms
             ====== ===================== =================== ================ =============== ============

[50.00%] ··· ...arks.ProjectRandomJobBench.time_select_by_id                 ok
[50.00%] ··· ====== ===================== =================== ================ =============== =============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0         7.37±0.03μs
              1000            10                   0                100               0         7.34±0.05μs
             ====== ===================== =================== ================ =============== =============

[50.00%] · For signac commit 2824cb13 <feature/jobscursor-getitem~1>:
[50.00%] ·· Building for virtualenv-py3.12
[50.00%] ·· Benchmarking virtualenv-py3.12
[53.57%] ··· Running (benchmarks.ProjectBench.time_determine_len--)..
[60.71%] ··· Running (benchmarks.ProjectBench.time_iterate_load_sp--)..
[67.86%] ··· Running (benchmarks.ProjectRandomJobBench.time_search_lean_filter--)..
[75.00%] ··· Running (benchmarks.ProjectRandomJobBench.time_select_by_id--).
[78.57%] ··· benchmarks.ProjectBench.time_determine_len                      ok
[78.57%] ··· ====== ===================== =================== ================ =============== =============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0           299±3μs
              1000            10                   0                100               0         2.85±0.02ms
             ====== ===================== =================== ================ =============== =============

[82.14%] ··· benchmarks.ProjectBench.time_iterate                            ok
[82.14%] ··· ====== ===================== =================== ================ =============== ============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ------------
              100             10                   0                100               0         14.3±0.1ms
              1000            10                   0                100               0          188±30ms
             ====== ===================== =================== ================ =============== ============

[85.71%] ··· benchmarks.ProjectBench.time_iterate_load_sp                    ok
[85.71%] ··· ====== ===================== =================== ================ =============== ============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ------------
              100             10                   0                100               0          141±5ms
              1000            10                   0                100               0         1.41±0.03s
             ====== ===================== =================== ================ =============== ============

[89.29%] ··· ...hmarks.ProjectBench.time_iterate_single_pass                 ok
[89.29%] ··· ====== ===================== =================== ================ =============== =============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0         1.45±0.01ms
              1000            10                   0                100               0          14.1±0.3ms
             ====== ===================== =================== ================ =============== =============

[92.86%] ··· ...rojectRandomJobBench.time_search_lean_filter                 ok
[92.86%] ··· ====== ===================== =================== ================ =============== =============
· Creating environments
· Discovering benchmarks
· Running 14 total benchmarks (2 commits * 1 environments * 7 benchmarks)
[ 0.00%] · For signac commit 79f9a157 <feature/jobscursor-getitem>:
[ 0.00%] ·· Benchmarking virtualenv-py3.12
[ 3.57%] ··· Running (benchmarks.ProjectBench.time_determine_len--)..
[10.71%] ··· Running (benchmarks.ProjectBench.time_iterate_load_sp--)..
[17.86%] ··· Running (benchmarks.ProjectRandomJobBench.time_search_lean_filter--)..
[25.00%] ··· Running (benchmarks.ProjectRandomJobBench.time_select_by_id--).
[28.57%] ··· benchmarks.ProjectBench.time_determine_len                      ok
[28.57%] ··· ====== ===================== =================== ================ =============== =============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0           302±7μs
              1000            10                   0                100               0         2.83±0.03ms
             ====== ===================== =================== ================ =============== =============

[32.14%] ··· benchmarks.ProjectBench.time_iterate                            ok
[32.14%] ··· ====== ===================== =================== ================ =============== ============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ------------
              100             10                   0                100               0         14.3±0.2ms
              1000            10                   0                100               0          140±3ms
             ====== ===================== =================== ================ =============== ============

[35.71%] ··· benchmarks.ProjectBench.time_iterate_load_sp                    ok
[35.71%] ··· ====== ===================== =================== ================ =============== ============
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ------------
"result.txt" 128L, 11398C                                                                            1,1           Top
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- -------------
              100             10                   0                100               0         1.83±0.01ms
              1000            10                   0                100               0           115±20ms
             ====== ===================== =================== ================ =============== =============

[96.43%] ··· ...rojectRandomJobBench.time_search_rich_filter                 ok
[96.43%] ··· ====== ===================== =================== ================ =============== ==========
               N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
             ------ --------------------- ------------------- ---------------- --------------- ----------
              100             10                   0                100               0         7.75±9ms
              1000            10                   0                100               0         186±10ms
             ====== ===================== =================== ================ =============== ==========

[100.00%] ··· ...arks.ProjectRandomJobBench.time_select_by_id                 ok
[100.00%] ··· ====== ===================== =================== ================ =============== =============
                N     num_statepoint_keys   num_document_keys   data_size_mean   data_size_std
              ------ --------------------- ------------------- ---------------- --------------- -------------
               100             10                   0                100               0          7.48±0.2μs
               1000            10                   0                100               0         7.43±0.03μs
              ====== ===================== =================== ================ =============== =============

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request python Pull requests that update Python code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants