+
+
+ DataFrame
+ |
+
+
+ A pandas dataframe with the requested attributes as columns and indexed by l2_id.
|
diff --git a/objects.inv b/objects.inv
index f2a612f9..dd1e0f4a 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search/search_index.json b/search/search_index.json
index 94f487e8..6a8481ac 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"CAVE is short for Connectome Annotation Versioning Engine. CAVE is a set of microservices that provide a framework for storing and versioning connectomics data and large sets of dynamic annotations, metadata, and segmentations. This repository supplies client-side code to easily interact with the microservices in CAVE.
"},{"location":"#documentation-layout","title":"Documentation Layout","text":"To learn how to install caveclient, visit Installation.
To see hands-on examples of using caveclient in a notebook, visit the Tutorials.
To see API reference documentation for interacting with a caveclient.CAVEclient object, visit the Client API.
To see a glossary of terms used in the documentation, visit the Glossary (work in progress).
For information about how to contribute to the documentation or the package, visit the Contributing page. Feedback on the documentation is welcome! Please open an issue or use the \"Edit this page\" button at the top right of any page to suggest changes.
To read a full description of the Connectome Annotation Versioning Engine, including a description of the various services please read this paper.
"},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#600-october-14-2024","title":"6.0.0 (October 14, 2024)","text":" - Refactored CAVEclient internals away from a factory syntax. Most users should not notice a difference, but code that relied on specific subclient type logic may be affected.
"},{"location":"changelog/#5250","title":"5.25.0","text":" - Added ability to suppress table warnings in client.materialize.query_table and similar methods by setting
log_warnings=False
"},{"location":"changelog/#5200-april-8-2024","title":"5.20.0 (April 8, 2024)","text":" - Added generalized support for detecting server versions to provide timely exceptions to users
- Used new support to check that chunkegraph has updated version before using spatial bounds kwarg on client.chunkedgraph.level2_chunk_graph
- Added support for postign and getting segment properties files to client.state
"},{"location":"changelog/#5180","title":"5.18.0","text":" - Added serialization support for pandas.index
"},{"location":"changelog/#5173","title":"5.17.3","text":" - Minor documentation typo fix
"},{"location":"changelog/#5172","title":"5.17.2","text":" - Bug fixes related to table_manager interface
"},{"location":"changelog/#5171","title":"5.17.1","text":" - Bug fixes related to table_manager interface
"},{"location":"changelog/#5170","title":"5.17.0","text":" - Fix attrs in dataframe attributes of client.materialize results to remove numpy arrays to allow concatenation of dataframes
- Added getting multiple schemas in one call to improve initialization of table_manager interface of materialization
"},{"location":"changelog/#5161","title":"5.16.1","text":" - Bugfix on client.chunkedgrpah.level2_chunk_graph
"},{"location":"changelog/#5160","title":"5.16.0","text":" - Added bounding box query to client.chunkedgraph.level2_chunk_graph
- Fix default materialization version client when server not advertising correctly
"},{"location":"changelog/#5151-jan-18-2024","title":"5.15.1 (Jan 18, 2024)","text":" - minor improvements to release process
"},{"location":"changelog/#5150-jan-18-2024","title":"5.15.0 (Jan 18, 2024)","text":" - Improved documentation with types
- Improved testing on more python versions
- Bugfixes for pyton 3.12 compatability
"},{"location":"changelog/#5140-november-24-2023","title":"5.14.0 (November 24, 2023)","text":" - Made automatic detection of neuroglancer versioning when constructing link shortener links
"},{"location":"changelog/#5130-october-26-2023","title":"5.13.0 (October 26, 2023)","text":" - Add option to get expired versions to client.materialize.get_versions
"},{"location":"changelog/#5121-october-16-2023","title":"5.12.1 (October 16, 2023)","text":" - Bugfixes for client.chunkedgraph.get_latest_roots
"},{"location":"changelog/#5120-october-16-2023","title":"5.12.0 (October 16, 2023)","text":" - Improved logic for client.chunkedgraph.get_latest_roots to work forward or backwards in time
"},{"location":"changelog/#5110-september-19-2023","title":"5.11.0 (September 19, 2023)","text":" - Added filter_regex_dict options to client.materialize.query_table interface
"},{"location":"changelog/#5102-august-162023","title":"5.10.2 (August 16,2023)","text":" - Fixed pyarrow support for live_live query
"},{"location":"changelog/#5101-august-142023","title":"5.10.1 (August 14,2023)","text":" - Changed random_sample argument to be an integer number of annotations rather than a floating fraction of table
- Added option to live_query
"},{"location":"changelog/#590-august-14-2023","title":"5.9.0 (August 14, 2023)","text":" - Added support for native pyarrow deserialization, allowing upgrade to pyarrow version
"},{"location":"changelog/#580","title":"5.8.0","text":" - Allowed int64 root ids to serialize properly
- Added warning that client.materialize.tables interface is in beta
"},{"location":"changelog/#570","title":"5.7.0","text":" - Fix to ensure stop_layer is at least 1
- Added client.chunkedgraph.suggest_latest_roots
"},{"location":"changelog/#560","title":"5.6.0","text":" - Added views to client.materialize.tables interface
- Added optional argument to allow invalid root ids when querying live live, versus creating an exception
"},{"location":"changelog/#551","title":"5.5.1","text":" - documentation fixes on client.materialize.join_query
"},{"location":"changelog/#550","title":"5.5.0","text":" - added methods for different neuroglancer state formats to client.state.
"},{"location":"changelog/#543","title":"5.4.3","text":" - Added 'view' querying options to materialization
- Added client.materialize.tables interface
- Added client.materialize.get_tables_metadata to get all metadata in one call
"},{"location":"changelog/#520","title":"5.2.0","text":" - Added local caching of datastack names > server_address to simplify initialization of clients with servers other than global.daf-apis.com.
Cache is saved on a local file ~/.cloudvolume/secrets/cave_datastack_to_server_map.json
Cache will populate the first time caveclient.CAVEclient('my_datastack', server_address=\"https://my_server.com\") is called. Subsequent calls can then just be caveclient.CAVEclient('my_datastack').
"},{"location":"changelog/#510","title":"5.1.0","text":" - Added get_oldest_timestamp call to chunkedgraph
"},{"location":"changelog/#501","title":"5.0.1","text":" - Fixed bug with desired_resolution being set at the client level was being ignored in >5.0.0
"},{"location":"changelog/#500","title":"5.0.0","text":" - Added support for the new CAVE Materialization 3.0 API Includes support for the new materialization API, which allows for server side conversion of the units of position, and ensures that all positions are returned with the same units, even after joins.
- Added support for querying databases that were materialized without merging tables together. This will allow for faster materializations.
- Removed support for LiveLive query from the Materialization 2.0 API client. Note.. <5.0.0 clients interacting with MaterializationEngine >4.7.0 servers will use live live query but will doubly convert the units of position if you ask for a desired resolution, as the old client will also do a conversion server side.
- Fixed interaction with api version querying of servers from individual clients to work with verify=False. (useful for testing)
- Stored infromation from client about mapping between dataframe and table names and original column names.
- Added support for suffixes and select columns to be passed by dictionary rather than list making the selection an application of suffixes more explicit when there are collisions between column names in joined tables.
"},{"location":"changelog/#older-upgrade-notes","title":"Older Upgrade Notes","text":"Change all select_column calls to pass dictionaries rather than lists. Change all suffix calls to pass dictionaries rather than lists. Advocate for your server administrator to upgrade to MaterializationEngine 4.7.0 or later, so you can use the new MaterializationEngine 3.0 API and client.
"},{"location":"contributing/","title":"Contributing","text":"Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
You can contribute in many ways:
"},{"location":"contributing/#types-of-contributions","title":"Types of Contributions","text":""},{"location":"contributing/#report-bugs","title":"Report Bugs","text":"Report bugs to our issues page.
If you are reporting a bug, please include:
- Your operating system name and version.
- Any details about your local setup that might be helpful in troubleshooting.
- Detailed steps to reproduce the bug, in the form of a minimal reproducible example.
"},{"location":"contributing/#fix-bugs","title":"Fix Bugs","text":"Look through the GitHub issues for bugs. Anything tagged with \"bug\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#implement-features","title":"Implement Features","text":"Look through the GitHub issues for features. Anything tagged with \"enhancement\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#write-documentation","title":"Write Documentation","text":"caveclient could always use more documentation, whether as part of the official caveclient docs, in docstrings, or even on the web in blog posts, articles, and such.
"},{"location":"contributing/#submit-feedback","title":"Submit Feedback","text":"The best way to send feedback is to create an issue on GitHub.
If you are proposing a feature:
- Explain in detail how it would work.
- Keep the scope as narrow as possible, to make it easier to implement.
- Remember that while contributions are welcome, developer/maintainer time is limited.
"},{"location":"contributing/#get-started","title":"Get Started","text":"Ready to contribute? Here's how to set up caveclient for local development.
- Fork the repo on GitHub.
- Clone your fork locally
git clone git@github.com:your_name_here/CAVEclient.git\n
- Ensure pip is installed.
- Create a virtual environment (here we use venv):
python3 -m venv .venv\n
source .venv/bin/activate\n
- Create a branch for local development:
git checkout -b name-of-your-bugfix-or-feature\n
- Make your changes locally
- Install development requirements:
pip install -r test_requirements.txt\npip install -e .\n
- When you're done making changes, check that your changes pass the tests by running pytest:
pytest tests\n
Note that once you submit your pull request, GitHub Actions will run the tests also, including on multiple operating systems and Python versions. Your pull request will have to pass on all of these before it can be merged.
- Ensure your contribution meets style guidelines. First, install ruff:
pip install ruff\n
- Fix linting and formatting. From the root of the repository, run the following commands:
ruff check . --extend-select I --fix\nruff format .\n
- Commit your changes and push your branch to GitHub:
git add .\ngit commit -m \"Your detailed description of your changes.\"\ngit push origin name-of-your-bugfix-or-feature\n
- Submit a pull request through the GitHub website.
"},{"location":"contributing/#pull-request-guidelines","title":"Pull Request Guidelines","text":"Before you submit a pull request, check that it meets these guidelines:
- The pull request should include tests if adding a new feature.
- The docs should be updated with whatever changes you have made. Put your new functionality into a function with a docstring, and make sure the new functionality is documented after building the documentation.
"},{"location":"contributing/#documentation-style","title":"Documentation style","text":"We use mkdocs to build the documentation. In particular, we use the mkdocs-material theme, and a variety of other extensions.
Note
More information codifying our documentation style and principles coming soon. For now, just try to follow the style of the existing documentation.
"},{"location":"glossary/","title":"Glossary","text":"Warning
This glossary is a work in progress; for now we are documenting the commonly used terms that we need to define. Please feel free to contribute definitions or additional terms.
- Datastack
- Voxel resolution
- Segmentation resolution
- MIP
- Segmentation
- View
"},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#stable-release","title":"Stable release","text":"To install caveclient, run this command in your terminal:
pip install caveclient\n
This is the preferred method to install caveclient, as it will always install the most recent stable release.
You can also specify a particular version, e.g.
pip install caveclient==5.0.0\n
If you don't have pip installed, this Python installation guide can guide you through the process.
"},{"location":"installation/#from-source","title":"From source","text":"The source for caveclient can be downloaded from the Github repo.
You can either clone the public repository:
git clone git://github.com/CAVEconnectome/CAVEclient\n
Or download the tarball:
curl -OJL https://github.com/CAVEconnectome/CAVEclient/tarball/master\n
Once you have a copy of the source, you can install it with:
pip install .\n
Or in editable mode, it can be installed with:
pip install -e .\n
"},{"location":"api/","title":"Overview","text":"The most common method of interacting with the CAVE Framework is by instantiating a client (caveclient.CAVEclient) and then using that client to interact with various services. Under the hood, the CAVEclient is a collection of individual clients, which can be accessed via properties. For example, to access the materialization client, you can use client.materialize, which (up to the exact version) will actually return a MaterializationClient object.
These pages describe the functionality of each of the individual clients, but note that some features may not be available if the server you are accessing is using older versions of particular services.
"},{"location":"api/annotation/","title":"client.annotation","text":"Client for interacting with the annotation engine.
Methods:
Name Description create_table Creates a new data table based on an existing schema
delete_annotation Delete one or more annotations in a table. Annotations that are
delete_table Marks a table for deletion requires super admin privileges
get_annotation Retrieve an annotation or annotations by id(s) and table name.
get_annotation_count Get number of annotations in a table
get_table_metadata Get metadata about a table
get_tables Gets a list of table names for a aligned_volume_name
post_annotation Post one or more new annotations to a table in the AnnotationEngine.
post_annotation_df Post one or more new annotations to a table in the AnnotationEngine.
process_position_columns Process a dataframe into a list of dictionaries
raise_for_status Raises requests.HTTPError, if one occurred.
stage_annotations Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema.
update_annotation Update one or more new annotations to a table in the AnnotationEngine.
update_annotation_df Update one or more annotations to a table in the AnnotationEngine using a
update_metadata Update the metadata on an existing table
upload_staged_annotations Upload annotations directly from an Annotation Guide object.
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.create_table","title":"create_table(table_name, schema_name, description, voxel_resolution, reference_table=None, track_target_id_updates=None, flat_segmentation_source=None, user_id=None, aligned_volume_name=None, write_permission='PRIVATE', read_permission='PUBLIC', notice_text=None)","text":"Creates a new data table based on an existing schema
Parameters:
Name Type Description Default table_name str Name of the new table. Cannot be the same as an existing table
required schema_name str Name of the schema for the new table.
required description str Human readable description for what is in the table. Should include information about who generated the table What data it covers, and how it should be interpreted. And who should you talk to if you want to use it. An Example: a manual synapse table to detect chandelier synapses on 81 PyC cells with complete AISs [created by Agnes - agnesb@alleninstitute.org, uploaded by Forrest]
required voxel_resolution List[float] voxel resolution points will be uploaded in, typically nm, i.e [1,1,1] means nanometers [4,4,40] would be 4nm, 4nm, 40nm voxels
required reference_table str If the schema you are using is a reference schema Meaning it is an annotation of another annotation. Then you need to specify what the target table those annotations are in.
None track_target_id_updates bool Indicates whether to automatically update reference table's foreign key if target annotation table row is updated.
None flat_segmentation_source str the source to a flat segmentation that corresponds to this table i.e. precomputed:\\gs:\\mybucket his_tables_annotation
None user_id int If you are uploading this schema on someone else's behalf and you want to link this table with their ID, you can specify it here Otherwise, the table will be created with your userID in the user_id column.
None aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table (DEFAULT) GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable
'PRIVATE' read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data (DEFAULT)
'PUBLIC' notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None, no text. If you want to remove text, send empty string.
None Returns:
Type Description json Response JSON
Examples:
Basic annotation table:
description = \"Some description about the table\"\nvoxel_res = [4,4,40]\nclient.create_table(\"some_synapse_table\", \"synapse\", description, voxel_res)\n
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_annotation","title":"delete_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Delete one or more annotations in a table. Annotations that are deleted are recorded as 'non-valid' but are not physically removed from the table.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required annotation_ids (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_table","title":"delete_table(table_name, aligned_volume_name=None)","text":"Marks a table for deletion requires super admin privileges
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation","title":"get_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Retrieve an annotation or annotations by id(s) and table name.
Parameters:
Name Type Description Default table_name str Name of the table
required annotation_ids int or iterable ID or IDS of the annotation to retreive
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description list Annotation data
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation_count","title":"get_annotation_count(table_name, aligned_volume_name=None)","text":"Get number of annotations in a table
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description int number of annotations
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_table_metadata","title":"get_table_metadata(table_name, aligned_volume_name=None)","text":"Get metadata about a table
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json metadata about table
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_tables","title":"get_tables(aligned_volume_name=None)","text":"Gets a list of table names for a aligned_volume_name
Parameters:
Name Type Description Default aligned_volume_name str or None Name of the aligned_volume, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Description list List of table names
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation","title":"post_annotation(table_name, data, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations refer to 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required data (dict or list) A list of (or a single) dict of schematized annotation data matching the target table.
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation_df","title":"post_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations see 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required df DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
required position_columns Optional[Union[Iterable[str], Mapping[str, str]]] if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.process_position_columns","title":"process_position_columns(df, position_columns) staticmethod","text":"Process a dataframe into a list of dictionaries
Parameters:
Name Type Description Default df DataFrame Dataframe to process
required position_columns Optional[Union[Iterable[str], Mapping[str, str]]] See .post_annotation_df
required Returns:
Type Description dict Annotations ready for posting
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.raise_for_status","title":"raise_for_status(r, log_warning=True) staticmethod","text":"Raises requests.HTTPError, if one occurred.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.stage_annotations","title":"stage_annotations(table_name=None, schema_name=None, update=False, id_field=False, table_resolution=None, annotation_resolution=None)","text":"Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema. StagedAnnotation objects can be uploaded directly with upload_staged_annotations.
Parameters:
Name Type Description Default table_name str Table name to stage annotations for, by default None.
None schema_name str Schema name to use to make annotations. Only needed if the table_name is not set, by default None
None update bool Set to True if individual annotations are going to be updated, by default False.
False id_field bool Set to True if id fields are to be specified. Not needed if update is True, which always needs id fields. Optional, by default False
False table_resolution list - like or None Voxel resolution of spatial points in the table in nanometers. This is found automatically from the info service if a table name is provided, by default None. If annotation_resolution is also set, this allows points to be scaled correctly for the table.
None annotation_resolution list - like Voxel resolution of spatial points provided by the user when creating annotations. If the table resolution is also available (manually or from the info service), annotations are correctly rescaled for the volume. By default, None.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation","title":"update_annotation(table_name, data, aligned_volume_name=None)","text":"Update one or more new annotations to a table in the AnnotationEngine. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
Notes If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required data (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation_df","title":"update_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Update one or more annotations to a table in the AnnotationEngine using a dataframe as format. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
Notes If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required df DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
required position_columns Iterable[str] or Mapping[str, str] or None if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_metadata","title":"update_metadata(table_name, description=None, flat_segmentation_source=None, read_permission=None, write_permission=None, user_id=None, notice_text=None, aligned_volume_name=None)","text":"Update the metadata on an existing table
Parameters:
Name Type Description Default table_name str required description str Defaults to None (will not update).
None flat_segmentation_source str Defaults to None (will not update).
None read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data Defaults to None (will not update).
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable Defaults to None (will not update).
None user_id int Note, if you use this you will not be able to update the metadata on this table any longer and depending on permissions may not be able to read or write to it Defaults to None. (will not update)
None notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None. (will not update)
None aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.upload_staged_annotations","title":"upload_staged_annotations(staged_annos, aligned_volume_name=None)","text":"Upload annotations directly from an Annotation Guide object. This method uses the options specified in the object, including table name and if the annotation is an update or not.
Parameters:
Name Type Description Default staged_annos AnnotationGuide AnnotationGuide object with a specified table name and a collection of annotations already filled in.
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description List or dict If new annotations are posted, a list of ids. If annotations are being updated, a dictionary with the mapping from old ids to new ids.
"},{"location":"api/auth/","title":"client.auth","text":"Client to find and use auth tokens to access the other services.
Methods:
Name Description get_group_users Get users in a group
get_new_token Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
get_token Load a token with a given key the specified token file
get_tokens Get the tokens setup for this users
get_user_information Get user data.
save_token Conveniently save a token in the correct format.
setup_token Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment.
Attributes:
Name Type Description request_header Formatted request header with the specified token
token Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.request_header","title":"request_header property","text":"Formatted request header with the specified token
"},{"location":"api/auth/#caveclient.auth.AuthClient.token","title":"token property writable","text":"Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_group_users","title":"get_group_users(group_id)","text":"Get users in a group
Parameters:
Name Type Description Default group_id int ID value for a given group
required Returns:
Type Description list List of dicts of user ids. Returns empty list if group does not exist.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_new_token","title":"get_new_token(open=False, no_text=False)","text":"Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Default open bool If True, opens a web browser to the web page where you can generate a new token.
False"},{"location":"api/auth/#caveclient.auth.AuthClient.get_token","title":"get_token(token_key=None)","text":"Load a token with a given key the specified token file
Parameters:
Name Type Description Default token_key str or None key in the token file JSON, by default None. If None, uses 'token'.
None"},{"location":"api/auth/#caveclient.auth.AuthClient.get_tokens","title":"get_tokens()","text":"Get the tokens setup for this users
Returns:
Type Description list[dict]: a list of dictionary of tokens, each with the keys \"id\": the id of this token \"token\": the token (str) \"user_id\": the users id (should be your ID)
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_user_information","title":"get_user_information(user_ids)","text":"Get user data.
Parameters:
Name Type Description Default user_ids list of int user_ids to look up
required"},{"location":"api/auth/#caveclient.auth.AuthClient.save_token","title":"save_token(token=None, token_key=default_token_key, overwrite=False, token_file=None, switch_token=True, write_to_server_file=True, ignore_readonly=True)","text":"Conveniently save a token in the correct format.
After getting a new token by following the instructions in authclient.get_new_token(), you can save it with a fully default configuration by running:
token = 'my_shiny_new_token'
authclient.save_token(token=token)
Now on next load, authclient=AuthClient() will make an authclient instance using this token. If you would like to specify more information about the json file where the token will be stored, see the parameters below.
Parameters:
Name Type Description Default token str New token to save, by default None
None token_key str Key for the token in the token_file json, by default \"token\"
default_token_key overwrite bool Allow an existing token to be changed, by default False
False token_file str Path to the token file, by default None. If None, uses the default file location specified above.
None switch_token bool If True, switch the auth client over into using the new token, by default True
True write_to_server_file bool If True, will write token to a server specific file to support this machine interacting with multiple auth servers.
True ignore_readonly bool If True, will only attempt to save a token if the directory is writeable.
True"},{"location":"api/auth/#caveclient.auth.AuthClient.setup_token","title":"setup_token(make_new=True, open=True)","text":"Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Default make_new bool If True, will make a new token, else prompt you to open a page to retrieve an existing token.
True open bool If True, opens a web browser to the web page where you can retrieve a token.
True"},{"location":"api/chunkedgraph/","title":"client.chunkedgraph","text":"Client for interacting with the chunkedgraph.
Methods:
Name Description get_roots Get the root ID for a list of supervoxels.
get_root_id Get the root ID for a specified supervoxel.
get_merge_log Get the merge log (splits and merges) for an object.
get_change_log Get the change log (splits and merges) for an object.
get_user_operations Get operation details for a user ID. Currently, this is only available to
get_tabular_change_log Get a detailed changelog for neurons.
get_leaves Get all supervoxels for a root ID.
do_merge Perform a merge on the chunked graph.
undo_operation Undo an operation.
execute_split Execute a multicut split based on points or supervoxels.
preview_split Get supervoxel connected components from a preview multicut split.
get_children Get the children of a node in the chunked graph hierarchy.
get_contact_sites Get contacts for a root ID.
find_path Find a path between two locations on a root ID using the level 2 chunked
get_subgraph Get subgraph of root id within a bounding box.
level2_chunk_graph Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
remesh_level2_chunks Submit specific level 2 chunks to be remeshed in case of a problem.
get_operation_details Get the details of a list of operations.
get_lineage_graph Returns the lineage graph for a root ID, optionally cut off in the past or
get_latest_roots Returns root IDs that are related to the given root_id at a given
get_original_roots Returns root IDs that are the latest successors of a given root ID.
is_latest_roots Check whether these root IDs are still a root at this timestamp.
suggest_latest_roots Suggest latest roots for a given root id, based on overlap of component
is_valid_nodes Check whether nodes are valid for given timestamp range.
get_root_timestamps Retrieves timestamps when roots where created.
get_past_ids For a set of root IDs, get the list of IDs at a past or future time point
get_delta_roots Get the list of roots that have changed between timetamp_past and
get_oldest_timestamp Get the oldest timestamp in the database.
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
timestamp Optional[datetime] The default timestamp for queries which expect a timestamp. If None, uses the
segmentation_info Complete segmentation metadata
base_resolution MIP 0 resolution for voxels assumed by the ChunkedGraph
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.timestamp","title":"timestamp: Optional[datetime.datetime] property writable","text":"The default timestamp for queries which expect a timestamp. If None, uses the current time.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.segmentation_info","title":"segmentation_info property","text":"Complete segmentation metadata
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.base_resolution","title":"base_resolution property","text":"MIP 0 resolution for voxels assumed by the ChunkedGraph
Returns:
Type Description list 3-long list of x/y/z voxel dimensions in nm
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_roots","title":"get_roots(supervoxel_ids, timestamp=None, stop_layer=None)","text":"Get the root ID for a list of supervoxels.
Parameters:
Name Type Description Default supervoxel_ids list or np.array of int Supervoxel IDs to look up.
required timestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None stop_layer int or None If True, looks up IDs only up to a given stop layer. Default is None.
None Returns:
Type Description np.array of np.uint64 Root IDs containing each supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_id","title":"get_root_id(supervoxel_id, timestamp=None, level2=False)","text":"Get the root ID for a specified supervoxel.
Parameters:
Name Type Description Default supervoxel_id int Supervoxel id value
required timestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description int64 Root ID containing the supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_merge_log","title":"get_merge_log(root_id)","text":"Get the merge log (splits and merges) for an object.
Parameters:
Name Type Description Default root_id int Object root ID to look up.
required Returns:
Type Description list List of merge events in the history of the object.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_change_log","title":"get_change_log(root_id, filtered=True)","text":"Get the change log (splits and merges) for an object.
Parameters:
Name Type Description Default root_id int Object root ID to look up.
required filtered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query object root_id (filtered=False). Defaults to True.
True Returns:
Type Description dict Dictionary summarizing split and merge events in the object history, containing the following keys:
- \"n_merges\": int
- \"n_splits\": int
- \"operations_ids\": list of int
- Identifiers for each operation
- \"past_ids\": list of int
- Previous root ids for this object
- \"user_info\": dict of dict
- Dictionary keyed by user (string) to a dictionary specifying how many merges and splits that user performed on this object
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_user_operations","title":"get_user_operations(user_id, timestamp_start, include_undo=True, timestamp_end=None)","text":"Get operation details for a user ID. Currently, this is only available to admins.
Parameters:
Name Type Description Default user_id int User ID to query (use 0 for all users (admin only)).
required timestamp_start datetime Timestamp to start filter (UTC).
required include_undo bool Whether to include undos. Defaults to True.
True timestamp_end datetime Timestamp to end filter (UTC). Defaults to now.
None Returns:
Type Description DataFrame DataFrame including the following columns:
- \"operation_id\": int
- Identifier for the operation.
- \"timestamp\": datetime.datetime
- Timestamp of the operation.
- \"user_id\": int
- User who performed the operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_tabular_change_log","title":"get_tabular_change_log(root_ids, filtered=True)","text":"Get a detailed changelog for neurons.
Parameters:
Name Type Description Default root_ids list of int Object root IDs to look up.
required filtered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query objects in root_ids (filtered=False). Defaults to True.
True Returns:
Type Description dict of pd.DataFrame The keys are the root IDs, and the values are DataFrames with the following columns and datatypes:
- \"operation_id\": int
- Identifier for the operation.
- \"timestamp\": int
- Timestamp of the operation, provided in milliseconds. To convert to datetime, use
datetime.datetime.utcfromtimestamp(timestamp/1000).
- \"user_id\": int
- User who performed the operation.
- \"before_root_ids: list of int
- Root IDs of objects that existed before the operation.
- \"after_root_ids: list of int
- Root IDs of objects created by the operation. Note that this only records the root id that was kept as part of the query object, so there will only be one in this list.
- \"is_merge\": bool
- Whether the operation was a merge.
- \"user_name\": str
- Name of the user who performed the operation.
- \"user_affiliation\": str
- Affiliation of the user who performed the operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves","title":"get_leaves(root_id, bounds=None, stop_layer=None)","text":"Get all supervoxels for a root ID.
Parameters:
Name Type Description Default root_id int Root ID to query.
required bounds If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.
None stop_layer int If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).
None Returns:
Type Description np.array of np.int64 Array of supervoxel IDs (or node ids if stop_layer>1).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.do_merge","title":"do_merge(supervoxels, coords, resolution=(4, 4, 40))","text":"Perform a merge on the chunked graph.
Parameters:
Name Type Description Default supervoxels iterable An N-long list of supervoxels to merge.
required coords array An Nx3 array of coordinates of the supervoxels in units of resolution.
required resolution tuple What to multiply coords by to get nanometers. Defaults to (4,4,40).
(4, 4, 40)"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.undo_operation","title":"undo_operation(operation_id)","text":"Undo an operation.
Parameters:
Name Type Description Default operation_id int Operation ID to undo.
required Returns:
Type Description dict"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.execute_split","title":"execute_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None)","text":"Execute a multicut split based on points or supervoxels.
Parameters:
Name Type Description Default source_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
required sink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
required root_id int Root ID of object to do split preview.
required source_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None Returns:
Name Type Description operation_id int Unique ID of the split operation
new_root_ids list of int List of new root IDs resulting from the split operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.preview_split","title":"preview_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None, return_additional_ccs=False)","text":"Get supervoxel connected components from a preview multicut split.
Parameters:
Name Type Description Default source_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
required sink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
required root_id int Root ID of object to do split preview.
required source_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None return_additional_ccs bool If True, returns any additional connected components beyond the ones with source and sink points. In most situations, this can be ignored. By default, False.
False Returns:
Name Type Description source_connected_component list Supervoxel IDs in the component with the most source points.
sink_connected_component list Supervoxel IDs in the component with the most sink points.
successful_split bool True if the split worked.
other_connected_components (optional) : list of lists of int List of lists of supervoxel IDs for any other resulting connected components. Only returned if return_additional_ccs is True.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_children","title":"get_children(node_id)","text":"Get the children of a node in the chunked graph hierarchy.
Parameters:
Name Type Description Default node_id int Node ID to query.
required Returns:
Type Description np.array of np.int64 IDs of child nodes.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_contact_sites","title":"get_contact_sites(root_id, bounds, calc_partners=False)","text":"Get contacts for a root ID.
Parameters:
Name Type Description Default root_id int Root ID to query.
required bounds Bounds within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]] for which to find contacts. Running this query without bounds is too slow.
required calc_partners bool If True, get partner root IDs. By default, False.
False Returns:
Type Description dict Dict relating ids to contacts
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.find_path","title":"find_path(root_id, src_pt, dst_pt, precision_mode=False)","text":"Find a path between two locations on a root ID using the level 2 chunked graph.
Parameters:
Name Type Description Default root_id int Root ID to query.
required src_pt array 3-element array of xyz coordinates in nm for the source point.
required dst_pt array 3-element array of xyz coordinates in nm for the destination point.
required precision_mode bool Whether to perform the search in precision mode. Defaults to False.
False Returns:
Name Type Description centroids_list array Array of centroids along the path.
l2_path np.array of int Array of level 2 chunk IDs along the path.
failed_l2_ids np.array of int Array of level 2 chunk IDs that failed to find a path.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_subgraph","title":"get_subgraph(root_id, bounds)","text":"Get subgraph of root id within a bounding box.
Parameters:
Name Type Description Default root_id int Root (or any node ID) of chunked graph to query.
required bounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates.
required Returns:
Type Description np.array of np.int64 Node IDs in the subgraph.
np.array of np.double Affinities of edges in the subgraph.
np.array of np.int32 Areas of nodes in the subgraph.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.level2_chunk_graph","title":"level2_chunk_graph(root_id, bounds=None)","text":"Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
Parameters:
Name Type Description Default root_id int Root id of object
required bounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates (use client.chunkedgraph.base_resolution to view this default resolution for your chunkedgraph client). Note that the result will include any level 2 nodes which have chunk boundaries within some part of this bounding box, meaning that the representative point for a given level 2 node could still be slightly outside of these bounds. If None, returns all level 2 chunks for the root ID.
None Returns:
Type Description list of list Edge list for level 2 chunked graph. Each element of the list is an edge, and each edge is a list of two node IDs (source and target).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.remesh_level2_chunks","title":"remesh_level2_chunks(chunk_ids)","text":"Submit specific level 2 chunks to be remeshed in case of a problem.
Parameters:
Name Type Description Default chunk_ids list List of level 2 chunk IDs.
required"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_operation_details","title":"get_operation_details(operation_ids)","text":"Get the details of a list of operations.
Parameters:
Name Type Description Default operation_ids Iterable[int] List/array of operation IDs.
required Returns:
Type Description dict of str to dict A dict of dicts of operation info, keys are operation IDs (as strings), values are a dictionary of operation info for the operation. These dictionaries contain the following keys:
- \"added_edges\"/\"removed_edges\": list of list of int
- List of edges added (if a merge) or removed (if a split) by this operation. Each edge is a list of two supervoxel IDs (source and target).
- \"roots\": list of int
- List of root IDs that were created by this operation.
- \"sink_coords\": list of list of int
- List of sink coordinates for this operation. The sink is one of the points placed by the user when specifying the operation. Each sink coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
- \"source_coords\": list of list of int
- List of source coordinates for this operation. The source is one of the points placed by the user when specifying the operation. Each source coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
- \"timestamp\": str
- Timestamp of the operation.
- \"user\": str
- User ID number who performed the operation (as a string).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_lineage_graph","title":"get_lineage_graph(root_id, timestamp_past=None, timestamp_future=None, as_nx_graph=False, exclude_links_to_future=False, exclude_links_to_past=False)","text":"Returns the lineage graph for a root ID, optionally cut off in the past or the future.
Each change in the chunked graph creates a new root ID for the object after that change. This function returns a graph of all root IDs for a given object, tracing the history of the object in terms of merges and splits.
Parameters:
Name Type Description Default root_id int Object root ID.
required timestamp_past datetime or None Cutoff for the lineage graph backwards in time. By default, None.
None timestamp_future datetime or None Cutoff for the lineage graph going forwards in time. By default, uses the timestamp property for this client, which defaults to the current time.
None as_nx_graph If True, a NetworkX graph is returned.
False exclude_links_to_future If True, links from nodes before timestamp_future to after timestamp_future are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False exclude_links_to_past If True, links from nodes before timestamp_past to after timestamp_past are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False Returns:
Type Description dict Dictionary describing the lineage graph and operations for the root ID. Not returned if as_nx_graph is True. The dictionary contains the following keys:
- \"directed\" : bool
- Whether the graph is directed.
- \"graph\" : dict
- Dictionary of graph attributes.
- \"links\" : list of dict
- Each element of the list is a dictionary describing an edge in the lineage graph as \"source\" and \"target\" keys.
- \"multigraph\" : bool
- Whether the graph is a multigraph.
- \"nodes\" : list of dict
- Each element of the list is a dictionary describing a node in the lineage graph, usually with \"id\", \"timestamp\", and \"operation_id\" keys.
DiGraph NetworkX directed graph of the lineage graph. Only returned if as_nx_graph is True.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_latest_roots","title":"get_latest_roots(root_id, timestamp=None, timestamp_future=None)","text":"Returns root IDs that are related to the given root_id at a given timestamp. Can be used to find the \"latest\" root IDs associated with an object.
Parameters:
Name Type Description Default root_id int Object root ID.
required timestamp datetime or None Timestamp of where to query IDs from. If None, uses the timestamp property for this client, which defaults to the current time.
None timestamp_future datetime or None DEPRECATED name, use timestamp instead. Timestamp to suggest IDs from (note can be in the past relative to the root). By default, None.
None Returns:
Type Description ndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_original_roots","title":"get_original_roots(root_id, timestamp_past=None)","text":"Returns root IDs that are the latest successors of a given root ID.
Parameters:
Name Type Description Default root_id int Object root ID.
required timestamp_past datetime or None Cutoff for the search going backwards in time. By default, None.
None Returns:
Type Description ndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_latest_roots","title":"is_latest_roots(root_ids, timestamp=None)","text":"Check whether these root IDs are still a root at this timestamp.
Parameters:
Name Type Description Default root_ids array-like of int Root IDs to check.
required timestamp datetime Timestamp to check whether these IDs are valid root IDs in the chunked graph. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description np.array of bool Array of whether these are valid root IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.suggest_latest_roots","title":"suggest_latest_roots(root_id, timestamp=None, stop_layer=None, return_all=False, return_fraction_overlap=False)","text":"Suggest latest roots for a given root id, based on overlap of component chunk IDs. Note that edits change chunk IDs, and so this effectively measures the fraction of unchanged chunks at a given chunk layer, which sets the size scale of chunks. Higher layers are coarser.
Parameters:
Name Type Description Default root_id int Root ID of the potentially outdated object.
required timestamp datetime Datetime at which \"latest\" roots are being computed, by default None. If None, uses the timestamp property for this client, which defaults to the current time. Note that this has to be a timestamp after the creation of the root_id.
None stop_layer int Chunk level at which to compute overlap, by default None. No value will take the 4th from the top layer, which emphasizes speed and works well for larger objects. Lower values are slower but more fine-grained. Values under 2 (i.e. supervoxels) are not recommended except in extremely fine grained scenarios.
None return_all bool If True, return all current IDs sorted from most overlap to least, by default False. If False, only the top is returned.
False return_fraction_overlap bool If True, return all fractions sorted by most overlap to least, by default False. If False, only the top value is returned.
False"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_valid_nodes","title":"is_valid_nodes(node_ids, start_timestamp=None, end_timestamp=None)","text":"Check whether nodes are valid for given timestamp range.
Valid is defined as existing in the chunked graph. This makes no statement about these IDs being roots, supervoxel or anything in-between. It also does not take into account whether a root ID has since been edited.
Parameters:
Name Type Description Default node_ids array-like of int Node IDs to check.
required start_timestamp datetime Timestamp to check whether these IDs were valid after this timestamp. Defaults to None (assumes now).
None end_timestamp datetime Timestamp to check whether these IDs were valid before this timestamp. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description np.array of bool Array of whether these are valid IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_timestamps","title":"get_root_timestamps(root_ids, latest=False, timestamp=None)","text":"Retrieves timestamps when roots where created.
Parameters:
Name Type Description Default root_ids Iterable of root IDs to query.
required latest bool If False, returns the first timestamp that the root_id was valid for each root ID. If True, returns the newest/latest timestamp for each root ID. Note, this will return the timestamp at which the query was run when the root is currently valid. This means that you will get a different answer if you make this same query at a later time if you don't specify a timestamp parameter.
False timestamp datetime Timestamp to query when using latest=True. Use this to provide consistent results for a particular timestamp. If an ID is still valid at a point in the future past this timestamp, the query will still return this timestamp as the latest moment in time. An error will occur if you provide a timestamp for which the root ID is not valid. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description np.array of datetime.datetime Array of timestamps when root_ids were created.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_past_ids","title":"get_past_ids(root_ids, timestamp_past=None, timestamp_future=None)","text":"For a set of root IDs, get the list of IDs at a past or future time point that could contain parts of the same object.
Parameters:
Name Type Description Default root_ids Iterable of int Iterable of root IDs to query.
required timestamp_past datetime or None Time of a point in the past for which to look up root ids. Default is None.
None timestamp_future datetime or None Time of a point in the future for which to look up root ids. Not implemented on the server currently. Default is None.
None Returns:
Type Description dict Dict with keys \"future_id_map\" and \"past_id_map\". Each is a dict whose keys are the supplied root_ids and whose values are the list of related root IDs at timestamp_past/timestamp_future.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_delta_roots","title":"get_delta_roots(timestamp_past, timestamp_future=datetime.datetime.now(datetime.timezone.utc))","text":"Get the list of roots that have changed between timetamp_past and timestamp_future.
Parameters:
Name Type Description Default timestamp_past datetime Past timepoint to query
required timestamp_future datetime Future timepoint to query. Defaults to datetime.datetime.now(datetime.timezone.utc).
now(utc) Returns:
Name Type Description old_roots np.ndarray of np.int64 Roots that have expired in that interval.
new_roots np.ndarray of np.int64 Roots that are new in that interval.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_oldest_timestamp","title":"get_oldest_timestamp()","text":"Get the oldest timestamp in the database.
Returns:
Type Description datetime Oldest timestamp in the database.
"},{"location":"api/client/","title":"caveclient.CAVEclient","text":"A manager for all clients sharing common datastack and authentication information.
This client wraps all the other clients and keeps track of the things that need to be consistent across them. To instantiate a client:
from caveclient import CAVEclient\n\nclient = CAVEclient(datastack_name='my_datastack',\n server_address='www.myserver.com',\n auth_token_file='~/.mysecrets/secrets.json')\n
Then
client.annotation is an AnnotationClient (see client.annotation)client.auth is an AuthClient (see client.auth)client.chunkedgraph is a ChunkedGraphClient (see client.chunkedgraph)client.info is an InfoServiceClient (see client.info)client.l2cache is an L2CacheClient (see client.l2cache)client.materialize is a MaterializationClient (see client.materialize)client.skeleton is a SkeletonClient (see client.skeleton)client.schema is a SchemaClient (see client.schema)client.state is a neuroglancer JSONService (see client.state)
All subclients are loaded lazily and share the same datastack name, server address, and auth tokens where used.
Parameters:
Name Type Description Default datastack_name str Datastack name for the services. Almost all services need this and will not work if it is not passed.
None server_address str or None URL of the framework server. If None, chooses the default server global.daf-apis.com. Optional, defaults to None.
None auth_token_file str or None Path to a json file containing the auth token. If None, uses the default location. See Auth client documentation. Optional, defaults to None.
default_token_file auth_token_key str Dictionary key for the token in the the JSON file. Optional, default is 'token'.
'token' auth_token str or None Direct entry of an auth token. If None, uses the file arguments to find the token. Optional, default is None.
None max_retries int or None Sets the default number of retries on failed requests. Optional, by default 2.
None pool_maxsize int or None Sets the max number of threads in a requests pool, although this value will be exceeded if pool_block is set to False. Optional, uses requests defaults if None.
None pool_block If True, prevents the number of threads in a requests pool from exceeding the max size. Optional, uses requests defaults (False) if None.
None desired_resolution Iterable[float] or None If given, should be a list or array of the desired resolution you want queries returned in useful for materialization queries.
None info_cache Pre-computed info cache, bypassing the lookup of datastack info from the info service. Should only be used in cases where this information is cached and thus repetitive lookups can be avoided.
None version Optional[int] The default materialization version of the datastack to use. If None, the latest version is used. Optional, defaults to None.
None See Also set_session_defaults
get_session_defaults
Methods:
Name Description change_auth Change the authentication token and reset services.
Attributes:
Name Type Description annotation AnnotationClient A client for the annotation service. See client.annotation
auth AuthClient A client for the auth service. See client.auth for more information.
chunkedgraph ChunkedGraphClient A client for the chunkedgraph service. See client.chunkedgraph
datastack_name str The name of the datastack for the client.
info InfoServiceClient A client for the info service. See client.info for more information.
l2cache L2CacheClient A client for the L2 cache service. See client.l2cache
materialize MaterializationClient A client for the materialization service. See client.materialize
schema SchemaClient A client for the EM Annotation Schemas service. See client.schema
server_address The server address for the client.
skeleton SkeletonClient A client for the skeleton service. See client.skeleton
state JSONService A client for the neuroglancer state service. See client.state
timestamp Optional[datetime] The default timestamp to use for queries which rely on a timestamp.
version Optional[int] The default materialization version of the datastack to use for queries which
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.annotation","title":"annotation: AnnotationClient property","text":"A client for the annotation service. See client.annotation for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.auth","title":"auth: AuthClient property","text":"A client for the auth service. See client.auth for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.chunkedgraph","title":"chunkedgraph: ChunkedGraphClient property","text":"A client for the chunkedgraph service. See client.chunkedgraph for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.datastack_name","title":"datastack_name: str property","text":"The name of the datastack for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.info","title":"info: InfoServiceClient property","text":"A client for the info service. See client.info for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.l2cache","title":"l2cache: L2CacheClient property","text":"A client for the L2 cache service. See client.l2cache for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.materialize","title":"materialize: MaterializationClient property","text":"A client for the materialization service. See client.materialize for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.schema","title":"schema: SchemaClient property","text":"A client for the EM Annotation Schemas service. See client.schema for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.server_address","title":"server_address property","text":"The server address for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.skeleton","title":"skeleton: SkeletonClient property","text":"A client for the skeleton service. See client.skeleton for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.state","title":"state: JSONService property","text":"A client for the neuroglancer state service. See client.state for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.timestamp","title":"timestamp: Optional[datetime] property","text":"The default timestamp to use for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.version","title":"version: Optional[int] property writable","text":"The default materialization version of the datastack to use for queries which expect a version. Also sets the timestamp to the corresponding timestamp of the version for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.change_auth","title":"change_auth(auth_token_file=None, auth_token_key=None, auth_token=None)","text":"Change the authentication token and reset services.
Parameters:
Name Type Description Default auth_token_file str New auth token json file path, by default None, which defaults to the existing state.
None auth_token_key str New dictionary key under which the token is stored in the json file, by default None, which defaults to the existing state.
None auth_token str Direct entry of a new token, by default None.
None"},{"location":"api/config/","title":"Configuration","text":""},{"location":"api/config/#caveclient.set_session_defaults","title":"caveclient.set_session_defaults(max_retries=3, pool_block=False, pool_maxsize=10, backoff_factor=0.1, backoff_max=120, status_forcelist=(502, 503, 504))","text":"Set global default values to configure how all clients will communicate with servers. Should be done prior to initializing a client.
Note that these values are only used when not set at the client level.
Parameters:
Name Type Description Default max_retries int The maximum number of retries each connection should attempt. Set to 0 to fail on the first retry.
3 pool_block bool Whether the connection pool should block for connections.
False pool_maxsize int The maximum number of connections to save in the pool.
10 backoff_factor Union[float, int] A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). The query will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. For example, if the backoff_factor is 0.1, then will sleep for [0.0s, 0.2s, 0.4s, 0.8s, \u2026] between retries. No backoff will ever be longer than backoff_max.
0.1 backoff_max Union[float, int] The maximum backoff time.
120 status_forcelist Optional[Collection] A set of integer HTTP status codes that we should force a retry on.
(502, 503, 504) Usage from caveclient import set_session_defaults\n\nset_session_defaults(\n max_retries=5, # would increase the default number of retries\n backoff_factor=0.5, # would increase the default backoff factor between retries\n backoff_max=240, # would increase the default maximum backoff time\n status_forcelist=(502, 503, 504, 505), # would add 505 to the default list\n)\n\nset_session_defaults() # would revert all defaults to their original values\n
Notes Calling this function will set the default values for all clients created after the call.
Calling this function with any arguments missing will reset that value to the default value.
See Also: urllib3.util.Retry
requests.adapters.HTTPAdapter
"},{"location":"api/config/#caveclient.get_session_defaults","title":"caveclient.get_session_defaults()","text":"Get the current default values for session configuration.
Returns:
Type Description dict Dictionary of current default values for session configuration.
"},{"location":"api/datastack_lookup/","title":"Datastack lookup","text":""},{"location":"api/datastack_lookup/#caveclient.datastack_lookup.reset_server_address_cache","title":"caveclient.datastack_lookup.reset_server_address_cache(datastack, filename=None)","text":"Remove one or more datastacks from the datastack-to-server cache.
Parameters:
Name Type Description Default datastack str or list of str Datastack names to remove from the cache, by default None
required filename str Name of the cache file, by default None
None"},{"location":"api/info/","title":"client.info","text":"Client for interacting with the info service.
Methods:
Name Description annotation_endpoint AnnotationEngine endpoint for a dataset.
get_aligned_volume_info Gets the info record for a aligned_volume
get_datastack_info Gets the info record for a datastack
get_datastacks Query which datastacks are available at the info service
get_datastacks_by_aligned_volume Lookup what datastacks are associated with this aligned volume
image_cloudvolume Generate a cloudvolume instance based on the image source, using authentication if needed and
image_source Cloud path to the imagery for the dataset
refresh_stored_data Reload the stored info values from the server.
segmentation_cloudvolume Generate a cloudvolume instance based on the segmentation source, using authentication if needed and
segmentation_source Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
synapse_segmentation_source Cloud path to the synapse segmentation for a dataset
viewer_resolution Get the viewer resolution metadata for this datastack
viewer_site Get the base Neuroglancer URL for the dataset
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.annotation_endpoint","title":"annotation_endpoint(datastack_name=None, use_stored=True)","text":"AnnotationEngine endpoint for a dataset.
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Description str Location of the AnnotationEngine
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_aligned_volume_info","title":"get_aligned_volume_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a aligned_volume
Parameters:
Name Type Description Default datastack_name str datastack_name to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that dataset, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Description dict or None The complete info record for the aligned_volume
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastack_info","title":"get_datastack_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a datastack
Parameters:
Name Type Description Default datastack_name str datastack to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that datastack, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Description dict or None The complete info record for the datastack
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks","title":"get_datastacks()","text":"Query which datastacks are available at the info service
Returns:
Type Description list List of datastack names
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks_by_aligned_volume","title":"get_datastacks_by_aligned_volume(aligned_volume=None)","text":"Lookup what datastacks are associated with this aligned volume
Args: aligned_volume (str, optional): aligned volume to lookup. Defaults to None.
Raises: ValueError: if no aligned volume is specified
Returns: list: a list of datastack string
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_cloudvolume","title":"image_cloudvolume(**kwargs)","text":"Generate a cloudvolume instance based on the image source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_source","title":"image_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the imagery for the dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Description str Formatted cloud path to the flat segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.refresh_stored_data","title":"refresh_stored_data()","text":"Reload the stored info values from the server.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_cloudvolume","title":"segmentation_cloudvolume(use_client_secret=True, **kwargs)","text":"Generate a cloudvolume instance based on the segmentation source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_source","title":"segmentation_source(datastack_name=None, format_for='raw', use_stored=True)","text":"Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"graphene://https://\" type path is used If 'neuroglancer', a \"graphene://https://\" type path is used, as needed by Neuroglancer.
'raw' Returns:
Type Description str Formatted cloud path to the Graphene segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.synapse_segmentation_source","title":"synapse_segmentation_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the synapse segmentation for a dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the dataset to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Description str Formatted cloud path to the synapse segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_resolution","title":"viewer_resolution(datastack_name=None, use_stored=True)","text":"Get the viewer resolution metadata for this datastack
Parameters:
Name Type Description Default datastack_name If None use the default one configured in the client
None use_stored Use the cached value, if False go get a new value from server
True Returns:
Type Description array Voxel resolution as a len(3) np.array
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_site","title":"viewer_site(datastack_name=None, use_stored=True)","text":"Get the base Neuroglancer URL for the dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Description str Base URL for the Neuroglancer viewer
"},{"location":"api/l2cache/","title":"client.l2cache","text":"Client for interacting with the level2 cache service.
Methods:
Name Description cache_metadata Retrieves the meta data for the cache
get_l2data Gets the attributed statistics data for L2 ids.
has_cache Checks if the l2 cache is available for the dataset
table_mapping Retrieves table mappings for l2 cache.
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.cache_metadata","title":"cache_metadata()","text":"Retrieves the meta data for the cache
Returns:
Type Description dict keys are attribute names, values are datatypes
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data","title":"get_l2data(l2_ids, attributes=None)","text":"Gets the attributed statistics data for L2 ids.
Parameters:
Name Type Description Default l2_ids list or ndarray a list of level 2 ids
required attributes list a list of attributes to retrieve. Defaults to None which will return all that are available. Available stats are ['area_nm2', 'chunk_intersect_count', 'max_dt_nm', 'mean_dt_nm', 'pca', 'pca_val', 'rep_coord_nm', 'size_nm3']. See docs for more description.
None Returns:
Type Description dict keys are l2 ids, values are data
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.has_cache","title":"has_cache(datastack_name=None)","text":"Checks if the l2 cache is available for the dataset
Parameters:
Name Type Description Default datastack_name str The name of the datastack to check, by default None (if None, uses the client's datastack)
None Returns:
Type Description bool True if the l2 cache is available, False otherwise
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.table_mapping","title":"table_mapping()","text":"Retrieves table mappings for l2 cache.
Returns:
Type Description dict keys are pcg table names, values are dicts with fields l2cache_id and cv_path.
"},{"location":"api/materialize/","title":"client.materialize","text":"Client for interacting with the materialization engine.
Methods:
Name Description get_table_metadata Get metadata about a table
get_tables Gets a list of table names for a datastack
get_tables_metadata Get metadata about tables
get_timestamp Get datetime.datetime timestamp for a materialization version.
get_unique_string_values Get unique string values for a table
get_version_metadata Get metadata about a version
get_versions Get the versions available
get_versions_metadata Get the metadata for all the versions that are presently available and valid
get_view_metadata Get metadata for a view
get_view_schema Get schema for a view
get_view_schemas Get schema for a view
get_views Get all available views for a version
ingest_annotation_table Trigger supervoxel lookup and root ID lookup of new annotations in a table.
join_query Generic query on materialization tables
live_live_query Beta method for querying cave annotation tables with root IDs and annotations
live_query Generic query on materialization tables
lookup_supervoxel_ids Trigger supervoxel lookups of new annotations in a table.
map_filters Translate a list of filter dictionaries from a point in the
most_recent_version Get the most recent version of materialization for this datastack name
query_table Generic query on materialization tables
query_view Generic query on a view
synapse_query Convenience method for querying synapses.
Attributes:
Name Type Description cg_client The chunked graph client.
datastack_name The name of the datastack.
homepage HTML The homepage for the materialization engine.
server_version Optional[Version] The version of the service running on the remote server. Note that this
tables TableManager The table manager for the materialization engine.
version int The version of the materialization. Can be used to set up the
views ViewManager The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.cg_client","title":"cg_client property","text":"The chunked graph client.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.datastack_name","title":"datastack_name property","text":"The name of the datastack.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.homepage","title":"homepage: HTML property","text":"The homepage for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.tables","title":"tables: TableManager property","text":"The table manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.version","title":"version: int property writable","text":"The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.
Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.views","title":"views: ViewManager property","text":"The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_table_metadata","title":"get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)","text":"Get metadata about a table
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required datastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Description dict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables","title":"get_tables(datastack_name=None, version=None)","text":"Gets a list of table names for a datastack
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None Returns:
Type Description list List of table names
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables_metadata","title":"get_tables_metadata(datastack_name=None, version=None, log_warning=True)","text":"Get metadata about tables
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Description dict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_timestamp","title":"get_timestamp(version=None, datastack_name=None)","text":"Get datetime.datetime timestamp for a materialization version.
Parameters:
Name Type Description Default version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Description datetime Datetime when the materialization version was frozen.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_unique_string_values","title":"get_unique_string_values(table, datastack_name=None)","text":"Get unique string values for a table
Parameters:
Name Type Description Default table str Table to query
required datastack_name Optional[str] Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Description dict[str] A dictionary of column names and their unique values
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_version_metadata","title":"get_version_metadata(version=None, datastack_name=None)","text":"Get metadata about a version
Parameters:
Name Type Description Default version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Description dict Dictionary of metadata about the version
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions","title":"get_versions(datastack_name=None, expired=False)","text":"Get the versions available
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Description dict Dictionary of versions available
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions_metadata","title":"get_versions_metadata(datastack_name=None, expired=False)","text":"Get the metadata for all the versions that are presently available and valid
Parameters:
Name Type Description Default datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Description list[dict] List of metadata dictionaries
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_metadata","title":"get_view_metadata(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get metadata for a view
Parameters:
Name Type Description Default view_name str Name of view to query.
required materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Description dict Metadata of view
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schema","title":"get_view_schema(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Default view_name str Name of view to query.
required materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Description dict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schemas","title":"get_view_schemas(materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Default materialization_version Optional[int] Version to query. If None, will use version set by client.
None log_warning bool Whether to log warnings.
True Returns:
Type Description dict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_views","title":"get_views(version=None, datastack_name=None)","text":"Get all available views for a version
Parameters:
Name Type Description Default version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Description list List of views
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.ingest_annotation_table","title":"ingest_annotation_table(table_name, datastack_name=None)","text":"Trigger supervoxel lookup and root ID lookup of new annotations in a table.
Parameters:
Name Type Description Default table_name str Table to trigger
required datastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Description dict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.join_query","title":"join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Default tables list of lists with length 2 or 'str' list of two lists: first entries are table names, second entries are the columns used for the join.
required filter_in_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None
None filter_out_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None
None filter_greater_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict of dicts outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None.
None filter_regex_dict dict of dicts outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None
None select_columns dict of lists of str keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated.
None offset int result offset to use. Defaults to None. Will only return top K results.
None limit int maximum results to return (server will set upper limit, see get_server_config)
None suffixes dict suffixes to use for duplicate columns, keys are table names, values are the suffix
None datastack_name str datastack to query. If None defaults to one specified in client.
None return_df bool whether to return as a dataframe default True, if False, data is returned as json (slower)
True split_positions bool whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True desired_resolution Iterable What resolution to convert position columns to. Defaults to None will use defaults.
None random_sample int if given, will do a tablesample of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type Description DataFrame a pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_live_query","title":"live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, allow_invalid_root_ids=False, random_sample=None, log_warning=True)","text":"Beta method for querying cave annotation tables with root IDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.
Parameters:
Name Type Description Default table str Principle table to query
required timestamp datetime Timestamp to query
required joins List of joins, where each join is a list of [table1,column1, table2, column2]
None filter_in_dict A dictionary with tables as keys, values are dicts with column keys and list values to accept.
None filter_out_dict A dictionary with tables as keys, values are dicts with column keys and list values to reject.
None filter_equal_dict A dictionary with tables as keys, values are dicts with column keys and values to equate.
None filter_greater_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive upper-bound.
None filter_less_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive lower-bound.
None filter_greater_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive upper-bound.
None filter_less_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive lower-bound.
None filter_spatial_dict A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds.
None filter_regex_dict A dictionary with tables as keys, values are dicts with column keys and values of regex strings.
None select_columns A dictionary with tables as keys, values are lists of columns to select.
None offset int Value to offset query by.
None limit int Limit of query.
None datastack_name str Datastack to query. Defaults to set by client.
None split_positions bool Whether to split positions into separate columns, True is faster.
False metadata bool Whether to attach metadata to dataframe.
True suffixes dict What suffixes to use on joins, keys are table_names, values are suffixes.
None desired_resolution Iterable What resolution to convert position columns to.
None allow_missing_lookups bool If there are annotations without supervoxels and root IDs yet, allow results.
False allow_invalid_root_ids bool If True, ignore root ids not valid at the given timestamp, otherwise raise an error.
False random_sample int If given, will do a table sample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type Description Results of query
Examples:
>>> from caveclient import CAVEclient\n>>> client = CAVEclient('minnie65_public_v117')\n>>> live_live_query(\"table_name\", datetime.datetime.now(datetime.timezone.utc),\n>>> joins=[[table_name, table_column, joined_table, joined_column],\n>>> [joined_table, joincol2, third_table, joincol_third]]\n>>> suffixes={\n>>> \"table_name\":\"suffix1\",\n>>> \"joined_table\":\"suffix2\",\n>>> \"third_table\":\"suffix3\"\n>>> },\n>>> select_columns= {\n>>> \"table_name\":[ \"column\",\"names\"],\n>>> \"joined_table\":[\"joined_colum\"]\n>>> },\n>>> filter_in_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[included,values]\n>>> }\n>>> },\n>>> filter_out_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[excluded,values]\n>>> }\n>>> },\n>>> filter_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_spatial_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": [[min_x, min_y, min_z], [max_x, max_y, max_z]]\n>>> }\n>>> filter_regex_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": \"regex_string\"\n>>> }\n
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_query","title":"live_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Default table str Table to query
required timestamp datetime Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time.
required filter_in_dict dict Keys are column names, values are allowed entries.
None filter_out_dict dict Keys are column names, values are not allowed entries.
None filter_equal_dict dict Keys are column names, values are specified entry.
None filter_greater_dict dict Keys are column names, values are exclusive upper-bounds.
None filter_less_dict dict Keys are column names, values are exclusive lower-bounds.
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bounds.
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bounds.
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]].
None filter_regex_dict dict Keys are column names, values are regex strings.
None select_columns list of str Columns to select.
None offset int Offset in query result.
None limit int Maximum results to return (server will set upper limit, see get_server_config).
None datastack_name str Datastack to query. If None, defaults to one specified in client.
None split_positions bool Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower).
False post_filter bool Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True.
True metadata bool Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table.
True desired_resolution Iterable Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata.
None random_sample int If given, will do a tablesample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type Description DataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.lookup_supervoxel_ids","title":"lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)","text":"Trigger supervoxel lookups of new annotations in a table.
Parameters:
Name Type Description Default table_name str Table to trigger
required annotation_ids list List of annotation ids to lookup. Default is None, which will trigger lookup of entire table.
None datastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Description dict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.map_filters","title":"map_filters(filters, timestamp, timestamp_past)","text":"Translate a list of filter dictionaries from a point in the future to a point in the past
Parameters:
Name Type Description Default filters list[dict] filter dictionaries with root_ids
required timestamp datetime timestamp to query
required timestamp_past datetime timestamp to query from
required Returns:
Type Description list[dict] filter dictionaries with past root_ids
dict mapping of future root_ids to past root_ids
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.most_recent_version","title":"most_recent_version(datastack_name=None)","text":"Get the most recent version of materialization for this datastack name
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Description int Most recent version of materialization for this datastack name
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_table","title":"query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Default table str Table to query
required filter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound value, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound value, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound value, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound value, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None timestamp datetime Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type Description DataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_view","title":"query_view(view_name, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None)","text":"Generic query on a view
Parameters:
Name Type Description Default view_name str View to query
required filter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None Returns:
Type Description DataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.synapse_query","title":"synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)","text":"Convenience method for querying synapses.
Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.
Parameters:
Name Type Description Default pre_ids Union[int, Iterable, ndarray] Pre-synaptic cell(s) to query.
None post_ids Union[int, Iterable, ndarray] Post-synaptic cell(s) to query.
None bounding_box Optional[Union[list, ndarray]] [[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset.
None bounding_box_column str Which synapse location column to filter by.
'post_pt_position' timestamp datetime Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version.
None remove_autapses bool Whether to remove autapses from query results.
True include_zeros bool Whether to include synapses to/from id=0 (out of segmentation).
True limit int Number of synapses to limit. Server-side limit still applies.
None offset int Number of synapses to offset query.
None split_positions bool Whether to split positions into separate columns, True is faster.
False desired_resolution Iterable[float] List or array of the desired resolution you want queries returned in useful for materialization queries.
None materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Whether to attach metadata to dataframe in the df.attr dictionary.
True Returns:
Type Description DataFrame Results of query.
"},{"location":"api/schema/","title":"client.schema","text":"Client for interacting with the schema service.
Methods:
Name Description get_schemas Get the available schema types
schema_definition Get the definition of a specified schema_type
schema_definition_all Get the definition of all schema_types
schema_definition_multi Get the definition of multiple schema_types
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.get_schemas","title":"get_schemas()","text":"Get the available schema types
Returns:
Type Description list List of schema types available on the Schema service.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition","title":"schema_definition(schema_type)","text":"Get the definition of a specified schema_type
Parameters:
Name Type Description Default schema_type str Name of a schema_type
required Returns:
Type Description json Schema definition
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_all","title":"schema_definition_all()","text":"Get the definition of all schema_types
Returns:
Type Description dict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_multi","title":"schema_definition_multi(schema_types)","text":"Get the definition of multiple schema_types
Parameters:
Name Type Description Default schema_types list List of schema names
required Returns:
Type Description dict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/skeleton/","title":"client.skeleton","text":"Client for interacting with the skeleton service.
Methods:
Name Description get_skeleton Gets basic skeleton information for a datastack
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_skeleton","title":"get_skeleton(root_id, datastack_name=None, skeleton_version=0, output_format='none', log_warning=True)","text":"Gets basic skeleton information for a datastack
Parameters:
Name Type Description Default root_id int The root id of the skeleton to retrieve
required datastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate and retrieve. Options are documented in SkeletonService. Use 0 for latest.
0 output_format string The format to retrieve. Options are:
- 'none': No return value (this can be used to generate a skeleton without retrieving it)
- 'precomputed': A cloudvolume.Skeleton object
- 'json': A dictionary
- 'jsoncompressed': A dictionary using compression for transmission (generally faster than 'json')
- 'arrays': A dictionary (literally a subset of the json response)
- 'arrayscompressed': A dictionary using compression for transmission (generally faster than 'arrays')
- 'swc': A pandas DataFrame
- 'h5': An BytesIO object containing bytes for an h5 file
'none' Returns:
Type Description Skeleton of the requested type. See output_format for details.
"},{"location":"api/state/","title":"client.state","text":"Client to interface with the JSON state service.
Methods:
Name Description build_neuroglancer_url Build a URL for a Neuroglancer deployment that will automatically retrieve specified state.
get_neuroglancer_info Get the info field from a Neuroglancer deployment
get_property_json Download a Neuroglancer JSON state
get_state_json Download a Neuroglancer JSON state
save_state_json_local Save a Neuroglancer JSON state to a JSON file locally.
upload_property_json Upload a Neuroglancer JSON state
upload_state_json Upload a Neuroglancer JSON state
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
state_service_endpoint Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.state_service_endpoint","title":"state_service_endpoint property","text":"Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.build_neuroglancer_url","title":"build_neuroglancer_url(state_id, ngl_url=None, target_site=None, static_url=False, format_properties=False)","text":"Build a URL for a Neuroglancer deployment that will automatically retrieve specified state. If the datastack is specified, this is prepopulated from the info file field \"viewer_site\". If no ngl_url is specified in either the function or the client, a fallback neuroglancer deployment is used.
Parameters:
Name Type Description Default state_id int State id to retrieve
required ngl_url str Base url of a neuroglancer deployment. If None, defaults to the value for the datastack or the client. As a fallback, a default deployment is used.
None target_site seunglab or cave - explorer or mainline or None Set this to 'seunglab' for a seunglab deployment, or either 'cave-explorer'/'mainline' for a google main branch deployment. If None, checks the info field of the neuroglancer endpoint to determine which to use. Default is None.
None static_url bool If True, treats \"state_id\" as a static URL directly to the JSON and does not use the state service.
False format_properties bool If True, formats the url as a segment_properties info file
False Returns:
Type Description str The full URL requested
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_neuroglancer_info","title":"get_neuroglancer_info(ngl_url=None)","text":"Get the info field from a Neuroglancer deployment
Parameters:
Name Type Description Default ngl_url str(optional) URL to a Neuroglancer deployment. If None, defaults to the value for the datastack or the client.
None Returns:
Type Description dict JSON-formatted info field from the Neuroglancer deployment
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_property_json","title":"get_property_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Default state_id int ID of a JSON state uploaded to the state service.
required Returns:
Type Description dict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_state_json","title":"get_state_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Default state_id int ID of a JSON state uploaded to the state service.
required Returns:
Type Description dict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.save_state_json_local","title":"save_state_json_local(json_state, filename, overwrite=False)","text":"Save a Neuroglancer JSON state to a JSON file locally.
Parameters:
Name Type Description Default json_state dict Dict representation of a neuroglancer state
required filename str Filename to save the state to
required overwrite bool Whether to overwrite the file if it exists. Default False.
False Returns:
Type Description None"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_property_json","title":"upload_property_json(property_json, state_id=None, max_size=2500000)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Default propery_json dict Dict representation of a neuroglancer segment properties json
required state_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None max_size Optional[int] Maximum size in bytes for the data to upload. Default is 2.5MB. Set to None for no limit.
2500000 Returns:
Type Description int state_id of the uploaded JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_state_json","title":"upload_state_json(json_state, state_id=None, timestamp=None)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Default json_state dict Dict representation of a neuroglancer state
required state_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None timestamp Timestamp for json state date. Requires state_id.
None Returns:
Type Description int state_id of the uploaded JSON state
"},{"location":"tutorials/","title":"Getting Started","text":"AnnotationFramework client is a package for simplifying interactions with HTML services associated with the CAVE (Connectome Annotation Versioning Engine).
For a larger introduction to CAVE and its services, see the main GitHub organization site: https://github.com/CAVEconnectome
"},{"location":"tutorials/#installation","title":"Installation","text":"The CAVEclient can be installed with pip:
$ pip install caveclient\n
"},{"location":"tutorials/#assumptions","title":"Assumptions","text":"The code is setup to work flexibly with any deployment of these services, but you need to specify the server_address if that address is not https://globalv1.daf-apis.com/ for each client when initializing it. Similarly, the clients can query the info service for metadata to simplify the interaction with a datastack, but you have to specify a datastack name.
"},{"location":"tutorials/advanced/","title":"Advanced Usage","text":""},{"location":"tutorials/advanced/#changing-session-configurations","title":"Changing session configurations","text":"It is possible to change details of how a client talks to the various servers it needs to interface with. For instance, the CAVEclient will attempt to retry specific failed requests to the server, but will only try a specific number of times, and will only wait specific amounts of time between retries. These values can be changed via the set_session_defaults method. For instance, to change the number of retries to 5, and to increase the delay between subsequent retries, you could do:
from caveclient import set_session_defaults\n\nset_session_defaults(max_retries=5, backoff_factor=0.5)\n
Note that this needs to happen before initializing the client for this to work properly. Some of these parameters are also adjustable at the client level.
To view the current session defaults, you can use the get_session_defaults method:
from caveclient import get_session_defaults\n\nclient.get_session_defaults()\n
More information on the available parameters can be found in the API documentation.
"},{"location":"tutorials/annotation/","title":"Annotation","text":"The AnnotationClient is used to interact with the AnnotationEngine service to create tables from existing schema, upload new data, and download existing annotations. Note that annotations in the AnnotationEngine are not linked to any particular segmentation, and thus do not include any root ids. An annotation client is accessed with client.annotation.
"},{"location":"tutorials/annotation/#getting-existing-tables","title":"Getting existing tables","text":"A list of the existing tables for the datastack can be found with get_tables().
all_tables = client.annotation.get_tables()\nall_tables[0]\n
Each table has three main properties that can be useful to know:
table_name : The table name, used to refer to it when uploading or downloading annotations. This is also passed through to the table in the Materialized database.schema_name : The name of the table's schema from EMAnnotationSchemas (see below).max_annotation_id : An upper limit on the number of annotations already contained in the table.
"},{"location":"tutorials/annotation/#downloading-annotations","title":"Downloading annotations","text":"You can download the JSON representation of a data point through the get_annotation() method. This can be useful if you need to look up information on unmaterialized data, or to see what a properly templated annotation looks like.
table_name = all_tables[0]['table_name'] # 'ais_analysis_soma'\nannotation_id = 100\nclient.annotation.get_annotation(annotation_ids=annotation_id, table_name=table_name)\n
"},{"location":"tutorials/annotation/#create-a-new-table","title":"Create a new table","text":"One can create a new table with a specified schema with the create_table() method:
client.annotation.create_table(table_name='test_table',\n schema_name='microns_func_coreg',\n voxel_resolution = [1,1,1],\n description=\"some text to describe your table\")\n
The voxel resolution is the units your position columns will be uploaded in [1,1,1] would imply a nm location, where as [4,4,40] would correspond to voxels of that size. If you are uploading points from a neuroglancer session, you want this to match the units of that neuroglancer view.
Note there are some optional metadata parameters to create_table()
notice_text : This is text that will show up to users who access this data as a warning. This could be used to warn users that the data is not complete or checked yet, or to advertise that a particular publication should be cited when using this table.read_permission : one of \"PRIVATE\" which means only you can read data in this table. \"PUBLIC\" (default) which means anyone can read this table that has read permissions to this dataset. So if and only if you can read the segmentation results of this data, you can read this table. \"GROUP\" which means that you must share a common group with this user for them to be able to read. We need to make a way to discover what groups you are in and who you share groups with.write_permission: one of \"PRIVATE\" (default), which means only you can write to this table. \"PUBLIC\" which means anyone can write to this table that has write permissions to this dataset. Note although this means anyone can add data, no annotations are ever truly overwritten. \"GROUP\" which means that you must share a common group with this user for them to be able to write. We need to make a way to discover what groups you are in and who you share groups with.
If you change your mind about what you want for metadata, some but not all fields can be updated with update_metadata(). This includes the description, the notice_text, and the permissions, but not the name, schema or voxel resolution.
# to update description\nclient.annotation.update_metadata(table_name='test_table',\n description=\"a new description for my table\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n notice_text=\"This table isn't done yet, don't trust it. Contact me\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n read_permisison=\"PUBLIC\")\n
New data can be generated as a dict or list of dicts following the schema and uploaded with post_annotation. For example, a microns_func_coreg point needs to have:
type set to microns_func_coregpt set to a dict with position as a key and the xyz location as a value.func_id set to an integer.
The following could would create a new annotation and then upload it to the service. Note that you get back the annotation id(s) of what you uploaded.
new_data = {'type': 'microns_func_coreg',\n 'pt': {'position': [1,2,3]},\n 'func_id': 0}\nclient.annotation.post_annotation(table_name='test_table', data=[new_data])\n
There are methods to simplify annotation uploads if you have a pandas dataframe whose structure mirrors the struction of the annotation schema you want to upload
import pandas as pd\n\ndf = pd.DataFrame([{'id':0,\n 'type': 'microns_func_coreg',\n 'pt_position': [1,2,3]},\n 'func_id': 0},\n {'id':1,\n 'type': 'microns_func_coreg',\n 'pt_position': [3,2,1]},\n 'func_id': 2}])\nclient.annotation.post_annotation_df('test_table', df)\n
Note that here I specified the IDs of my annotations, which you can do, but then its up to you to assure that the IDs don't collide with other IDs. If you leave them blank then the service will assign the IDs for you.
There is a similar method for updating update_annotation_df()
"},{"location":"tutorials/annotation/#staged-annotations","title":"Staged Annotations","text":"Staged annotations help ensure that the annotations you post follow the appropriate schema, both by providing guides to the field names and locally validating against a schema before uploading. The most common use case for staged annotations is to create a StagedAnnotation object for a given table, then add annotations to it individually or as a group, and finally upload to the annotation table.
To get a StagedAnnotation object, you can start with either a table name or a schema name. Here, we'll assume that there's already a table called \"my_table\" that is running a \"cell_type_local\" schema. If we want to add new annotations to the table, we simply use the table name with stage_annotations().
stage = client.annotation.stage_annotations(\"my_table\")\n
This stage object retrieves the schema for the table and hosts a local collection of annotations. Every time you add an annotation, it is immediately validated against the schema. To add an annotation, use the add method:
stage.add(\n cell_type = \"pyramidal_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\n
The argument names derive from fields in the schema and you must provide all required fields. Any number of annotations can be added to the stage. A dataframe of annotations can also be added with stage.add_dataframe, and requires an exact match between column names and schema fields. The key difference between this and posting a dataframe directly is that annotations added to a StagedAnnotations are validated locally, allowing any issues to be caught before uploading.
You can see the annotations as a list of dictionary records with stage.annotation_list or as a Pandas dataframe with stage.annotation_dataframe. Finally, if you initialized the stage with a table name, this information is stored in the stage and you can simply upload it from the client.
client.annotation.upload_staged_annotations(stage)\n
Updating annotations requires knowing the annotation id of the annotation you are updating, which is not required in the schema otherwise. In order to stage updated annotations, set the update parameter to True when creating the stage.
update_stage = client.annotation.stage_annotations(\"my_table\", update=True)\nupdate_stage.add(\n id=1,\n cell_type = \"stellate_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\n
The update also informs the framework client to treat the annotations as an update and it will use the appropriate methods automatically when uploading client.annotation.upload_staged_annotations.
If you want to specify ids when posting new annotations, id_field can be set to True when creating the StagedAnnotation object. This will enforce an id column but still post the data as new annotations.
If you might be adding spatial data in coordinates that might be different than the resolution for the table, you can also set the annotation_resolution when creating the stage. The stage will convert between the resolution you specify for your own annotations and the resolution that the table expects.
stage = client.annotation.stage_annotations(\"my_table\", annotation_resolution=[8,8,40])\nstage.add(\n cell_type='pyramidal_cell',\n classification_system=\"excitatory\",\n pt_position=[50,50,10],\n)\n
"},{"location":"tutorials/authentication/","title":"Authentication","text":"Authentication tokens are generally needed for programmatic access to our services. The AuthClient handles storing and loading your token or tokens and inserting it into requests in other clients.
We can access the auth client from client.auth. Once you have saved a token, you probably won't interact with this client very often, however it has some convenient features for saving new tokens the first time. Let's see if you have a token already. Probably not.
client = CAVEclient()\nauth = client.auth\nprint(f\"My current token is: {auth.token}\")\n
"},{"location":"tutorials/authentication/#getting-a-new-token","title":"Getting a new token","text":"To get a new token, you will need to manually acquire it. For convenience, the function client.auth.get_new_token() provides instructions for how to get and save the token.
By default, the token is saved to ~/.cloudvolume/secrets/cave-secret.json as a string under the key token. This makes it compatible by default with Cloudvolume projects, which can come in handy. The following steps will save a token to the default location.
auth.get_new_token()\n
new_token = 'abcdef1234567890' #This is the text you see after you visit the website.\nauth.save_token(token=new_token)\nprint(f\"My token is now: {auth.token}\")\n
Note that requesting a new token will invalidate your previous token on the same project. If you want to use the same token across different computers, you will need to share the same token information.
"},{"location":"tutorials/authentication/#loading-saved-tokens","title":"Loading saved tokens","text":"Try opening ~/.cloudvolume/secrets/cave-secret.json to see what we just created.
If we had wanted to use a different file or a different json key, we could have specified that in auth.save_token.
Because we used the default values, this token is used automatically when we initialize a new CAVEclient. If we wanted to use a different token file, token key, or even directly specify a token we could do so here.
client = CAVEclient(datastack_name)\nprint(f\"Now my basic token is: {client.auth.token}\")\n\nclient_direct = CAVEclient(datastack_name, auth_token='another_fake_token_678')\nprint(f\"A directly specified token is: {client_direct.auth.token}\")\n
If you use a CAVEclient, the AuthClient and its token will be automatically applied to any other services without further use.
"},{"location":"tutorials/chunkedgraph/","title":"Chunked Graph","text":"The chunkedgraph is a dynamic oct-tree connected components supervoxel graph.
A visual representation of an oct-tree (Wikipedia (WhiteTimberwolf) CC BY-SA 3.0)
As with any oct-tree, it is organized in hierarchical levels, with the bottom level 1 corresponding to the supervoxels of the segmentations, and the top level being the unique connected components of the supervoxel graph.
A figure illustrating the spatial chunking and editing of a the chunkedgraph. From Dorkenwald et. al 2021
The ChunkedGraph client allows one to interact with the ChunkedGraph service, which stores and updates the supervoxel agglomeration graph. This is most often useful for looking up an object root id of a supervoxel or looking up supervoxels belonging to a root id. The ChunkedGraph client is at client.chunkedgraph.
"},{"location":"tutorials/chunkedgraph/#look-up-a-supervoxel","title":"Look up a supervoxel","text":"Usually in Neuroglancer, one never notices supervoxel ids, but they are important for programmatic work. In order to look up the root id for a location in space, one needs to use the supervoxel segmentation to get the associated supervoxel id. The ChunkedGraph client makes this easy using the client.chunkedgraph.get_root_id() method.
sv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id)\n
However, as proofreading occurs, the root id that a supervoxel belongs to can change. By default, this function returns the current state, however one can also provide a UTC timestamp to get the root id at a particular moment in history. This can be useful for reproducible analysis. Note below that the root id for the same supervoxel is different than it is now.
import datetime\n\n# I looked up the UTC POSIX timestamp from a day in early 2019.\ntimestamp = datetime.datetime.utcfromtimestamp(1546595253)\n\nsv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id, timestamp=timestamp)\n
If you are doing this across lots of supervoxels (or any nodes) then you can do it more efficiently in one request with client.chunkedgraph.get_roots()
node_ids = [104200755619042523, 104200755619042524,104200755619042525]\nroot_ids = client.chunkedgraph.get_roots(node_ids)\n
"},{"location":"tutorials/chunkedgraph/#getting-supervoxels-for-a-root-id","title":"Getting supervoxels for a root id","text":"A root id is associated with a particular agglomeration of supervoxels, which can be found with the client.chunkedgraph.get_leaves() method. A new root id is generated for every new change in the chunkedgraph, so time stamps do not apply.
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id)\n
You can also query the chunkedgraph not all the way to the bottom, using the stop_layer option
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id,stop_layer=2)\n
This will get all the level 2 IDs for this root, which correspond to the lowest chunk of the hierarchy. An analogous option exists for client.chunkedgraph.get_roots(). This is useful to help find nodes to query within the l2cache, amongst other things.
"},{"location":"tutorials/chunkedgraph/#other-functions","title":"Other functions","text":"There are a variety of other interesting functions to explore in client.chunkedgraph.
"},{"location":"tutorials/framework/","title":"Introduction","text":""},{"location":"tutorials/framework/#caveclient-one-client-for-all-services","title":"CAVEclient: one client for all services","text":"The CAVE Framework consists of a number of different services, each with a specific set of tasks that it can perform through REST endpoints. The CAVEclient is designed to ease programmatic interaction with all of the various endpoints. In addition, most programmatic access requires the use of authentication tokens. In order to collect a given server, datastack name, and user token together into a coherent package that can be used on multiple endpoints, the CAVEclient builds appropriately configured clients for each of the specific services. Each of the individual services has their own specific documentation as well.
"},{"location":"tutorials/framework/#global-and-local-services","title":"Global and Local Services","text":"There are two categories of data in CAVE: Global and local. Local services are associated with a single so-called datastack, which refers to a precise collection of imagery and segmentation data that function together. For example, EM imagery and a specific pychunkedgraph segmentation would be one datastack, while the same EM imagery but an initial static segmentation would be another. Datastacks are referred to by a short name, for instance pinky100_public_flat_v185.
Global services are those that are potentially shared across multiple different specific datastacks. These include the info service, which can describe the properties of all available datastacks, the authentication service, and the state service that hosts neuroglancer states. Global services are associated with a particular URL (by default http://globalv1.daf-apis.com), but not a single datastack.
"},{"location":"tutorials/framework/#initializing-a-caveclient","title":"Initializing a CAVEclient","text":"Assuming that the services are on http://globalv1.daf-apis.com and authentication tokens are either not being used or set up with default values (see Authentication), a simple CAVEclient that can only access global services can be initialized:
from caveclient import CAVEclient\n\nclient = CAVEclient()\n
Just to confirm that this works, let's see if we can get the EM image source from the InfoService. If you get a list of names of datastacks, all is good. If you have not yet set up an authentication token or you get an authentication error, look at Getting a new token for information about how to set up your auth token.
client.info.get_datastacks()\n
If you have a specific datastack you want to use, you can initialize your CAVEclient with it. This gives you access to the full range of client functions.
client = CAVEclient(datastack_name='my_datastack')\n
"},{"location":"tutorials/framework/#using-other-server-addresses","title":"Using Other Server Addresses","text":"If your data is hosted by a different global server, you specify its address when initializing the client.
client = CAVEclient(datastack_name='my_datastack', server_address='http://global.myserver.com')\n
By default, if you pass both a server address and a datastack, the client will store the mapping from datastack to server address in the same location as the default for authentication tokens. Once stored, the client will automatically use the correct server address for the datastack if none is provided. You can override storing the server address by passing write_server_address=False. Datastacks can be removed from the cache using
caveclient.datastack_lookup.reset_server_address_cache(datastack_name).
"},{"location":"tutorials/framework/#accessing-specific-clients","title":"Accessing specific clients","text":"Each client can be accessed as a property of the main client. See the documentation at left for the capabilities of each. Assuming your client is named client, the subclients for each service are:
- Authentication Service :
client.auth - AnnotationEngine :
client.annotation - PyChunkedGraph :
client.chunkedgraph - InfoService :
client.info - EM Annotation Schemas :
client.schemas - JSON Neuroglancer State Service :
client.state - Skeleton Service :
client.skeletonservice
"},{"location":"tutorials/info/","title":"Info Service","text":"A datastack has a number of complex paths to various data sources that together comprise a datastack. Rather than hardcode these paths, the InfoService allows one to query the location of each data source. This is also convenient in case data sources change.
An InfoClient is accessed at client.info.
client = CAVEclient(datastack_name)\nprint(f\"This is an info client for {client.info.datastack_name} on {client.info.server_address}\")\n
"},{"location":"tutorials/info/#accessing-datastack-information","title":"Accessing datastack information","text":"All of the information accessible for the datastack can be seen as a dict using get_datastack_info().
info.get_datastack_info()\n
Individual entries can be found as well. Use tab autocomplete to see the various possibilities.
info.graphene_source()\n
"},{"location":"tutorials/info/#adjusting-formatting","title":"Adjusting formatting","text":"Because of the way neuroglancer looks up data versus cloudvolume, sometimes one needs to convert between gs:// style paths to https://storage.googleapis.com/ stype paths. All of the path sources in the info client accept a format_for argument that can handle this, and correctly adapts to graphene vs precomputed data sources.
neuroglancer_style_source = info.image_source(format_for='neuroglancer')\nprint(f\"With gs-style: { neuroglancer_style_source }\")\n\ncloudvolume_style_source = info.image_source(format_for='cloudvolume')\nprint(f\"With https-style: { cloudvolume_style_source }\")\n
"},{"location":"tutorials/l2cache/","title":"Level 2 Cache","text":"To understand the level 2 cache, you must understand the structure of the chunkedgraph so see the chunkedgraph tutorial.
Nodes on the second level or layer of the graph, corresponds to all the supervoxels that are locally connected to one another within a single level 2 spatial \"chunk\" of the data. The Level 2 Cache, is a service whose job it is to track and update relevant statistics about every level 2 node within the a chunkedgraph. The source code of this service can be found here.
"},{"location":"tutorials/l2cache/#finding-level-2-nodes","title":"Finding Level 2 Nodes","text":"The chunkedgraph can be used to find the level2 nodes of a rootID using a stop_layer=2 keyword argument on the client.chunkedgraph.get_leaves(). Conversely the level 2 node of a supervoxel can be found using the same keyword argument of client.chunkedgraph.get_roots(). Note if you don't specify a timestamp it will give you the level2 node that is presently associated with the object.
"},{"location":"tutorials/l2cache/#statistics","title":"Statistics","text":"The statistics that are available are:
- area_nm2: The surface area of the object in square nanometers. Does not include border touching voxels
- size_nm3: The volume of the object in cubic nanometers, based on counting voxels in the object.
- max_dt_nm: The maximum edge distance transform of that object in nanometers. Meant to capture the maximum \"thickness\" of the voxels in the node.
- mean_dt_nm: The average edge distance transform of that object in nanometers. Meant to capture the average \"thickness\" of voxels in that node.
- rep_coord_nm: A list of x,y,z coordinates in nanometers that represent a point within the object that is designed to be close to the \"center\" of the object. This is the location of the max_dt_nm value.
- chunk_intersect_count: A 2 x 3 matrix representing the 6 sides of the chunk, and whose values represent how many voxels border that side of the chunk. Meant to help understand significant the borders with other chunks are. Ordering is the [[x_bottom, y_bottom, z_bottom],[x_top, y_top, z_top]] where {xyz}_bottom refers to the face which has the smallest values for that dimension, and {xyz}_top refers to the face which has the largest.
- pca A 3x3 matrix representing the principal components of the xyz point cloud of voxels for this object. Ordering is NxD where N is the components and D are the xyz dimensions. Meant to help desribe the orientation of the level 2 chunk. Note that this is not calculated for very small objects and so might not be present for all level 2 nodes. You will see that its availability correlates strongly with size_nm3.
- pca_val The 3 principal component values for the PCA components.
"},{"location":"tutorials/l2cache/#retrieving-level-2-statistics","title":"Retrieving Level 2 Statistics","text":"Level 2 stats about nodes can be retreived using the client.l2cache.get_l2data() method. It simply takes a list of level 2 nodes you want to retrieve. Optionally you can specify only the attributes that you are interested in retrieving which will speed up the request.
"},{"location":"tutorials/l2cache/#missing-data","title":"Missing Data","text":"The service is constantly watching for changes made to objects and recalculating stats on new level2 nodes that are created, in order to keep its database of statistics current. This however takes some time, and is subject to sporadic rare failures. If you request stats on a level 2 node which are not in the database, you will receive an empty dictionary for that node. This will immediately trigger the system to recalculate the statistics of that missing data, and so it should be available shortly (on the order of seconds) if systems are operational. Please note that PCA is not calculated for very small objects because it is not meaningful. So if you are interested in differentiating whether PCA is not available because it hasn't been calculated, vs when its not available because it is not possible to calculate, you should ask for at least one other non PCA statistic as well. You will see that its availability correlates strongly with size_nm3.
"},{"location":"tutorials/l2cache/#use-cases","title":"Use Cases","text":""},{"location":"tutorials/l2cache/#calculate-total-area-and-volume-of-cells","title":"Calculate Total Area and Volume of Cells","text":"Say you want to calculate the total surface area and volume of a object in the dataset. The areas and volume of each component can simply be added together to do this.
import pandas as pd\nroot_id = 648518346349541252\nlvl2nodes = client.chunkedgraph.get_leaves(root_id,stop_layer=2)\nl2stats = client.l2cache.get_l2data(lvl2nodes, attributes=['size_nm3','area_nm2'])\nl2df = pd.DataFrame(l2stats).T\ntotal_area_um2=l2df.area_nm2.sum()/(1000*1000)\ntotal_volume_um3 = l2df.size_nm3.sum()/(1000*1000*1000)\n
By utilizing the bounds argument of get_leaves, you can also do simple spatially restricted analysis of objects. In fact, because you have data on each level2 node individually, you can segregate the neuron using any labelling of its topology.
"},{"location":"tutorials/l2cache/#skeletonization","title":"Skeletonization","text":"Level 2 nodes have \"cross chunk\" edges within the chunkedgraph which represent what level 2 nodes that object is locally connected to. This forms a graph between the level 2 nodes of the object that can be retrieved using the chunkedgraph function client.chunkedgraph. This graph represents a topological representation of the neuron at the resolution of individual chunks, and is guaranteed to be fully connected, unlike a voxel or mesh representation of the neuron which can have gaps where there are defects in the segmentation volume or incorrectly inferred edges at self contact locations.
The level 2 graph can be turned into a skeleton representation of a neuron using a graph based TEASAR like algorithm as described for skeletonizing meshes in this MeshParty Documentation. There is an implementation of this approach that utilizes the chunkedgraph and the L2cache if available here and on pypi as pcg-skel. In this implementation the l2cache is used to more accurately place the level 2 nodes in space using the rep_coord_nm value.
"},{"location":"tutorials/l2cache/#trajectory-distributions","title":"Trajectory Distributions","text":"If one is interested in the bulk direction of processes in a region of the brain, one can start with supervoxels in a region, find level 2 nodes that correspond to them, filter out components based on size, (or other criteria such as whether they are part of objects that have components in some other brain area) and look at the distribution of PCA components to understand the directions that those processes are moving within that region of space.
"},{"location":"tutorials/materialization/","title":"Materialization","text":"The Materialization client allows one to interact with the materialized annotation tables, that were posted to the annotation service (the annotations tutorial).
To see the entire class visit the API doc.
The service regularly looks up all annotations and the segids underneath all the boundspatialpoints. You can then query these tables to find out the IDs that underlie the annotations, or the annotations that now intersect with certain IDs.
For example, one common pattern is that you have identified a cell based on the location of its cell body, and you have an annotation there.
You want to know what are the inputs onto the cell, so you first query the annotation table with your soma annotation, asking for the current ID underneath that soma. Then you query a synapse table for all synapse annotations that have a post-synaptic ID equal to the ID from your soma annotation.
In this way your code stays the same, as the proofreading changes and you can track the connectivity of your cell over time.
"},{"location":"tutorials/materialization/#initializing-the-client","title":"Initializing the client","text":"By default when you initialize the overall client, it will choose the most recent materialization version available. This may or may not be desirable depending on your use case. If your code involves using specific IDs then you should be using a specific version that is tied to a timepoint where those IDs are valid.
To see what versions are available, use the client.materialize.get_versions() function.
client.materialize.get_versions()\n
Each version has a timestamp it was run on as well as a date when it will expire. You can query all this metadata for a specific version using client.materialize.get_version_metadata() or all versions using client.materialize.get_versions_metadata().
To change the default version, alter the .version property of the client. This will change the version for all subsequent calls which expect one, unless you specify a different version in the method call. Note that this also sets the timestamp property of the client to the timestamp of the version for methods which expect a timestamp.
client.version = 9\n
You can also specify the version when you initialize the client, e.g.:
client = CAVEclient('minnie65_public', version=661)\n
Or, you can specify the version when making a particular method call.
"},{"location":"tutorials/materialization/#browsing-versions","title":"Browsing versions","text":"To see what tables are available in a version you can use client.materialize.get_tables().
If you want to read about the description of what that table is, use the annotationengine client client.materialize.get_table_metadata().
If you want to read more about the schema for the annotation table use the schema service client.schema.schema_definition().
Note, the materialization service has a human readable webpage that links to the other services that might be more convenient for you to browse, to get a link there in ipython display client.materialize.homepage
for some important tables, the info service has a pointer to which table you should use in the metadata for the datastack. client.info.get_datastack_info()['synapse_table'] and client.info.get_datastack_info()['soma_table'].
To see how many annotations are in a particular table use
nannotations=client.materialize.get_annotation_count('my_table')\n
"},{"location":"tutorials/materialization/#querying-tables","title":"Querying tables","text":"To query a small table, you can just download the whole thing using client.materialize.query_table() which will return a dataframe of the table.
Note however, some tables, such as the synapse table might be very large 200-300 million rows and the service will only return the first 200,000 results, and not in a deterministic manner. NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
To just get a preview, use the limit argument (but note again that this won't be a reproducible set)
df=client.materialize.query_table('my_table', limit=10)\n
For most applications, you will want to filter the query in some way.
We offer seven kinds of filters you can apply: filter_in_dict filter_out_dict, filter_equal_dict, filter_greater_dict, filter_less_dict, filter_greater_equal_dict, and filter_less_equal_dict. For query_table each is specified as a dictionary where the keys are column names, and the values are a list of values (or single value in the case of filter_equal).
So for example to query a synapse table for all synapses onto a neuron in flywire you would use
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID})\n
The speed of querying is affected by a number of factors, including the size of the data. To improve the performance of results, you can reduce the number of columns returned using select_columns.
So for example, if you are only interested in the root_ids and locations of pre_synaptic terminals you might limit the query with select_columns. Also, it is convenient to return the with positions as a column of np.array([x,y,z]) coordinates for many purposes. However, sometimes you might prefer to have them split out as separate x, y, z columns. To enable this option use split_columns=True. split_columns=True is faster, as combining them is an extra step.
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n select_columns=['id','pre_pt_root_id', 'pre_pt_position'],\n split_columns=True)\n
"},{"location":"tutorials/materialization/#desired-resolution","title":"Desired Resolution","text":"Often you want to have position information in different units. For example, to consider synapse locations or soma locations, you might want to have positions in nanometers or microns.
To create neuroglancer views, you might want positions in integer voxels of a size that aligns with the resolution you are used to using Neuroglancer at.
Annotation tables can be created and uploaded in varying resolutions according to whatever the user of the table felt was natural. This information is available in the metadata for that table. In addition, you may pass desired_resolution as a keyword argument which will automatically convert all spatial positions into voxels of that size in nanometers.
So if you want positions in nanometers, you would pass desired_resolution=[1,1,1]. If you want positions in microns you would pass desired_resolution=[1000,1000,1000]. If you want positions in 4,4,40nm voxel coordinates to use with cloud-volume or neuroglancer you would pass desired_resolution=[4,4,40].
"},{"location":"tutorials/materialization/#spatial-filters","title":"Spatial Filters","text":"You can also filter columns that are associated with spatial locations based upon being within a 3d bounding box.
This is done by adding a filter_spatial_dict argument to query_table. The units of the bounding box should be in the units of the voxel_resolution of the table (which can be obtained from client.materialize.get_table_metadata()).
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nsynapse_table = client.info.get_datastack_info('synapse_table')\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n filter_spatial_dict = {'post_pt_position': bounding_box})\n
"},{"location":"tutorials/materialization/#synapse-query","title":"Synapse Query","text":"For synapses in particular, we have a simplified method for querying them with a reduced syntax. client.materialize.synapse_query() lets you specify pre and post synaptic partners as keyword arguments and bounding boxes. The defaults make reasonable assumptions about what you want to query, namely that the synapse_table is the table that the info service advertises, and that if you specify a bounding box, that you want the post_pt_position. These can be overridden of course, but the above bounding box query is simplified to.
NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\ndf=client.materialize.query_table(post_ids = MYID,\n bounding_box=bounding_box)\n
"},{"location":"tutorials/materialization/#live-query","title":"Live Query","text":"In order to query the materialized tables above you can only use IDs that were present at the timestamp of the materialization. If you query the tables with an ID that is not valid during the time of the materialization you will get empty results.
To check if root_ids are valid at your materialization's timestamp, you can use client.chunkedgraph.is_latest_roots()
import numpy as np\nmat_time = client.materialize.get_timestamp()\nis_latest = client.chunkedgraph.is_latest_roots([MYID], timestamp=mat_time)\nassert(np.all(is_latest))\n
If you need to lookup what happened to that ID, you can use the chunkedgraph lineage tree, to look into the future or the past, depending on your application you can use client.chunkedgraph.get_lineage_graph().
Again, the ideal situation is that you have an annotation in the database which refers to your objects of interest, and querying that table by the id column will return the object in the most recent materialization.
However, sometimes you might be browsing and proofreadding the data and get an ID that is more recent that the most recent version available. For convenience, you can use client.materialize.live_query().
to automatically update the results of your query to a time in the future, such as now. For example, to pass now, use datetime.datetime.now(datetime.timezone.utc). Note all timestamps are in UTC throughout the codebase.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.live_query(synapse_table,\n datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\n
This will raise an ValueError exception if the IDs passed in your filters are not valid at the timestamp given
You can also pass a timestamp directly to query_table and it will call live_query automatically.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\n
Also, keep in mind if you run multiple queries and at each time pass datetime.datetime.now(datetime.timezone.utc), there is no guarantee that the IDs will be consistent from query to query, as proofreading might be happening at any time. For larger scale analysis constraining oneself to a materialized version will ensure consistent results.
Versions have varying expiration times in order to support the tradeoff between recency and consistency, so before undertaking an analysis project consider what version you want to query and what your plan will be to update your analysis to future versions.
"},{"location":"tutorials/materialization/#content-aware-interface-experimental","title":"Content-aware Interface (Experimental)","text":"As of version 5.8.0, we have introduced a new interface to query tables and views. This interface might have small but breaking changes in the near future. :::
In order to make the querying interface more consistent across tables, we have introduced an additional alternative interface to filtering and querying data via the client.materialize.tables object. When you instantiate this object, this object finds all of the existing tables and the list of their columns and lets you filter the tables as arguments in the function with suggestions. Moreover, the filtering arguments and the querying arguments are separated into two.
Let's see how this works with a simplest example --- downloading a table called nucleus_detection_v0. First, we reference the table as a function and then we run the query --- this is exactly the same as client.materialize.query_table('nucleus_detection_v0').
client = CAVEclient('minnie65_public')\nnuc_df = client.materialize.tables.nucleus_detection_v0().query()\n
Where things differ is when we add filters. If we want to query based on a set of values for the field \"id\", for example, we add that as an argument:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(id=my_ids).query()\n
Where in this example the id= queries the column id based on the schema. These values can be either individual elements (i.e. an integer or a string) or a list/array of elements, and any field can be used. The tooling will automatically sort out how to format the filtering appropriately when running the query. Importantly, the filtering is identical between querying all types of tables and queries. To see the complete list of fields that can be queried, you can tab-autocomplete or in Jupyter or IPython glance at the docstring with client.materialize.tables.nucleus_detection_v0?.
In addition to filtering by one or many values, you can also do spatial queries. To find only annotations within a particular bounding box, you need to specify the spatial point (e.g. pt_position). For each such spatial point, there will be an argument {spatial_point_position}_bbox (e.g. pt_position_bbox) that accepts a 2x3 list of coordinates specifying the upper and lower bounds for the query. For example, to find all nucleus centroids within a particular bounding box, it would be:
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n pt_position_bbox=bounding_box\n).query()\n
If you are not using any filters, you can omit the parenthesis and use the get_all or get_all_live functions directly, which act similarly to the query and live_query functions respectively. The first example could be rewritten as:
nuc_df = client.materialize.tables.nucleus_detection_v0.get_all()\n
If you want to list all available fields, you can use the .fields attribute. Similarly, you can get all numeric fields with the .numeric_fields attribute and all spatial fields (allowing bounding box queries) with .spatial_fields.
nuc_df = client.materialize.tables.nucleus_detection_v0.spatial_fields\n
If you need to specify the table programmatically, you can also use a dictionary-style approach to getting the table filtering function. For example, an equivalent version of the above line would be:
my_ids = [373879, 111162]\nmy_table = 'nucleus_detection_v0'\nnuc_df = client.materialize.tables[my_table](id=my_ids).query()\n
The query function can also take arguments relating to timestamps or formatting where they act just like in the other query method. In particular, the arguments that apply to query are: select_columns, offset, limit, split_positions, materialization_version, timestamp, metadata, desired_resolution, and get_counts. For example, to add a desired resolution and split positions in the above query, it would look like:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).query(\n split_positions=True,\n desired_resolution=[1,1,1],\n)\n
Inequalities can also be used in filtering numeric columns. Here, you can pass a dictionary instead of a list of values, with the keys being inequality operators (\">\", \">=\", \"<\", and \"<=\") and the values being the comparison. For example, to query for all nuclei with a volume greater than 1000:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 1000}\n).query()\n
You can also use multiple inequalities in the same dictionary to filter within a range. For example, to query for all nuclei with a volume between 500 and 750:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 500, \"<\": 750}\n).query()\n
If you want to do a live query instead of a materialized query, the filtering remains identical but we use the live_query function instead. The one required argument for live_query is the timestamp.
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).live_query(\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n)\n
The live query functions have similar but slightly different arguments: timestamp (required), offset, limit, split_positions, metadata, desired_resolution, and allow_missing_lookups.
Note that way that IPython handles docstrings means that while you can use ? to get the docstring of the filtering part of the function, you can't simply do something like client.materialize.tables.nucleus_detection_v0().query?. It will tell you the function can't be found, because technically the query function does not yet exist until the table filtering function is called.
Instead, if you want to glimpse the docstring of the query or live_query functions, you need to split it into two lines:
qry_func = client.materialize.tables.nucleus_detection_v0().query\nqry_func?\n
Finally, if the project you are working with has views, a similar interface is available to them via client.materialize.views. Currently views are not compatible with live query, and so only the .query function is available.
"},{"location":"tutorials/schemas/","title":"EM Annotation Schemas","text":"The EMAnnotationSchemas client lets one look up the available schemas and how they are defined. This is mostly used for programmatic interactions between services, but can be useful when looking up schema definitions for new tables.
"},{"location":"tutorials/schemas/#get-the-list-of-schema","title":"Get the list of schema","text":"One can get the list of all available schema with the schema method. Currently, new schema have to be generated on the server side, although we aim to have a generic set available to use.
client.schema.get_schemas()\n
"},{"location":"tutorials/schemas/#view-a-specific-schema","title":"View a specific schema","text":"The details of each schema can be viewed with the schema_definition method, formatted as per JSONSchema.
example_schema = client.schema.schema_definition('microns_func_coreg')\nexample_schema\n
This is mostly useful for programmatic interaction between services at the moment, but can also be used to inspect the expected form of an annotation by digging into the format.
example_schema['definitions']['FunctionalCoregistration']\n
"},{"location":"tutorials/state/","title":"JSON Neuroglancer State Service","text":"We store the JSON description of a Neuroglancer state in a simple database at the JSON Service. This is a convenient way to build states to distribute to people, or pull states to parse work by individuals. The JSON Client is at client.state
client.state\n
"},{"location":"tutorials/state/#retrieving-a-state","title":"Retrieving a state","text":"JSON states are found simply by their ID, which you get when uploading a state. You can download a state with get_state_json.
example_id = 4845531975188480\nexample_state = client.state.get_state_json(test_id)\nexample_state['layers'][0]\n
"},{"location":"tutorials/state/#uploading-a-state","title":"Uploading a state","text":"You can also upload states with upload_state_json. If you do this, the state id is returned by the function. Note that there is no easy way to query what you uploaded later, so be VERY CAREFUL with this state id if you wish to see it again.
Note: If you are working with a Neuroglancer Viewer object or similar, in order to upload, use viewer.state.to_json() to generate this representation.
example_state['layers'][0]['name'] = 'example_name'\nnew_id = client.state.upload_state_json(example_state)\n
test_state = client.state.get_state_json(new_id)\ntest_state['layers'][0]['name']\n
"},{"location":"tutorials/state/#generating-a-neuroglancer-url","title":"Generating a Neuroglancer URL","text":"Once you have a state ID, you want to turn it into a well-formatted link. So you don\\'t have to remember all the endpoints, we can do this from the state client.
ngl_base = 'neuromancer-seung-import.appspot.com'\nclient.state.build_neuroglancer_url(new_id, ngl_base)\n
Note that the neuroglancer base can be found in the info service under client.info.viewer_site().
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"CAVE is short for Connectome Annotation Versioning Engine. CAVE is a set of microservices that provide a framework for storing and versioning connectomics data and large sets of dynamic annotations, metadata, and segmentations. This repository supplies client-side code to easily interact with the microservices in CAVE.
"},{"location":"#documentation-layout","title":"Documentation Layout","text":"To learn how to install caveclient, visit Installation.
To see hands-on examples of using caveclient in a notebook, visit the Tutorials.
To see API reference documentation for interacting with a caveclient.CAVEclient object, visit the Client API.
To see a glossary of terms used in the documentation, visit the Glossary (work in progress).
For information about how to contribute to the documentation or the package, visit the Contributing page. Feedback on the documentation is welcome! Please open an issue or use the \"Edit this page\" button at the top right of any page to suggest changes.
To read a full description of the Connectome Annotation Versioning Engine, including a description of the various services please read this paper.
"},{"location":"changelog/","title":"Changelog","text":""},{"location":"changelog/#600-october-14-2024","title":"6.0.0 (October 14, 2024)","text":" - Refactored CAVEclient internals away from a factory syntax. Most users should not notice a difference, but code that relied on specific subclient type logic may be affected.
"},{"location":"changelog/#5250","title":"5.25.0","text":" - Added ability to suppress table warnings in client.materialize.query_table and similar methods by setting
log_warnings=False
"},{"location":"changelog/#5200-april-8-2024","title":"5.20.0 (April 8, 2024)","text":" - Added generalized support for detecting server versions to provide timely exceptions to users
- Used new support to check that chunkegraph has updated version before using spatial bounds kwarg on client.chunkedgraph.level2_chunk_graph
- Added support for postign and getting segment properties files to client.state
"},{"location":"changelog/#5180","title":"5.18.0","text":" - Added serialization support for pandas.index
"},{"location":"changelog/#5173","title":"5.17.3","text":" - Minor documentation typo fix
"},{"location":"changelog/#5172","title":"5.17.2","text":" - Bug fixes related to table_manager interface
"},{"location":"changelog/#5171","title":"5.17.1","text":" - Bug fixes related to table_manager interface
"},{"location":"changelog/#5170","title":"5.17.0","text":" - Fix attrs in dataframe attributes of client.materialize results to remove numpy arrays to allow concatenation of dataframes
- Added getting multiple schemas in one call to improve initialization of table_manager interface of materialization
"},{"location":"changelog/#5161","title":"5.16.1","text":" - Bugfix on client.chunkedgrpah.level2_chunk_graph
"},{"location":"changelog/#5160","title":"5.16.0","text":" - Added bounding box query to client.chunkedgraph.level2_chunk_graph
- Fix default materialization version client when server not advertising correctly
"},{"location":"changelog/#5151-jan-18-2024","title":"5.15.1 (Jan 18, 2024)","text":" - minor improvements to release process
"},{"location":"changelog/#5150-jan-18-2024","title":"5.15.0 (Jan 18, 2024)","text":" - Improved documentation with types
- Improved testing on more python versions
- Bugfixes for pyton 3.12 compatability
"},{"location":"changelog/#5140-november-24-2023","title":"5.14.0 (November 24, 2023)","text":" - Made automatic detection of neuroglancer versioning when constructing link shortener links
"},{"location":"changelog/#5130-october-26-2023","title":"5.13.0 (October 26, 2023)","text":" - Add option to get expired versions to client.materialize.get_versions
"},{"location":"changelog/#5121-october-16-2023","title":"5.12.1 (October 16, 2023)","text":" - Bugfixes for client.chunkedgraph.get_latest_roots
"},{"location":"changelog/#5120-october-16-2023","title":"5.12.0 (October 16, 2023)","text":" - Improved logic for client.chunkedgraph.get_latest_roots to work forward or backwards in time
"},{"location":"changelog/#5110-september-19-2023","title":"5.11.0 (September 19, 2023)","text":" - Added filter_regex_dict options to client.materialize.query_table interface
"},{"location":"changelog/#5102-august-162023","title":"5.10.2 (August 16,2023)","text":" - Fixed pyarrow support for live_live query
"},{"location":"changelog/#5101-august-142023","title":"5.10.1 (August 14,2023)","text":" - Changed random_sample argument to be an integer number of annotations rather than a floating fraction of table
- Added option to live_query
"},{"location":"changelog/#590-august-14-2023","title":"5.9.0 (August 14, 2023)","text":" - Added support for native pyarrow deserialization, allowing upgrade to pyarrow version
"},{"location":"changelog/#580","title":"5.8.0","text":" - Allowed int64 root ids to serialize properly
- Added warning that client.materialize.tables interface is in beta
"},{"location":"changelog/#570","title":"5.7.0","text":" - Fix to ensure stop_layer is at least 1
- Added client.chunkedgraph.suggest_latest_roots
"},{"location":"changelog/#560","title":"5.6.0","text":" - Added views to client.materialize.tables interface
- Added optional argument to allow invalid root ids when querying live live, versus creating an exception
"},{"location":"changelog/#551","title":"5.5.1","text":" - documentation fixes on client.materialize.join_query
"},{"location":"changelog/#550","title":"5.5.0","text":" - added methods for different neuroglancer state formats to client.state.
"},{"location":"changelog/#543","title":"5.4.3","text":" - Added 'view' querying options to materialization
- Added client.materialize.tables interface
- Added client.materialize.get_tables_metadata to get all metadata in one call
"},{"location":"changelog/#520","title":"5.2.0","text":" - Added local caching of datastack names > server_address to simplify initialization of clients with servers other than global.daf-apis.com.
Cache is saved on a local file ~/.cloudvolume/secrets/cave_datastack_to_server_map.json
Cache will populate the first time caveclient.CAVEclient('my_datastack', server_address=\"https://my_server.com\") is called. Subsequent calls can then just be caveclient.CAVEclient('my_datastack').
"},{"location":"changelog/#510","title":"5.1.0","text":" - Added get_oldest_timestamp call to chunkedgraph
"},{"location":"changelog/#501","title":"5.0.1","text":" - Fixed bug with desired_resolution being set at the client level was being ignored in >5.0.0
"},{"location":"changelog/#500","title":"5.0.0","text":" - Added support for the new CAVE Materialization 3.0 API Includes support for the new materialization API, which allows for server side conversion of the units of position, and ensures that all positions are returned with the same units, even after joins.
- Added support for querying databases that were materialized without merging tables together. This will allow for faster materializations.
- Removed support for LiveLive query from the Materialization 2.0 API client. Note.. <5.0.0 clients interacting with MaterializationEngine >4.7.0 servers will use live live query but will doubly convert the units of position if you ask for a desired resolution, as the old client will also do a conversion server side.
- Fixed interaction with api version querying of servers from individual clients to work with verify=False. (useful for testing)
- Stored infromation from client about mapping between dataframe and table names and original column names.
- Added support for suffixes and select columns to be passed by dictionary rather than list making the selection an application of suffixes more explicit when there are collisions between column names in joined tables.
"},{"location":"changelog/#older-upgrade-notes","title":"Older Upgrade Notes","text":"Change all select_column calls to pass dictionaries rather than lists. Change all suffix calls to pass dictionaries rather than lists. Advocate for your server administrator to upgrade to MaterializationEngine 4.7.0 or later, so you can use the new MaterializationEngine 3.0 API and client.
"},{"location":"contributing/","title":"Contributing","text":"Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
You can contribute in many ways:
"},{"location":"contributing/#types-of-contributions","title":"Types of Contributions","text":""},{"location":"contributing/#report-bugs","title":"Report Bugs","text":"Report bugs to our issues page.
If you are reporting a bug, please include:
- Your operating system name and version.
- Any details about your local setup that might be helpful in troubleshooting.
- Detailed steps to reproduce the bug, in the form of a minimal reproducible example.
"},{"location":"contributing/#fix-bugs","title":"Fix Bugs","text":"Look through the GitHub issues for bugs. Anything tagged with \"bug\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#implement-features","title":"Implement Features","text":"Look through the GitHub issues for features. Anything tagged with \"enhancement\" and \"help wanted\" is open to whoever wants to implement it.
"},{"location":"contributing/#write-documentation","title":"Write Documentation","text":"caveclient could always use more documentation, whether as part of the official caveclient docs, in docstrings, or even on the web in blog posts, articles, and such.
"},{"location":"contributing/#submit-feedback","title":"Submit Feedback","text":"The best way to send feedback is to create an issue on GitHub.
If you are proposing a feature:
- Explain in detail how it would work.
- Keep the scope as narrow as possible, to make it easier to implement.
- Remember that while contributions are welcome, developer/maintainer time is limited.
"},{"location":"contributing/#get-started","title":"Get Started","text":"Ready to contribute? Here's how to set up caveclient for local development.
- Fork the repo on GitHub.
- Clone your fork locally
git clone git@github.com:your_name_here/CAVEclient.git\n
- Ensure pip is installed.
- Create a virtual environment (here we use venv):
python3 -m venv .venv\n
source .venv/bin/activate\n
- Create a branch for local development:
git checkout -b name-of-your-bugfix-or-feature\n
- Make your changes locally
- Install development requirements:
pip install -r test_requirements.txt\npip install -e .\n
- When you're done making changes, check that your changes pass the tests by running pytest:
pytest tests\n
Note that once you submit your pull request, GitHub Actions will run the tests also, including on multiple operating systems and Python versions. Your pull request will have to pass on all of these before it can be merged.
- Ensure your contribution meets style guidelines. First, install ruff:
pip install ruff\n
- Fix linting and formatting. From the root of the repository, run the following commands:
ruff check . --extend-select I --fix\nruff format .\n
- Commit your changes and push your branch to GitHub:
git add .\ngit commit -m \"Your detailed description of your changes.\"\ngit push origin name-of-your-bugfix-or-feature\n
- Submit a pull request through the GitHub website.
"},{"location":"contributing/#pull-request-guidelines","title":"Pull Request Guidelines","text":"Before you submit a pull request, check that it meets these guidelines:
- The pull request should include tests if adding a new feature.
- The docs should be updated with whatever changes you have made. Put your new functionality into a function with a docstring, and make sure the new functionality is documented after building the documentation.
"},{"location":"contributing/#documentation-style","title":"Documentation style","text":"We use mkdocs to build the documentation. In particular, we use the mkdocs-material theme, and a variety of other extensions.
Note
More information codifying our documentation style and principles coming soon. For now, just try to follow the style of the existing documentation.
"},{"location":"glossary/","title":"Glossary","text":"Warning
This glossary is a work in progress; for now we are documenting the commonly used terms that we need to define. Please feel free to contribute definitions or additional terms.
- Datastack
- Voxel resolution
- Segmentation resolution
- MIP
- Segmentation
- View
"},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#stable-release","title":"Stable release","text":"To install caveclient, run this command in your terminal:
pip install caveclient\n
This is the preferred method to install caveclient, as it will always install the most recent stable release.
You can also specify a particular version, e.g.
pip install caveclient==5.0.0\n
If you don't have pip installed, this Python installation guide can guide you through the process.
"},{"location":"installation/#from-source","title":"From source","text":"The source for caveclient can be downloaded from the Github repo.
You can either clone the public repository:
git clone git://github.com/CAVEconnectome/CAVEclient\n
Or download the tarball:
curl -OJL https://github.com/CAVEconnectome/CAVEclient/tarball/master\n
Once you have a copy of the source, you can install it with:
pip install .\n
Or in editable mode, it can be installed with:
pip install -e .\n
"},{"location":"api/","title":"Overview","text":"The most common method of interacting with the CAVE Framework is by instantiating a client (caveclient.CAVEclient) and then using that client to interact with various services. Under the hood, the CAVEclient is a collection of individual clients, which can be accessed via properties. For example, to access the materialization client, you can use client.materialize, which (up to the exact version) will actually return a MaterializationClient object.
These pages describe the functionality of each of the individual clients, but note that some features may not be available if the server you are accessing is using older versions of particular services.
"},{"location":"api/annotation/","title":"client.annotation","text":"Client for interacting with the annotation engine.
Methods:
Name Description create_table Creates a new data table based on an existing schema
delete_annotation Delete one or more annotations in a table. Annotations that are
delete_table Marks a table for deletion requires super admin privileges
get_annotation Retrieve an annotation or annotations by id(s) and table name.
get_annotation_count Get number of annotations in a table
get_table_metadata Get metadata about a table
get_tables Gets a list of table names for a aligned_volume_name
post_annotation Post one or more new annotations to a table in the AnnotationEngine.
post_annotation_df Post one or more new annotations to a table in the AnnotationEngine.
process_position_columns Process a dataframe into a list of dictionaries
raise_for_status Raises requests.HTTPError, if one occurred.
stage_annotations Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema.
update_annotation Update one or more new annotations to a table in the AnnotationEngine.
update_annotation_df Update one or more annotations to a table in the AnnotationEngine using a
update_metadata Update the metadata on an existing table
upload_staged_annotations Upload annotations directly from an Annotation Guide object.
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.create_table","title":"create_table(table_name, schema_name, description, voxel_resolution, reference_table=None, track_target_id_updates=None, flat_segmentation_source=None, user_id=None, aligned_volume_name=None, write_permission='PRIVATE', read_permission='PUBLIC', notice_text=None)","text":"Creates a new data table based on an existing schema
Parameters:
Name Type Description Default table_name str Name of the new table. Cannot be the same as an existing table
required schema_name str Name of the schema for the new table.
required description str Human readable description for what is in the table. Should include information about who generated the table What data it covers, and how it should be interpreted. And who should you talk to if you want to use it. An Example: a manual synapse table to detect chandelier synapses on 81 PyC cells with complete AISs [created by Agnes - agnesb@alleninstitute.org, uploaded by Forrest]
required voxel_resolution List[float] voxel resolution points will be uploaded in, typically nm, i.e [1,1,1] means nanometers [4,4,40] would be 4nm, 4nm, 40nm voxels
required reference_table str If the schema you are using is a reference schema Meaning it is an annotation of another annotation. Then you need to specify what the target table those annotations are in.
None track_target_id_updates bool Indicates whether to automatically update reference table's foreign key if target annotation table row is updated.
None flat_segmentation_source str the source to a flat segmentation that corresponds to this table i.e. precomputed:\\gs:\\mybucket his_tables_annotation
None user_id int If you are uploading this schema on someone else's behalf and you want to link this table with their ID, you can specify it here Otherwise, the table will be created with your userID in the user_id column.
None aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table (DEFAULT) GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable
'PRIVATE' read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data (DEFAULT)
'PUBLIC' notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None, no text. If you want to remove text, send empty string.
None Returns:
Type Description json Response JSON
Examples:
Basic annotation table:
description = \"Some description about the table\"\nvoxel_res = [4,4,40]\nclient.create_table(\"some_synapse_table\", \"synapse\", description, voxel_res)\n
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_annotation","title":"delete_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Delete one or more annotations in a table. Annotations that are deleted are recorded as 'non-valid' but are not physically removed from the table.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required annotation_ids (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.delete_table","title":"delete_table(table_name, aligned_volume_name=None)","text":"Marks a table for deletion requires super admin privileges
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation","title":"get_annotation(table_name, annotation_ids, aligned_volume_name=None)","text":"Retrieve an annotation or annotations by id(s) and table name.
Parameters:
Name Type Description Default table_name str Name of the table
required annotation_ids int or iterable ID or IDS of the annotation to retreive
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description list Annotation data
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_annotation_count","title":"get_annotation_count(table_name, aligned_volume_name=None)","text":"Get number of annotations in a table
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description int number of annotations
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_table_metadata","title":"get_table_metadata(table_name, aligned_volume_name=None)","text":"Get metadata about a table
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required aligned_volume_name str Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json metadata about table
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.get_tables","title":"get_tables(aligned_volume_name=None)","text":"Gets a list of table names for a aligned_volume_name
Parameters:
Name Type Description Default aligned_volume_name str or None Name of the aligned_volume, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Description list List of table names
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation","title":"post_annotation(table_name, data, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations refer to 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required data (dict or list) A list of (or a single) dict of schematized annotation data matching the target table.
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.post_annotation_df","title":"post_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Post one or more new annotations to a table in the AnnotationEngine. All inserted annotations will be marked as 'valid'. To invalidate annotations see 'update_annotation', 'update_annotation_df' and 'delete_annotation' methods.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required df DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
required position_columns Optional[Union[Iterable[str], Mapping[str, str]]] if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.process_position_columns","title":"process_position_columns(df, position_columns) staticmethod","text":"Process a dataframe into a list of dictionaries
Parameters:
Name Type Description Default df DataFrame Dataframe to process
required position_columns Optional[Union[Iterable[str], Mapping[str, str]]] See .post_annotation_df
required Returns:
Type Description dict Annotations ready for posting
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.raise_for_status","title":"raise_for_status(r, log_warning=True) staticmethod","text":"Raises requests.HTTPError, if one occurred.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.stage_annotations","title":"stage_annotations(table_name=None, schema_name=None, update=False, id_field=False, table_resolution=None, annotation_resolution=None)","text":"Get a StagedAnnotations object to help produce correctly formatted annotations for a given table or schema. StagedAnnotation objects can be uploaded directly with upload_staged_annotations.
Parameters:
Name Type Description Default table_name str Table name to stage annotations for, by default None.
None schema_name str Schema name to use to make annotations. Only needed if the table_name is not set, by default None
None update bool Set to True if individual annotations are going to be updated, by default False.
False id_field bool Set to True if id fields are to be specified. Not needed if update is True, which always needs id fields. Optional, by default False
False table_resolution list - like or None Voxel resolution of spatial points in the table in nanometers. This is found automatically from the info service if a table name is provided, by default None. If annotation_resolution is also set, this allows points to be scaled correctly for the table.
None annotation_resolution list - like Voxel resolution of spatial points provided by the user when creating annotations. If the table resolution is also available (manually or from the info service), annotations are correctly rescaled for the volume. By default, None.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation","title":"update_annotation(table_name, data, aligned_volume_name=None)","text":"Update one or more new annotations to a table in the AnnotationEngine. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
Notes If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required data (dict or list) A list of (or a single) dict of schematized annotation data matching the target table. each dict must contain an \"id\" field which is the ID of the annotation to update
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON: a list of new annotation IDs.
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_annotation_df","title":"update_annotation_df(table_name, df, position_columns, aligned_volume_name=None)","text":"Update one or more annotations to a table in the AnnotationEngine using a dataframe as format. Updating is implemented by invalidating the old annotation and inserting a new annotation row, which will receive a new primary key ID.
Notes If annotations ids were user provided upon insertion the database will autoincrement from the current max id in the table.
Parameters:
Name Type Description Default table_name str Name of the table where annotations will be added
required df DataFrame A pandas dataframe containing the annotations. Columns should be fields in schema, position columns need to be called out in position_columns argument.
required position_columns Iterable[str] or Mapping[str, str] or None if None, will look for all columns with 'X_position' in the name and assume they go in fields called \"X\". if Iterable assumes each column given ends in _position. (i.e. ['pt_position'] if 'pt' is the name of the position field in schema) if Mapping, keys are names of columns in dataframe, values are the names of the fields (i.e. {'pt_column': 'pt'} would be correct if you had one column named 'pt_column' which needed to go into a schema with a position column called 'pt')
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description json Response JSON
"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.update_metadata","title":"update_metadata(table_name, description=None, flat_segmentation_source=None, read_permission=None, write_permission=None, user_id=None, notice_text=None, aligned_volume_name=None)","text":"Update the metadata on an existing table
Parameters:
Name Type Description Default table_name str required description str Defaults to None (will not update).
None flat_segmentation_source str Defaults to None (will not update).
None read_permission str What permissions to give the table for reading. One of PRIVATE: only you can read this table. Intended to be used for sorting out bugs. GROUP: only members that share a group with you can read (intended for within group vetting) PUBLIC: anyone with permissions to read this datastack can read this data Defaults to None (will not update).
None write_permission str What permissions to give the table for writing. One of PRIVATE: only you can write to this table GROUP: only members that share a group with you can write (excluding some groups) PUBLIC: Anyone can write to this table. Note all data is logged, and deletes are done by marking rows as deleted, so all data is always recoverable Defaults to None (will not update).
None user_id int Note, if you use this you will not be able to update the metadata on this table any longer and depending on permissions may not be able to read or write to it Defaults to None. (will not update)
None notice_text str Text the user will see when querying this table. Can be used to warn users of flaws, and uncertainty in the data, or to advertise citations that should be used with this table. Defaults to None. (will not update)
None aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None"},{"location":"api/annotation/#caveclient.annotationengine.AnnotationClient.upload_staged_annotations","title":"upload_staged_annotations(staged_annos, aligned_volume_name=None)","text":"Upload annotations directly from an Annotation Guide object. This method uses the options specified in the object, including table name and if the annotation is an update or not.
Parameters:
Name Type Description Default staged_annos AnnotationGuide AnnotationGuide object with a specified table name and a collection of annotations already filled in.
required aligned_volume_name str or None Name of the aligned_volume. If None, uses the one specified in the client.
None Returns:
Type Description List or dict If new annotations are posted, a list of ids. If annotations are being updated, a dictionary with the mapping from old ids to new ids.
"},{"location":"api/auth/","title":"client.auth","text":"Client to find and use auth tokens to access the other services.
Methods:
Name Description get_group_users Get users in a group
get_new_token Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
get_token Load a token with a given key the specified token file
get_tokens Get the tokens setup for this users
get_user_information Get user data.
save_token Conveniently save a token in the correct format.
setup_token Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment.
Attributes:
Name Type Description request_header Formatted request header with the specified token
token Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.request_header","title":"request_header property","text":"Formatted request header with the specified token
"},{"location":"api/auth/#caveclient.auth.AuthClient.token","title":"token property writable","text":"Secret token used to authenticate yourself to the Connectome Annotation Versioning Engine services.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_group_users","title":"get_group_users(group_id)","text":"Get users in a group
Parameters:
Name Type Description Default group_id int ID value for a given group
required Returns:
Type Description list List of dicts of user ids. Returns empty list if group does not exist.
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_new_token","title":"get_new_token(open=False, no_text=False)","text":"Currently, returns instructions for getting a new token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Default open bool If True, opens a web browser to the web page where you can generate a new token.
False"},{"location":"api/auth/#caveclient.auth.AuthClient.get_token","title":"get_token(token_key=None)","text":"Load a token with a given key the specified token file
Parameters:
Name Type Description Default token_key str or None key in the token file JSON, by default None. If None, uses 'token'.
None"},{"location":"api/auth/#caveclient.auth.AuthClient.get_tokens","title":"get_tokens()","text":"Get the tokens setup for this users
Returns:
Type Description list[dict]: a list of dictionary of tokens, each with the keys \"id\": the id of this token \"token\": the token (str) \"user_id\": the users id (should be your ID)
"},{"location":"api/auth/#caveclient.auth.AuthClient.get_user_information","title":"get_user_information(user_ids)","text":"Get user data.
Parameters:
Name Type Description Default user_ids list of int user_ids to look up
required"},{"location":"api/auth/#caveclient.auth.AuthClient.save_token","title":"save_token(token=None, token_key=default_token_key, overwrite=False, token_file=None, switch_token=True, write_to_server_file=True, ignore_readonly=True)","text":"Conveniently save a token in the correct format.
After getting a new token by following the instructions in authclient.get_new_token(), you can save it with a fully default configuration by running:
token = 'my_shiny_new_token'
authclient.save_token(token=token)
Now on next load, authclient=AuthClient() will make an authclient instance using this token. If you would like to specify more information about the json file where the token will be stored, see the parameters below.
Parameters:
Name Type Description Default token str New token to save, by default None
None token_key str Key for the token in the token_file json, by default \"token\"
default_token_key overwrite bool Allow an existing token to be changed, by default False
False token_file str Path to the token file, by default None. If None, uses the default file location specified above.
None switch_token bool If True, switch the auth client over into using the new token, by default True
True write_to_server_file bool If True, will write token to a server specific file to support this machine interacting with multiple auth servers.
True ignore_readonly bool If True, will only attempt to save a token if the directory is writeable.
True"},{"location":"api/auth/#caveclient.auth.AuthClient.setup_token","title":"setup_token(make_new=True, open=True)","text":"Currently, returns instructions for getting your auth token based on the current settings and saving it to the local environment. New OAuth tokens are currently not able to be retrieved programmatically.
Parameters:
Name Type Description Default make_new bool If True, will make a new token, else prompt you to open a page to retrieve an existing token.
True open bool If True, opens a web browser to the web page where you can retrieve a token.
True"},{"location":"api/chunkedgraph/","title":"client.chunkedgraph","text":"Client for interacting with the chunkedgraph.
Methods:
Name Description get_roots Get the root ID for a list of supervoxels.
get_root_id Get the root ID for a specified supervoxel.
get_merge_log Get the merge log (splits and merges) for an object.
get_change_log Get the change log (splits and merges) for an object.
get_user_operations Get operation details for a user ID. Currently, this is only available to
get_tabular_change_log Get a detailed changelog for neurons.
get_leaves Get all supervoxels for a root ID.
do_merge Perform a merge on the chunked graph.
undo_operation Undo an operation.
execute_split Execute a multicut split based on points or supervoxels.
preview_split Get supervoxel connected components from a preview multicut split.
get_children Get the children of a node in the chunked graph hierarchy.
get_contact_sites Get contacts for a root ID.
find_path Find a path between two locations on a root ID using the level 2 chunked
get_subgraph Get subgraph of root id within a bounding box.
level2_chunk_graph Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
remesh_level2_chunks Submit specific level 2 chunks to be remeshed in case of a problem.
get_operation_details Get the details of a list of operations.
get_lineage_graph Returns the lineage graph for a root ID, optionally cut off in the past or
get_latest_roots Returns root IDs that are related to the given root_id at a given
get_original_roots Returns root IDs that are the latest successors of a given root ID.
is_latest_roots Check whether these root IDs are still a root at this timestamp.
suggest_latest_roots Suggest latest roots for a given root id, based on overlap of component
is_valid_nodes Check whether nodes are valid for given timestamp range.
get_root_timestamps Retrieves timestamps when roots where created.
get_past_ids For a set of root IDs, get the list of IDs at a past or future time point
get_delta_roots Get the list of roots that have changed between timetamp_past and
get_oldest_timestamp Get the oldest timestamp in the database.
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
timestamp Optional[datetime] The default timestamp for queries which expect a timestamp. If None, uses the
segmentation_info Complete segmentation metadata
base_resolution MIP 0 resolution for voxels assumed by the ChunkedGraph
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.timestamp","title":"timestamp: Optional[datetime.datetime] property writable","text":"The default timestamp for queries which expect a timestamp. If None, uses the current time.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.segmentation_info","title":"segmentation_info property","text":"Complete segmentation metadata
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.base_resolution","title":"base_resolution property","text":"MIP 0 resolution for voxels assumed by the ChunkedGraph
Returns:
Type Description list 3-long list of x/y/z voxel dimensions in nm
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_roots","title":"get_roots(supervoxel_ids, timestamp=None, stop_layer=None)","text":"Get the root ID for a list of supervoxels.
Parameters:
Name Type Description Default supervoxel_ids list or np.array of int Supervoxel IDs to look up.
required timestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None stop_layer int or None If True, looks up IDs only up to a given stop layer. Default is None.
None Returns:
Type Description np.array of np.uint64 Root IDs containing each supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_id","title":"get_root_id(supervoxel_id, timestamp=None, level2=False)","text":"Get the root ID for a specified supervoxel.
Parameters:
Name Type Description Default supervoxel_id int Supervoxel id value
required timestamp datetime UTC datetime to specify the state of the chunkedgraph at which to query, by default None. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description int64 Root ID containing the supervoxel.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_merge_log","title":"get_merge_log(root_id)","text":"Get the merge log (splits and merges) for an object.
Parameters:
Name Type Description Default root_id int Object root ID to look up.
required Returns:
Type Description list List of merge events in the history of the object.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_change_log","title":"get_change_log(root_id, filtered=True)","text":"Get the change log (splits and merges) for an object.
Parameters:
Name Type Description Default root_id int Object root ID to look up.
required filtered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query object root_id (filtered=False). Defaults to True.
True Returns:
Type Description dict Dictionary summarizing split and merge events in the object history, containing the following keys:
- \"n_merges\": int
- \"n_splits\": int
- \"operations_ids\": list of int
- Identifiers for each operation
- \"past_ids\": list of int
- Previous root ids for this object
- \"user_info\": dict of dict
- Dictionary keyed by user (string) to a dictionary specifying how many merges and splits that user performed on this object
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_user_operations","title":"get_user_operations(user_id, timestamp_start, include_undo=True, timestamp_end=None)","text":"Get operation details for a user ID. Currently, this is only available to admins.
Parameters:
Name Type Description Default user_id int User ID to query (use 0 for all users (admin only)).
required timestamp_start datetime Timestamp to start filter (UTC).
required include_undo bool Whether to include undos. Defaults to True.
True timestamp_end datetime Timestamp to end filter (UTC). Defaults to now.
None Returns:
Type Description DataFrame DataFrame including the following columns:
- \"operation_id\": int
- Identifier for the operation.
- \"timestamp\": datetime.datetime
- Timestamp of the operation.
- \"user_id\": int
- User who performed the operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_tabular_change_log","title":"get_tabular_change_log(root_ids, filtered=True)","text":"Get a detailed changelog for neurons.
Parameters:
Name Type Description Default root_ids list of int Object root IDs to look up.
required filtered bool Whether to filter the change log to only include splits and merges which affect the final state of the object (filtered=True), as opposed to including edit history for objects which as some point were split from the query objects in root_ids (filtered=False). Defaults to True.
True Returns:
Type Description dict of pd.DataFrame The keys are the root IDs, and the values are DataFrames with the following columns and datatypes:
- \"operation_id\": int
- Identifier for the operation.
- \"timestamp\": int
- Timestamp of the operation, provided in milliseconds. To convert to datetime, use
datetime.datetime.utcfromtimestamp(timestamp/1000).
- \"user_id\": int
- User who performed the operation.
- \"before_root_ids: list of int
- Root IDs of objects that existed before the operation.
- \"after_root_ids: list of int
- Root IDs of objects created by the operation. Note that this only records the root id that was kept as part of the query object, so there will only be one in this list.
- \"is_merge\": bool
- Whether the operation was a merge.
- \"user_name\": str
- Name of the user who performed the operation.
- \"user_affiliation\": str
- Affiliation of the user who performed the operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_leaves","title":"get_leaves(root_id, bounds=None, stop_layer=None)","text":"Get all supervoxels for a root ID.
Parameters:
Name Type Description Default root_id int Root ID to query.
required bounds If specified, returns supervoxels within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]]. If None, finds all supervoxels.
None stop_layer int If specified, returns chunkedgraph nodes at layer stop_layer default will be stop_layer=1 (supervoxels).
None Returns:
Type Description np.array of np.int64 Array of supervoxel IDs (or node ids if stop_layer>1).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.do_merge","title":"do_merge(supervoxels, coords, resolution=(4, 4, 40))","text":"Perform a merge on the chunked graph.
Parameters:
Name Type Description Default supervoxels iterable An N-long list of supervoxels to merge.
required coords array An Nx3 array of coordinates of the supervoxels in units of resolution.
required resolution tuple What to multiply coords by to get nanometers. Defaults to (4,4,40).
(4, 4, 40)"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.undo_operation","title":"undo_operation(operation_id)","text":"Undo an operation.
Parameters:
Name Type Description Default operation_id int Operation ID to undo.
required Returns:
Type Description dict"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.execute_split","title":"execute_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None)","text":"Execute a multicut split based on points or supervoxels.
Parameters:
Name Type Description Default source_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
required sink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
required root_id int Root ID of object to do split preview.
required source_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None Returns:
Name Type Description operation_id int Unique ID of the split operation
new_root_ids list of int List of new root IDs resulting from the split operation.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.preview_split","title":"preview_split(source_points, sink_points, root_id, source_supervoxels=None, sink_supervoxels=None, return_additional_ccs=False)","text":"Get supervoxel connected components from a preview multicut split.
Parameters:
Name Type Description Default source_points array or list Nx3 list or array of 3d points in nm coordinates for source points (red).
required sink_points array or list Mx3 list or array of 3d points in nm coordinates for sink points (blue).
required root_id int Root ID of object to do split preview.
required source_supervoxels (array, list or None) If providing source supervoxels, an N-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None sink_supervoxels (array, list or None) If providing sink supervoxels, an M-length array of supervoxel IDs or Nones matched to source points. If None, treats as a full array of Nones. By default None.
None return_additional_ccs bool If True, returns any additional connected components beyond the ones with source and sink points. In most situations, this can be ignored. By default, False.
False Returns:
Name Type Description source_connected_component list Supervoxel IDs in the component with the most source points.
sink_connected_component list Supervoxel IDs in the component with the most sink points.
successful_split bool True if the split worked.
other_connected_components (optional) : list of lists of int List of lists of supervoxel IDs for any other resulting connected components. Only returned if return_additional_ccs is True.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_children","title":"get_children(node_id)","text":"Get the children of a node in the chunked graph hierarchy.
Parameters:
Name Type Description Default node_id int Node ID to query.
required Returns:
Type Description np.array of np.int64 IDs of child nodes.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_contact_sites","title":"get_contact_sites(root_id, bounds, calc_partners=False)","text":"Get contacts for a root ID.
Parameters:
Name Type Description Default root_id int Root ID to query.
required bounds Bounds within a 3x2 numpy array of bounds [[minx,maxx],[miny,maxy],[minz,maxz]] for which to find contacts. Running this query without bounds is too slow.
required calc_partners bool If True, get partner root IDs. By default, False.
False Returns:
Type Description dict Dict relating ids to contacts
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.find_path","title":"find_path(root_id, src_pt, dst_pt, precision_mode=False)","text":"Find a path between two locations on a root ID using the level 2 chunked graph.
Parameters:
Name Type Description Default root_id int Root ID to query.
required src_pt array 3-element array of xyz coordinates in nm for the source point.
required dst_pt array 3-element array of xyz coordinates in nm for the destination point.
required precision_mode bool Whether to perform the search in precision mode. Defaults to False.
False Returns:
Name Type Description centroids_list array Array of centroids along the path.
l2_path np.array of int Array of level 2 chunk IDs along the path.
failed_l2_ids np.array of int Array of level 2 chunk IDs that failed to find a path.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_subgraph","title":"get_subgraph(root_id, bounds)","text":"Get subgraph of root id within a bounding box.
Parameters:
Name Type Description Default root_id int Root (or any node ID) of chunked graph to query.
required bounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates.
required Returns:
Type Description np.array of np.int64 Node IDs in the subgraph.
np.array of np.double Affinities of edges in the subgraph.
np.array of np.int32 Areas of nodes in the subgraph.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.level2_chunk_graph","title":"level2_chunk_graph(root_id, bounds=None)","text":"Get graph of level 2 chunks, the smallest agglomeration level above supervoxels.
Parameters:
Name Type Description Default root_id int Root id of object
required bounds array 3x2 bounding box (x,y,z) x (min,max) in chunked graph coordinates (use client.chunkedgraph.base_resolution to view this default resolution for your chunkedgraph client). Note that the result will include any level 2 nodes which have chunk boundaries within some part of this bounding box, meaning that the representative point for a given level 2 node could still be slightly outside of these bounds. If None, returns all level 2 chunks for the root ID.
None Returns:
Type Description list of list Edge list for level 2 chunked graph. Each element of the list is an edge, and each edge is a list of two node IDs (source and target).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.remesh_level2_chunks","title":"remesh_level2_chunks(chunk_ids)","text":"Submit specific level 2 chunks to be remeshed in case of a problem.
Parameters:
Name Type Description Default chunk_ids list List of level 2 chunk IDs.
required"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_operation_details","title":"get_operation_details(operation_ids)","text":"Get the details of a list of operations.
Parameters:
Name Type Description Default operation_ids Iterable[int] List/array of operation IDs.
required Returns:
Type Description dict of str to dict A dict of dicts of operation info, keys are operation IDs (as strings), values are a dictionary of operation info for the operation. These dictionaries contain the following keys:
- \"added_edges\"/\"removed_edges\": list of list of int
- List of edges added (if a merge) or removed (if a split) by this operation. Each edge is a list of two supervoxel IDs (source and target).
- \"roots\": list of int
- List of root IDs that were created by this operation.
- \"sink_coords\": list of list of int
- List of sink coordinates for this operation. The sink is one of the points placed by the user when specifying the operation. Each sink coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
- \"source_coords\": list of list of int
- List of source coordinates for this operation. The source is one of the points placed by the user when specifying the operation. Each source coordinate is a list of three integers (x, y, z), corresponding to spatial coordinates in segmentation voxel space.
- \"timestamp\": str
- Timestamp of the operation.
- \"user\": str
- User ID number who performed the operation (as a string).
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_lineage_graph","title":"get_lineage_graph(root_id, timestamp_past=None, timestamp_future=None, as_nx_graph=False, exclude_links_to_future=False, exclude_links_to_past=False)","text":"Returns the lineage graph for a root ID, optionally cut off in the past or the future.
Each change in the chunked graph creates a new root ID for the object after that change. This function returns a graph of all root IDs for a given object, tracing the history of the object in terms of merges and splits.
Parameters:
Name Type Description Default root_id int Object root ID.
required timestamp_past datetime or None Cutoff for the lineage graph backwards in time. By default, None.
None timestamp_future datetime or None Cutoff for the lineage graph going forwards in time. By default, uses the timestamp property for this client, which defaults to the current time.
None as_nx_graph If True, a NetworkX graph is returned.
False exclude_links_to_future If True, links from nodes before timestamp_future to after timestamp_future are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False exclude_links_to_past If True, links from nodes before timestamp_past to after timestamp_past are removed. If False, the link(s) which has one node before timestamp and one node after timestamp is kept.
False Returns:
Type Description dict Dictionary describing the lineage graph and operations for the root ID. Not returned if as_nx_graph is True. The dictionary contains the following keys:
- \"directed\" : bool
- Whether the graph is directed.
- \"graph\" : dict
- Dictionary of graph attributes.
- \"links\" : list of dict
- Each element of the list is a dictionary describing an edge in the lineage graph as \"source\" and \"target\" keys.
- \"multigraph\" : bool
- Whether the graph is a multigraph.
- \"nodes\" : list of dict
- Each element of the list is a dictionary describing a node in the lineage graph, usually with \"id\", \"timestamp\", and \"operation_id\" keys.
DiGraph NetworkX directed graph of the lineage graph. Only returned if as_nx_graph is True.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_latest_roots","title":"get_latest_roots(root_id, timestamp=None, timestamp_future=None)","text":"Returns root IDs that are related to the given root_id at a given timestamp. Can be used to find the \"latest\" root IDs associated with an object.
Parameters:
Name Type Description Default root_id int Object root ID.
required timestamp datetime or None Timestamp of where to query IDs from. If None, uses the timestamp property for this client, which defaults to the current time.
None timestamp_future datetime or None DEPRECATED name, use timestamp instead. Timestamp to suggest IDs from (note can be in the past relative to the root). By default, None.
None Returns:
Type Description ndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_original_roots","title":"get_original_roots(root_id, timestamp_past=None)","text":"Returns root IDs that are the latest successors of a given root ID.
Parameters:
Name Type Description Default root_id int Object root ID.
required timestamp_past datetime or None Cutoff for the search going backwards in time. By default, None.
None Returns:
Type Description ndarray 1d array with all latest successors.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_latest_roots","title":"is_latest_roots(root_ids, timestamp=None)","text":"Check whether these root IDs are still a root at this timestamp.
Parameters:
Name Type Description Default root_ids array-like of int Root IDs to check.
required timestamp datetime Timestamp to check whether these IDs are valid root IDs in the chunked graph. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description np.array of bool Array of whether these are valid root IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.suggest_latest_roots","title":"suggest_latest_roots(root_id, timestamp=None, stop_layer=None, return_all=False, return_fraction_overlap=False)","text":"Suggest latest roots for a given root id, based on overlap of component chunk IDs. Note that edits change chunk IDs, and so this effectively measures the fraction of unchanged chunks at a given chunk layer, which sets the size scale of chunks. Higher layers are coarser.
Parameters:
Name Type Description Default root_id int Root ID of the potentially outdated object.
required timestamp datetime Datetime at which \"latest\" roots are being computed, by default None. If None, uses the timestamp property for this client, which defaults to the current time. Note that this has to be a timestamp after the creation of the root_id.
None stop_layer int Chunk level at which to compute overlap, by default None. No value will take the 4th from the top layer, which emphasizes speed and works well for larger objects. Lower values are slower but more fine-grained. Values under 2 (i.e. supervoxels) are not recommended except in extremely fine grained scenarios.
None return_all bool If True, return all current IDs sorted from most overlap to least, by default False. If False, only the top is returned.
False return_fraction_overlap bool If True, return all fractions sorted by most overlap to least, by default False. If False, only the top value is returned.
False"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.is_valid_nodes","title":"is_valid_nodes(node_ids, start_timestamp=None, end_timestamp=None)","text":"Check whether nodes are valid for given timestamp range.
Valid is defined as existing in the chunked graph. This makes no statement about these IDs being roots, supervoxel or anything in-between. It also does not take into account whether a root ID has since been edited.
Parameters:
Name Type Description Default node_ids array-like of int Node IDs to check.
required start_timestamp datetime Timestamp to check whether these IDs were valid after this timestamp. Defaults to None (assumes now).
None end_timestamp datetime Timestamp to check whether these IDs were valid before this timestamp. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description np.array of bool Array of whether these are valid IDs.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_root_timestamps","title":"get_root_timestamps(root_ids, latest=False, timestamp=None)","text":"Retrieves timestamps when roots where created.
Parameters:
Name Type Description Default root_ids Iterable of root IDs to query.
required latest bool If False, returns the first timestamp that the root_id was valid for each root ID. If True, returns the newest/latest timestamp for each root ID. Note, this will return the timestamp at which the query was run when the root is currently valid. This means that you will get a different answer if you make this same query at a later time if you don't specify a timestamp parameter.
False timestamp datetime Timestamp to query when using latest=True. Use this to provide consistent results for a particular timestamp. If an ID is still valid at a point in the future past this timestamp, the query will still return this timestamp as the latest moment in time. An error will occur if you provide a timestamp for which the root ID is not valid. If None, uses the timestamp property for this client, which defaults to the current time.
None Returns:
Type Description np.array of datetime.datetime Array of timestamps when root_ids were created.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_past_ids","title":"get_past_ids(root_ids, timestamp_past=None, timestamp_future=None)","text":"For a set of root IDs, get the list of IDs at a past or future time point that could contain parts of the same object.
Parameters:
Name Type Description Default root_ids Iterable of int Iterable of root IDs to query.
required timestamp_past datetime or None Time of a point in the past for which to look up root ids. Default is None.
None timestamp_future datetime or None Time of a point in the future for which to look up root ids. Not implemented on the server currently. Default is None.
None Returns:
Type Description dict Dict with keys \"future_id_map\" and \"past_id_map\". Each is a dict whose keys are the supplied root_ids and whose values are the list of related root IDs at timestamp_past/timestamp_future.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_delta_roots","title":"get_delta_roots(timestamp_past, timestamp_future=datetime.datetime.now(datetime.timezone.utc))","text":"Get the list of roots that have changed between timetamp_past and timestamp_future.
Parameters:
Name Type Description Default timestamp_past datetime Past timepoint to query
required timestamp_future datetime Future timepoint to query. Defaults to datetime.datetime.now(datetime.timezone.utc).
now(utc) Returns:
Name Type Description old_roots np.ndarray of np.int64 Roots that have expired in that interval.
new_roots np.ndarray of np.int64 Roots that are new in that interval.
"},{"location":"api/chunkedgraph/#caveclient.chunkedgraph.ChunkedGraphClient.get_oldest_timestamp","title":"get_oldest_timestamp()","text":"Get the oldest timestamp in the database.
Returns:
Type Description datetime Oldest timestamp in the database.
"},{"location":"api/client/","title":"caveclient.CAVEclient","text":"A manager for all clients sharing common datastack and authentication information.
This client wraps all the other clients and keeps track of the things that need to be consistent across them. To instantiate a client:
from caveclient import CAVEclient\n\nclient = CAVEclient(datastack_name='my_datastack',\n server_address='www.myserver.com',\n auth_token_file='~/.mysecrets/secrets.json')\n
Then
client.annotation is an AnnotationClient (see client.annotation)client.auth is an AuthClient (see client.auth)client.chunkedgraph is a ChunkedGraphClient (see client.chunkedgraph)client.info is an InfoServiceClient (see client.info)client.l2cache is an L2CacheClient (see client.l2cache)client.materialize is a MaterializationClient (see client.materialize)client.skeleton is a SkeletonClient (see client.skeleton)client.schema is a SchemaClient (see client.schema)client.state is a neuroglancer JSONService (see client.state)
All subclients are loaded lazily and share the same datastack name, server address, and auth tokens where used.
Parameters:
Name Type Description Default datastack_name str Datastack name for the services. Almost all services need this and will not work if it is not passed.
None server_address str or None URL of the framework server. If None, chooses the default server global.daf-apis.com. Optional, defaults to None.
None auth_token_file str or None Path to a json file containing the auth token. If None, uses the default location. See Auth client documentation. Optional, defaults to None.
default_token_file auth_token_key str Dictionary key for the token in the the JSON file. Optional, default is 'token'.
'token' auth_token str or None Direct entry of an auth token. If None, uses the file arguments to find the token. Optional, default is None.
None max_retries int or None Sets the default number of retries on failed requests. Optional, by default 2.
None pool_maxsize int or None Sets the max number of threads in a requests pool, although this value will be exceeded if pool_block is set to False. Optional, uses requests defaults if None.
None pool_block If True, prevents the number of threads in a requests pool from exceeding the max size. Optional, uses requests defaults (False) if None.
None desired_resolution Iterable[float] or None If given, should be a list or array of the desired resolution you want queries returned in useful for materialization queries.
None info_cache Pre-computed info cache, bypassing the lookup of datastack info from the info service. Should only be used in cases where this information is cached and thus repetitive lookups can be avoided.
None version Optional[int] The default materialization version of the datastack to use. If None, the latest version is used. Optional, defaults to None.
None See Also set_session_defaults
get_session_defaults
Methods:
Name Description change_auth Change the authentication token and reset services.
Attributes:
Name Type Description annotation AnnotationClient A client for the annotation service. See client.annotation
auth AuthClient A client for the auth service. See client.auth for more information.
chunkedgraph ChunkedGraphClient A client for the chunkedgraph service. See client.chunkedgraph
datastack_name str The name of the datastack for the client.
info InfoServiceClient A client for the info service. See client.info for more information.
l2cache L2CacheClient A client for the L2 cache service. See client.l2cache
materialize MaterializationClient A client for the materialization service. See client.materialize
schema SchemaClient A client for the EM Annotation Schemas service. See client.schema
server_address The server address for the client.
skeleton SkeletonClient A client for the skeleton service. See client.skeleton
state JSONService A client for the neuroglancer state service. See client.state
timestamp Optional[datetime] The default timestamp to use for queries which rely on a timestamp.
version Optional[int] The default materialization version of the datastack to use for queries which
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.annotation","title":"annotation: AnnotationClient property","text":"A client for the annotation service. See client.annotation for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.auth","title":"auth: AuthClient property","text":"A client for the auth service. See client.auth for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.chunkedgraph","title":"chunkedgraph: ChunkedGraphClient property","text":"A client for the chunkedgraph service. See client.chunkedgraph for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.datastack_name","title":"datastack_name: str property","text":"The name of the datastack for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.info","title":"info: InfoServiceClient property","text":"A client for the info service. See client.info for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.l2cache","title":"l2cache: L2CacheClient property","text":"A client for the L2 cache service. See client.l2cache for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.materialize","title":"materialize: MaterializationClient property","text":"A client for the materialization service. See client.materialize for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.schema","title":"schema: SchemaClient property","text":"A client for the EM Annotation Schemas service. See client.schema for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.server_address","title":"server_address property","text":"The server address for the client.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.skeleton","title":"skeleton: SkeletonClient property","text":"A client for the skeleton service. See client.skeleton for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.state","title":"state: JSONService property","text":"A client for the neuroglancer state service. See client.state for more information.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.timestamp","title":"timestamp: Optional[datetime] property","text":"The default timestamp to use for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.version","title":"version: Optional[int] property writable","text":"The default materialization version of the datastack to use for queries which expect a version. Also sets the timestamp to the corresponding timestamp of the version for queries which rely on a timestamp.
"},{"location":"api/client/#caveclient.frameworkclient.CAVEclientFull.change_auth","title":"change_auth(auth_token_file=None, auth_token_key=None, auth_token=None)","text":"Change the authentication token and reset services.
Parameters:
Name Type Description Default auth_token_file str New auth token json file path, by default None, which defaults to the existing state.
None auth_token_key str New dictionary key under which the token is stored in the json file, by default None, which defaults to the existing state.
None auth_token str Direct entry of a new token, by default None.
None"},{"location":"api/config/","title":"Configuration","text":""},{"location":"api/config/#caveclient.set_session_defaults","title":"caveclient.set_session_defaults(max_retries=3, pool_block=False, pool_maxsize=10, backoff_factor=0.1, backoff_max=120, status_forcelist=(502, 503, 504))","text":"Set global default values to configure how all clients will communicate with servers. Should be done prior to initializing a client.
Note that these values are only used when not set at the client level.
Parameters:
Name Type Description Default max_retries int The maximum number of retries each connection should attempt. Set to 0 to fail on the first retry.
3 pool_block bool Whether the connection pool should block for connections.
False pool_maxsize int The maximum number of connections to save in the pool.
10 backoff_factor Union[float, int] A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). The query will sleep for: {backoff factor} * (2 ^ ({number of total retries} - 1)) seconds. For example, if the backoff_factor is 0.1, then will sleep for [0.0s, 0.2s, 0.4s, 0.8s, \u2026] between retries. No backoff will ever be longer than backoff_max.
0.1 backoff_max Union[float, int] The maximum backoff time.
120 status_forcelist Optional[Collection] A set of integer HTTP status codes that we should force a retry on.
(502, 503, 504) Usage from caveclient import set_session_defaults\n\nset_session_defaults(\n max_retries=5, # would increase the default number of retries\n backoff_factor=0.5, # would increase the default backoff factor between retries\n backoff_max=240, # would increase the default maximum backoff time\n status_forcelist=(502, 503, 504, 505), # would add 505 to the default list\n)\n\nset_session_defaults() # would revert all defaults to their original values\n
Notes Calling this function will set the default values for all clients created after the call.
Calling this function with any arguments missing will reset that value to the default value.
See Also: urllib3.util.Retry
requests.adapters.HTTPAdapter
"},{"location":"api/config/#caveclient.get_session_defaults","title":"caveclient.get_session_defaults()","text":"Get the current default values for session configuration.
Returns:
Type Description dict Dictionary of current default values for session configuration.
"},{"location":"api/datastack_lookup/","title":"Datastack lookup","text":""},{"location":"api/datastack_lookup/#caveclient.datastack_lookup.reset_server_address_cache","title":"caveclient.datastack_lookup.reset_server_address_cache(datastack, filename=None)","text":"Remove one or more datastacks from the datastack-to-server cache.
Parameters:
Name Type Description Default datastack str or list of str Datastack names to remove from the cache, by default None
required filename str Name of the cache file, by default None
None"},{"location":"api/info/","title":"client.info","text":"Client for interacting with the info service.
Methods:
Name Description annotation_endpoint AnnotationEngine endpoint for a dataset.
get_aligned_volume_info Gets the info record for a aligned_volume
get_datastack_info Gets the info record for a datastack
get_datastacks Query which datastacks are available at the info service
get_datastacks_by_aligned_volume Lookup what datastacks are associated with this aligned volume
image_cloudvolume Generate a cloudvolume instance based on the image source, using authentication if needed and
image_source Cloud path to the imagery for the dataset
refresh_stored_data Reload the stored info values from the server.
segmentation_cloudvolume Generate a cloudvolume instance based on the segmentation source, using authentication if needed and
segmentation_source Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
synapse_segmentation_source Cloud path to the synapse segmentation for a dataset
viewer_resolution Get the viewer resolution metadata for this datastack
viewer_site Get the base Neuroglancer URL for the dataset
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.annotation_endpoint","title":"annotation_endpoint(datastack_name=None, use_stored=True)","text":"AnnotationEngine endpoint for a dataset.
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Description str Location of the AnnotationEngine
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_aligned_volume_info","title":"get_aligned_volume_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a aligned_volume
Parameters:
Name Type Description Default datastack_name str datastack_name to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that dataset, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Description dict or None The complete info record for the aligned_volume
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastack_info","title":"get_datastack_info(datastack_name=None, use_stored=True)","text":"Gets the info record for a datastack
Parameters:
Name Type Description Default datastack_name str datastack to look up. If None, uses the one specified by the client. By default None
None use_stored bool If True and the information has already been queried for that datastack, then uses the cached version. If False, re-queries the infromation. By default True
True Returns:
Type Description dict or None The complete info record for the datastack
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks","title":"get_datastacks()","text":"Query which datastacks are available at the info service
Returns:
Type Description list List of datastack names
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.get_datastacks_by_aligned_volume","title":"get_datastacks_by_aligned_volume(aligned_volume=None)","text":"Lookup what datastacks are associated with this aligned volume
Args: aligned_volume (str, optional): aligned volume to lookup. Defaults to None.
Raises: ValueError: if no aligned volume is specified
Returns: list: a list of datastack string
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_cloudvolume","title":"image_cloudvolume(**kwargs)","text":"Generate a cloudvolume instance based on the image source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.image_source","title":"image_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the imagery for the dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Description str Formatted cloud path to the flat segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.refresh_stored_data","title":"refresh_stored_data()","text":"Reload the stored info values from the server.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_cloudvolume","title":"segmentation_cloudvolume(use_client_secret=True, **kwargs)","text":"Generate a cloudvolume instance based on the segmentation source, using authentication if needed and sensible default values for reading CAVE resources. By default, fill_missing is True and bounded is False. All keyword arguments are passed onto the CloudVolume initialization function, and defaults can be overridden.
Requires cloudvolume to be installed, which is not included by default.
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.segmentation_source","title":"segmentation_source(datastack_name=None, format_for='raw', use_stored=True)","text":"Cloud path to the chunkgraph-backed Graphene segmentation for a dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"graphene://https://\" type path is used If 'neuroglancer', a \"graphene://https://\" type path is used, as needed by Neuroglancer.
'raw' Returns:
Type Description str Formatted cloud path to the Graphene segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.synapse_segmentation_source","title":"synapse_segmentation_source(datastack_name=None, use_stored=True, format_for='raw')","text":"Cloud path to the synapse segmentation for a dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the dataset to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True format_for 'raw', 'cloudvolume', or 'neuroglancer' Formats the path for different uses. If 'raw' (default), the path in the InfoService is passed along. If 'cloudvolume', a \"precomputed://gs://\" type path is converted to a full https URL. If 'neuroglancer', a full https URL is converted to a \"precomputed://gs://\" type path.
'raw' Returns:
Type Description str Formatted cloud path to the synapse segmentation
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_resolution","title":"viewer_resolution(datastack_name=None, use_stored=True)","text":"Get the viewer resolution metadata for this datastack
Parameters:
Name Type Description Default datastack_name If None use the default one configured in the client
None use_stored Use the cached value, if False go get a new value from server
True Returns:
Type Description array Voxel resolution as a len(3) np.array
"},{"location":"api/info/#caveclient.infoservice.InfoServiceClient.viewer_site","title":"viewer_site(datastack_name=None, use_stored=True)","text":"Get the base Neuroglancer URL for the dataset
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack to look up. If None, uses the value specified by the client. Default is None.
None use_stored bool If True, uses the cached value if available. If False, re-queries the InfoService. Default is True.
True Returns:
Type Description str Base URL for the Neuroglancer viewer
"},{"location":"api/l2cache/","title":"client.l2cache","text":"Client for interacting with the level2 cache service.
Methods:
Name Description cache_metadata Retrieves the meta data for the cache
get_l2data Gets the attributed statistics data for L2 ids.
get_l2data_table Gets the attributed statistics data for L2 ids, returned as a dataframe.
has_cache Checks if the l2 cache is available for the dataset
table_mapping Retrieves table mappings for l2 cache.
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.cache_metadata","title":"cache_metadata()","text":"Retrieves the meta data for the cache
Returns:
Type Description dict keys are attribute names, values are datatypes
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data","title":"get_l2data(l2_ids, attributes=None)","text":"Gets the attributed statistics data for L2 ids.
Parameters:
Name Type Description Default l2_ids list or ndarray A list of level 2 ids.
required attributes list A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:
area_nm2chunk_intersect_countmax_dt_nmmean_dt_nmpcapca_valrep_coord_nmsize_nm3
See the tutorial for more description.
None Returns:
Type Description dict Keys are l2 ids, values are data.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.get_l2data_table","title":"get_l2data_table(l2_ids, attributes=None, split_columns=True)","text":"Gets the attributed statistics data for L2 ids, returned as a dataframe.
Parameters:
Name Type Description Default l2_ids list or ndarray A list of level 2 ids.
required attributes list A list of attributes to retrieve. Defaults to None, which will return all that are available. Available stats are:
area_nm2chunk_intersect_countmax_dt_nmmean_dt_nmpcapca_valrep_coord_nmsize_nm3
See the tutorial for more description.
None split_columns bool Whether to split columns with multiple values into separate columns.
True Returns:
Type Description DataFrame A pandas dataframe with the requested attributes as columns and indexed by l2_id.
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.has_cache","title":"has_cache(datastack_name=None)","text":"Checks if the l2 cache is available for the dataset
Parameters:
Name Type Description Default datastack_name str The name of the datastack to check, by default None (if None, uses the client's datastack)
None Returns:
Type Description bool True if the l2 cache is available, False otherwise
"},{"location":"api/l2cache/#caveclient.l2cache.L2CacheClient.table_mapping","title":"table_mapping()","text":"Retrieves table mappings for l2 cache.
Returns:
Type Description dict keys are pcg table names, values are dicts with fields l2cache_id and cv_path.
"},{"location":"api/materialize/","title":"client.materialize","text":"Client for interacting with the materialization engine.
Methods:
Name Description get_table_metadata Get metadata about a table
get_tables Gets a list of table names for a datastack
get_tables_metadata Get metadata about tables
get_timestamp Get datetime.datetime timestamp for a materialization version.
get_unique_string_values Get unique string values for a table
get_version_metadata Get metadata about a version
get_versions Get the versions available
get_versions_metadata Get the metadata for all the versions that are presently available and valid
get_view_metadata Get metadata for a view
get_view_schema Get schema for a view
get_view_schemas Get schema for a view
get_views Get all available views for a version
ingest_annotation_table Trigger supervoxel lookup and root ID lookup of new annotations in a table.
join_query Generic query on materialization tables
live_live_query Beta method for querying cave annotation tables with root IDs and annotations
live_query Generic query on materialization tables
lookup_supervoxel_ids Trigger supervoxel lookups of new annotations in a table.
map_filters Translate a list of filter dictionaries from a point in the
most_recent_version Get the most recent version of materialization for this datastack name
query_table Generic query on materialization tables
query_view Generic query on a view
synapse_query Convenience method for querying synapses.
Attributes:
Name Type Description cg_client The chunked graph client.
datastack_name The name of the datastack.
homepage HTML The homepage for the materialization engine.
server_version Optional[Version] The version of the service running on the remote server. Note that this
tables TableManager The table manager for the materialization engine.
version int The version of the materialization. Can be used to set up the
views ViewManager The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.cg_client","title":"cg_client property","text":"The chunked graph client.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.datastack_name","title":"datastack_name property","text":"The name of the datastack.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.homepage","title":"homepage: HTML property","text":"The homepage for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.tables","title":"tables: TableManager property","text":"The table manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.version","title":"version: int property writable","text":"The version of the materialization. Can be used to set up the client to default to a specific version when timestamps or versions are not specified in queries. If not set, defaults to the most recent version.
Note that if this materialization client is attached to a CAVEclient, the version must be set at the CAVEclient level.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.views","title":"views: ViewManager property","text":"The view manager for the materialization engine.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_table_metadata","title":"get_table_metadata(table_name, datastack_name=None, version=None, log_warning=True)","text":"Get metadata about a table
Parameters:
Name Type Description Default table_name str name of table to mark for deletion
required datastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Description dict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables","title":"get_tables(datastack_name=None, version=None)","text":"Gets a list of table names for a datastack
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None Returns:
Type Description list List of table names
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_tables_metadata","title":"get_tables_metadata(datastack_name=None, version=None, log_warning=True)","text":"Get metadata about tables
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack_name. If None, uses the one specified in the client.
None version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to print out warnings to the logger. Defaults to True.
True Returns:
Type Description dict Metadata dictionary for table
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_timestamp","title":"get_timestamp(version=None, datastack_name=None)","text":"Get datetime.datetime timestamp for a materialization version.
Parameters:
Name Type Description Default version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Description datetime Datetime when the materialization version was frozen.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_unique_string_values","title":"get_unique_string_values(table, datastack_name=None)","text":"Get unique string values for a table
Parameters:
Name Type Description Default table str Table to query
required datastack_name Optional[str] Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Description dict[str] A dictionary of column names and their unique values
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_version_metadata","title":"get_version_metadata(version=None, datastack_name=None)","text":"Get metadata about a version
Parameters:
Name Type Description Default version int or None The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None Returns:
Type Description dict Dictionary of metadata about the version
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions","title":"get_versions(datastack_name=None, expired=False)","text":"Get the versions available
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Description dict Dictionary of versions available
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_versions_metadata","title":"get_versions_metadata(datastack_name=None, expired=False)","text":"Get the metadata for all the versions that are presently available and valid
Parameters:
Name Type Description Default datastack_name str or None Datastack name, by default None. If None, defaults to the value set in the client.
None expired bool Whether to include expired versions, by default False.
False Returns:
Type Description list[dict] List of metadata dictionaries
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_metadata","title":"get_view_metadata(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get metadata for a view
Parameters:
Name Type Description Default view_name str Name of view to query.
required materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Description dict Metadata of view
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schema","title":"get_view_schema(view_name, materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Default view_name str Name of view to query.
required materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None log_warning bool Whether to log warnings.
True Returns:
Type Description dict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_view_schemas","title":"get_view_schemas(materialization_version=None, datastack_name=None, log_warning=True)","text":"Get schema for a view
Parameters:
Name Type Description Default materialization_version Optional[int] Version to query. If None, will use version set by client.
None log_warning bool Whether to log warnings.
True Returns:
Type Description dict Schema of view.
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.get_views","title":"get_views(version=None, datastack_name=None)","text":"Get all available views for a version
Parameters:
Name Type Description Default version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None datastack_name str Datastack to query. If None, uses the one specified in the client.
None Returns:
Type Description list List of views
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.ingest_annotation_table","title":"ingest_annotation_table(table_name, datastack_name=None)","text":"Trigger supervoxel lookup and root ID lookup of new annotations in a table.
Parameters:
Name Type Description Default table_name str Table to trigger
required datastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Description dict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.join_query","title":"join_query(tables, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, suffixes=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Default tables list of lists with length 2 or 'str' list of two lists: first entries are table names, second entries are the columns used for the join.
required filter_in_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are allowed entries, by default None
None filter_out_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are specified entry, by default None
None filter_greater_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict of dicts outer layer: keys are table names inner layer: keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict of dicts outer layer: keys are table names, inner layer: keys are column names. Values are bounding boxes as [[min_x, min_y,min_z],[max_x, max_y, max_z]], expressed in units of the voxel_resolution of this dataset. Defaults to None.
None filter_regex_dict dict of dicts outer layer: keys are table names. inner layer: keys are column names, values are regex strings. Defaults to None
None select_columns dict of lists of str keys are table names,values are the list of columns from that table. Defaults to None, which will select all tables. Will be passed to server as select_column_maps. Passing a list will be passed as select_columns which is deprecated.
None offset int result offset to use. Defaults to None. Will only return top K results.
None limit int maximum results to return (server will set upper limit, see get_server_config)
None suffixes dict suffixes to use for duplicate columns, keys are table names, values are the suffix
None datastack_name str datastack to query. If None defaults to one specified in client.
None return_df bool whether to return as a dataframe default True, if False, data is returned as json (slower)
True split_positions bool whether to break position columns into x,y,z columns default False, if False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool toggle to return metadata If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True desired_resolution Iterable What resolution to convert position columns to. Defaults to None will use defaults.
None random_sample int if given, will do a tablesample of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type Description DataFrame a pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_live_query","title":"live_live_query(table, timestamp, joins=None, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, metadata=True, suffixes=None, desired_resolution=None, allow_missing_lookups=False, allow_invalid_root_ids=False, random_sample=None, log_warning=True)","text":"Beta method for querying cave annotation tables with root IDs and annotations at a particular timestamp. Note: this method requires more explicit mapping of filters and selection to table as its designed to test a more general endpoint that should eventually support complex joins.
Parameters:
Name Type Description Default table str Principle table to query
required timestamp datetime Timestamp to query
required joins List of joins, where each join is a list of [table1,column1, table2, column2]
None filter_in_dict A dictionary with tables as keys, values are dicts with column keys and list values to accept.
None filter_out_dict A dictionary with tables as keys, values are dicts with column keys and list values to reject.
None filter_equal_dict A dictionary with tables as keys, values are dicts with column keys and values to equate.
None filter_greater_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive upper-bound.
None filter_less_dict A dictionary with tables as keys, values are dicts with column keys and values as exclusive lower-bound.
None filter_greater_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive upper-bound.
None filter_less_equal_dict A dictionary with tables as keys, values are dicts with column keys and values as inclusive lower-bound.
None filter_spatial_dict A dictionary with tables as keys, values are dicts with column keys and values of 2x3 list of bounds.
None filter_regex_dict A dictionary with tables as keys, values are dicts with column keys and values of regex strings.
None select_columns A dictionary with tables as keys, values are lists of columns to select.
None offset int Value to offset query by.
None limit int Limit of query.
None datastack_name str Datastack to query. Defaults to set by client.
None split_positions bool Whether to split positions into separate columns, True is faster.
False metadata bool Whether to attach metadata to dataframe.
True suffixes dict What suffixes to use on joins, keys are table_names, values are suffixes.
None desired_resolution Iterable What resolution to convert position columns to.
None allow_missing_lookups bool If there are annotations without supervoxels and root IDs yet, allow results.
False allow_invalid_root_ids bool If True, ignore root ids not valid at the given timestamp, otherwise raise an error.
False random_sample int If given, will do a table sample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type Description Results of query
Examples:
>>> from caveclient import CAVEclient\n>>> client = CAVEclient('minnie65_public_v117')\n>>> live_live_query(\"table_name\", datetime.datetime.now(datetime.timezone.utc),\n>>> joins=[[table_name, table_column, joined_table, joined_column],\n>>> [joined_table, joincol2, third_table, joincol_third]]\n>>> suffixes={\n>>> \"table_name\":\"suffix1\",\n>>> \"joined_table\":\"suffix2\",\n>>> \"third_table\":\"suffix3\"\n>>> },\n>>> select_columns= {\n>>> \"table_name\":[ \"column\",\"names\"],\n>>> \"joined_table\":[\"joined_colum\"]\n>>> },\n>>> filter_in_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[included,values]\n>>> }\n>>> },\n>>> filter_out_dict= {\n>>> \"table_name\":{\n>>> \"column_name\":[excluded,values]\n>>> }\n>>> },\n>>> filter_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_greater_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_less_equal_dict\"={\n>>> \"table_name\":{\n>>> \"column_name\":value\n>>> },\n>>> filter_spatial_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": [[min_x, min_y, min_z], [max_x, max_y, max_z]]\n>>> }\n>>> filter_regex_dict\"= {\n>>> \"table_name\": {\n>>> \"column_name\": \"regex_string\"\n>>> }\n
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.live_query","title":"live_query(table, timestamp, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, split_positions=False, post_filter=True, metadata=True, merge_reference=True, desired_resolution=None, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Default table str Table to query
required timestamp datetime Time to materialize (in utc). Pass datetime.datetime.now(datetime.timezone.utc) for present time.
required filter_in_dict dict Keys are column names, values are allowed entries.
None filter_out_dict dict Keys are column names, values are not allowed entries.
None filter_equal_dict dict Keys are column names, values are specified entry.
None filter_greater_dict dict Keys are column names, values are exclusive upper-bounds.
None filter_less_dict dict Keys are column names, values are exclusive lower-bounds.
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bounds.
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bounds.
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]].
None filter_regex_dict dict Keys are column names, values are regex strings.
None select_columns list of str Columns to select.
None offset int Offset in query result.
None limit int Maximum results to return (server will set upper limit, see get_server_config).
None datastack_name str Datastack to query. If None, defaults to one specified in client.
None split_positions bool Whether to break position columns into x,y,z columns. If False data is returned as one column with [x,y,z] array (slower).
False post_filter bool Whether to filter down the result based upon the filters specified. If False, it will return the query with present root_ids in the root_id columns, but the rows will reflect the filters translated into their past IDs. So if, for example, a cell had a false merger split off since the last materialization, those annotations on that incorrect portion of the cell will be included if this is False, but will be filtered down if this is True.
True metadata bool Toggle to return metadata. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that table.
True desired_resolution Iterable Desired resolution you want all spatial points returned in. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata.
None random_sample int If given, will do a tablesample of the table to return that many annotations.
None log_warning bool Whether to log warnings.
True Returns:
Type Description DataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.lookup_supervoxel_ids","title":"lookup_supervoxel_ids(table_name, annotation_ids=None, datastack_name=None)","text":"Trigger supervoxel lookups of new annotations in a table.
Parameters:
Name Type Description Default table_name str Table to trigger
required annotation_ids list List of annotation ids to lookup. Default is None, which will trigger lookup of entire table.
None datastack_name str Datastack to trigger it. Defaults to what is set in client.
None Returns:
Type Description dict Status code of response from server
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.map_filters","title":"map_filters(filters, timestamp, timestamp_past)","text":"Translate a list of filter dictionaries from a point in the future to a point in the past
Parameters:
Name Type Description Default filters list[dict] filter dictionaries with root_ids
required timestamp datetime timestamp to query
required timestamp_past datetime timestamp to query from
required Returns:
Type Description list[dict] filter dictionaries with past root_ids
dict mapping of future root_ids to past root_ids
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.most_recent_version","title":"most_recent_version(datastack_name=None)","text":"Get the most recent version of materialization for this datastack name
Parameters:
Name Type Description Default datastack_name str or None Name of the datastack, by default None. If None, uses the one specified in the client. Will be set correctly if you are using the framework_client
None Returns:
Type Description int Most recent version of materialization for this datastack name
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_table","title":"query_table(table, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, timestamp=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None, log_warning=True)","text":"Generic query on materialization tables
Parameters:
Name Type Description Default table str Table to query
required filter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound value, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound value, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound value, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound value, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None timestamp datetime Timestamp to query, by default None. If passsed will do a live query. Error if also passing a materialization version
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None log_warning bool Whether to log warnings, by default True
True Returns:
Type Description DataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.query_view","title":"query_view(view_name, filter_in_dict=None, filter_out_dict=None, filter_equal_dict=None, filter_greater_dict=None, filter_less_dict=None, filter_greater_equal_dict=None, filter_less_equal_dict=None, filter_spatial_dict=None, filter_regex_dict=None, select_columns=None, offset=None, limit=None, datastack_name=None, return_df=True, split_positions=False, materialization_version=None, metadata=True, merge_reference=True, desired_resolution=None, get_counts=False, random_sample=None)","text":"Generic query on a view
Parameters:
Name Type Description Default view_name str View to query
required filter_in_dict dict Keys are column names, values are allowed entries, by default None
None filter_out_dict dict Keys are column names, values are not allowed entries, by default None
None filter_equal_dict dict Keys are column names, values are specified entry, by default None
None filter_greater_dict dict Keys are column names, values are exclusive upper-bound, by default None
None filter_less_dict dict Keys are column names, values are exclusive lower-bound, by default None
None filter_greater_equal_dict dict Keys are column names, values are inclusive upper-bound, by default None
None filter_less_equal_dict dict Keys are column names, values are inclusive lower-bound, by default None
None filter_spatial_dict dict Keys are column names, values are bounding boxes expressed in units of the voxel_resolution of this dataset. Bounding box is [[min_x, min_y,min_z],[max_x, max_y, max_z]], by default None
None filter_regex_dict dict Keys are column names, values are regex strings, by default None
None select_columns list of str Columns to select, by default None
None offset int Result offset to use, by default None. Will only return top K results.
None limit int Maximum results to return (server will set upper limit, see get_server_config), by default None
None datastack_name str Datastack to query, by default None. If None, defaults to one specified in client.
None return_df bool Whether to return as a dataframe, by default True. If False, data is returned as json (slower).
True split_positions bool Whether to break position columns into x,y,z columns, by default False. If False data is returned as one column with [x,y,z] array (slower)
False materialization_version int The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Toggle to return metadata (default True), by default True. If True (and return_df is also True), return table and query metadata in the df.attr dictionary.
True merge_reference bool Toggle to automatically join reference table, by default True. If True, metadata will be queries and if its a reference table it will perform a join on the reference table to return the rows of that
True desired_resolution Iterable[float] Desired resolution you want all spatial points returned in, by default None. If None, defaults to one specified in client, if that is None then points are returned as stored in the table and should be in the resolution specified in the table metadata
None get_counts bool Whether to get counts of the query, by default False
False random_sample int If given, will do a tablesample of the of the table to return that many annotations
None Returns:
Type Description DataFrame A pandas dataframe of results of query
"},{"location":"api/materialize/#caveclient.materializationengine.MaterializationClient.synapse_query","title":"synapse_query(pre_ids=None, post_ids=None, bounding_box=None, bounding_box_column='post_pt_position', timestamp=None, remove_autapses=True, include_zeros=True, limit=None, offset=None, split_positions=False, desired_resolution=None, materialization_version=None, synapse_table=None, datastack_name=None, metadata=True)","text":"Convenience method for querying synapses.
Will use the synapse table specified in the info service by default. It will also remove autapses by default. NOTE: This is not designed to allow querying of the entire synapse table. A query with no filters will return only a limited number of rows (configured by the server) and will do so in a non-deterministic fashion. Please contact your dataset administrator if you want access to the entire table.
Parameters:
Name Type Description Default pre_ids Union[int, Iterable, ndarray] Pre-synaptic cell(s) to query.
None post_ids Union[int, Iterable, ndarray] Post-synaptic cell(s) to query.
None bounding_box Optional[Union[list, ndarray]] [[min_x, min_y, min_z],[max_x, max_y, max_z]] bounding box to filter synapse locations. Expressed in units of the voxel_resolution of this dataset.
None bounding_box_column str Which synapse location column to filter by.
'post_pt_position' timestamp datetime Timestamp to query. If passed recalculate query at timestamp, do not pass with materialization_version.
None remove_autapses bool Whether to remove autapses from query results.
True include_zeros bool Whether to include synapses to/from id=0 (out of segmentation).
True limit int Number of synapses to limit. Server-side limit still applies.
None offset int Number of synapses to offset query.
None split_positions bool Whether to split positions into separate columns, True is faster.
False desired_resolution Iterable[float] List or array of the desired resolution you want queries returned in useful for materialization queries.
None materialization_version Optional[int] The version of the datastack to query. If None, will query the client version, which defaults to the most recent version.
None metadata bool Whether to attach metadata to dataframe in the df.attr dictionary.
True Returns:
Type Description DataFrame Results of query.
"},{"location":"api/schema/","title":"client.schema","text":"Client for interacting with the schema service.
Methods:
Name Description get_schemas Get the available schema types
schema_definition Get the definition of a specified schema_type
schema_definition_all Get the definition of all schema_types
schema_definition_multi Get the definition of multiple schema_types
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.get_schemas","title":"get_schemas()","text":"Get the available schema types
Returns:
Type Description list List of schema types available on the Schema service.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition","title":"schema_definition(schema_type)","text":"Get the definition of a specified schema_type
Parameters:
Name Type Description Default schema_type str Name of a schema_type
required Returns:
Type Description json Schema definition
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_all","title":"schema_definition_all()","text":"Get the definition of all schema_types
Returns:
Type Description dict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/schema/#caveclient.emannotationschemas.SchemaClient.schema_definition_multi","title":"schema_definition_multi(schema_types)","text":"Get the definition of multiple schema_types
Parameters:
Name Type Description Default schema_types list List of schema names
required Returns:
Type Description dict Dictionary of schema definitions. Keys are schema names, values are definitions.
"},{"location":"api/skeleton/","title":"client.skeleton","text":"Client for interacting with the skeleton service.
Methods:
Name Description get_skeleton Gets basic skeleton information for a datastack
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/skeleton/#caveclient.skeletonservice.SkeletonClient.get_skeleton","title":"get_skeleton(root_id, datastack_name=None, skeleton_version=0, output_format='none', log_warning=True)","text":"Gets basic skeleton information for a datastack
Parameters:
Name Type Description Default root_id int The root id of the skeleton to retrieve
required datastack_name str The name of the datastack to check
None skeleton_version int The skeleton version to generate and retrieve. Options are documented in SkeletonService. Use 0 for latest.
0 output_format string The format to retrieve. Options are:
- 'none': No return value (this can be used to generate a skeleton without retrieving it)
- 'precomputed': A cloudvolume.Skeleton object
- 'json': A dictionary
- 'jsoncompressed': A dictionary using compression for transmission (generally faster than 'json')
- 'arrays': A dictionary (literally a subset of the json response)
- 'arrayscompressed': A dictionary using compression for transmission (generally faster than 'arrays')
- 'swc': A pandas DataFrame
- 'h5': An BytesIO object containing bytes for an h5 file
'none' Returns:
Type Description Skeleton of the requested type. See output_format for details.
"},{"location":"api/state/","title":"client.state","text":"Client to interface with the JSON state service.
Methods:
Name Description build_neuroglancer_url Build a URL for a Neuroglancer deployment that will automatically retrieve specified state.
get_neuroglancer_info Get the info field from a Neuroglancer deployment
get_property_json Download a Neuroglancer JSON state
get_state_json Download a Neuroglancer JSON state
save_state_json_local Save a Neuroglancer JSON state to a JSON file locally.
upload_property_json Upload a Neuroglancer JSON state
upload_state_json Upload a Neuroglancer JSON state
Attributes:
Name Type Description server_version Optional[Version] The version of the service running on the remote server. Note that this
state_service_endpoint Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.server_version","title":"server_version: Optional[Version] property","text":"The version of the service running on the remote server. Note that this refers to the software running on the server and has nothing to do with the version of the datastack itself.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.state_service_endpoint","title":"state_service_endpoint property","text":"Endpoint URL for posting JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.build_neuroglancer_url","title":"build_neuroglancer_url(state_id, ngl_url=None, target_site=None, static_url=False, format_properties=False)","text":"Build a URL for a Neuroglancer deployment that will automatically retrieve specified state. If the datastack is specified, this is prepopulated from the info file field \"viewer_site\". If no ngl_url is specified in either the function or the client, a fallback neuroglancer deployment is used.
Parameters:
Name Type Description Default state_id int State id to retrieve
required ngl_url str Base url of a neuroglancer deployment. If None, defaults to the value for the datastack or the client. As a fallback, a default deployment is used.
None target_site seunglab or cave - explorer or mainline or None Set this to 'seunglab' for a seunglab deployment, or either 'cave-explorer'/'mainline' for a google main branch deployment. If None, checks the info field of the neuroglancer endpoint to determine which to use. Default is None.
None static_url bool If True, treats \"state_id\" as a static URL directly to the JSON and does not use the state service.
False format_properties bool If True, formats the url as a segment_properties info file
False Returns:
Type Description str The full URL requested
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_neuroglancer_info","title":"get_neuroglancer_info(ngl_url=None)","text":"Get the info field from a Neuroglancer deployment
Parameters:
Name Type Description Default ngl_url str(optional) URL to a Neuroglancer deployment. If None, defaults to the value for the datastack or the client.
None Returns:
Type Description dict JSON-formatted info field from the Neuroglancer deployment
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_property_json","title":"get_property_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Default state_id int ID of a JSON state uploaded to the state service.
required Returns:
Type Description dict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.get_state_json","title":"get_state_json(state_id)","text":"Download a Neuroglancer JSON state
Parameters:
Name Type Description Default state_id int ID of a JSON state uploaded to the state service.
required Returns:
Type Description dict JSON specifying a Neuroglancer state.
"},{"location":"api/state/#caveclient.jsonservice.JSONService.save_state_json_local","title":"save_state_json_local(json_state, filename, overwrite=False)","text":"Save a Neuroglancer JSON state to a JSON file locally.
Parameters:
Name Type Description Default json_state dict Dict representation of a neuroglancer state
required filename str Filename to save the state to
required overwrite bool Whether to overwrite the file if it exists. Default False.
False Returns:
Type Description None"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_property_json","title":"upload_property_json(property_json, state_id=None, max_size=2500000)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Default propery_json dict Dict representation of a neuroglancer segment properties json
required state_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None max_size Optional[int] Maximum size in bytes for the data to upload. Default is 2.5MB. Set to None for no limit.
2500000 Returns:
Type Description int state_id of the uploaded JSON state
"},{"location":"api/state/#caveclient.jsonservice.JSONService.upload_state_json","title":"upload_state_json(json_state, state_id=None, timestamp=None)","text":"Upload a Neuroglancer JSON state
Parameters:
Name Type Description Default json_state dict Dict representation of a neuroglancer state
required state_id int ID of a JSON state uploaded to the state service. Using a state_id is an admin feature.
None timestamp Timestamp for json state date. Requires state_id.
None Returns:
Type Description int state_id of the uploaded JSON state
"},{"location":"tutorials/","title":"Getting Started","text":"AnnotationFramework client is a package for simplifying interactions with HTML services associated with the CAVE (Connectome Annotation Versioning Engine).
For a larger introduction to CAVE and its services, see the main GitHub organization site: https://github.com/CAVEconnectome
"},{"location":"tutorials/#installation","title":"Installation","text":"The CAVEclient can be installed with pip:
$ pip install caveclient\n
"},{"location":"tutorials/#assumptions","title":"Assumptions","text":"The code is setup to work flexibly with any deployment of these services, but you need to specify the server_address if that address is not https://globalv1.daf-apis.com/ for each client when initializing it. Similarly, the clients can query the info service for metadata to simplify the interaction with a datastack, but you have to specify a datastack name.
"},{"location":"tutorials/advanced/","title":"Advanced Usage","text":""},{"location":"tutorials/advanced/#changing-session-configurations","title":"Changing session configurations","text":"It is possible to change details of how a client talks to the various servers it needs to interface with. For instance, the CAVEclient will attempt to retry specific failed requests to the server, but will only try a specific number of times, and will only wait specific amounts of time between retries. These values can be changed via the set_session_defaults method. For instance, to change the number of retries to 5, and to increase the delay between subsequent retries, you could do:
from caveclient import set_session_defaults\n\nset_session_defaults(max_retries=5, backoff_factor=0.5)\n
Note that this needs to happen before initializing the client for this to work properly. Some of these parameters are also adjustable at the client level.
To view the current session defaults, you can use the get_session_defaults method:
from caveclient import get_session_defaults\n\nclient.get_session_defaults()\n
More information on the available parameters can be found in the API documentation.
"},{"location":"tutorials/annotation/","title":"Annotation","text":"The AnnotationClient is used to interact with the AnnotationEngine service to create tables from existing schema, upload new data, and download existing annotations. Note that annotations in the AnnotationEngine are not linked to any particular segmentation, and thus do not include any root ids. An annotation client is accessed with client.annotation.
"},{"location":"tutorials/annotation/#getting-existing-tables","title":"Getting existing tables","text":"A list of the existing tables for the datastack can be found with get_tables().
all_tables = client.annotation.get_tables()\nall_tables[0]\n
Each table has three main properties that can be useful to know:
table_name : The table name, used to refer to it when uploading or downloading annotations. This is also passed through to the table in the Materialized database.schema_name : The name of the table's schema from EMAnnotationSchemas (see below).max_annotation_id : An upper limit on the number of annotations already contained in the table.
"},{"location":"tutorials/annotation/#downloading-annotations","title":"Downloading annotations","text":"You can download the JSON representation of a data point through the get_annotation() method. This can be useful if you need to look up information on unmaterialized data, or to see what a properly templated annotation looks like.
table_name = all_tables[0]['table_name'] # 'ais_analysis_soma'\nannotation_id = 100\nclient.annotation.get_annotation(annotation_ids=annotation_id, table_name=table_name)\n
"},{"location":"tutorials/annotation/#create-a-new-table","title":"Create a new table","text":"One can create a new table with a specified schema with the create_table() method:
client.annotation.create_table(table_name='test_table',\n schema_name='microns_func_coreg',\n voxel_resolution = [1,1,1],\n description=\"some text to describe your table\")\n
The voxel resolution is the units your position columns will be uploaded in [1,1,1] would imply a nm location, where as [4,4,40] would correspond to voxels of that size. If you are uploading points from a neuroglancer session, you want this to match the units of that neuroglancer view.
Note there are some optional metadata parameters to create_table()
notice_text : This is text that will show up to users who access this data as a warning. This could be used to warn users that the data is not complete or checked yet, or to advertise that a particular publication should be cited when using this table.read_permission : one of \"PRIVATE\" which means only you can read data in this table. \"PUBLIC\" (default) which means anyone can read this table that has read permissions to this dataset. So if and only if you can read the segmentation results of this data, you can read this table. \"GROUP\" which means that you must share a common group with this user for them to be able to read. We need to make a way to discover what groups you are in and who you share groups with.write_permission: one of \"PRIVATE\" (default), which means only you can write to this table. \"PUBLIC\" which means anyone can write to this table that has write permissions to this dataset. Note although this means anyone can add data, no annotations are ever truly overwritten. \"GROUP\" which means that you must share a common group with this user for them to be able to write. We need to make a way to discover what groups you are in and who you share groups with.
If you change your mind about what you want for metadata, some but not all fields can be updated with update_metadata(). This includes the description, the notice_text, and the permissions, but not the name, schema or voxel resolution.
# to update description\nclient.annotation.update_metadata(table_name='test_table',\n description=\"a new description for my table\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n notice_text=\"This table isn't done yet, don't trust it. Contact me\")\n\n# to make your table readable by anybody who can read this dataset\nclient.annotation.update_metadata(table_name='test_table',\n read_permisison=\"PUBLIC\")\n
New data can be generated as a dict or list of dicts following the schema and uploaded with post_annotation. For example, a microns_func_coreg point needs to have:
type set to microns_func_coregpt set to a dict with position as a key and the xyz location as a value.func_id set to an integer.
The following could would create a new annotation and then upload it to the service. Note that you get back the annotation id(s) of what you uploaded.
new_data = {'type': 'microns_func_coreg',\n 'pt': {'position': [1,2,3]},\n 'func_id': 0}\nclient.annotation.post_annotation(table_name='test_table', data=[new_data])\n
There are methods to simplify annotation uploads if you have a pandas dataframe whose structure mirrors the struction of the annotation schema you want to upload
import pandas as pd\n\ndf = pd.DataFrame([{'id':0,\n 'type': 'microns_func_coreg',\n 'pt_position': [1,2,3]},\n 'func_id': 0},\n {'id':1,\n 'type': 'microns_func_coreg',\n 'pt_position': [3,2,1]},\n 'func_id': 2}])\nclient.annotation.post_annotation_df('test_table', df)\n
Note that here I specified the IDs of my annotations, which you can do, but then its up to you to assure that the IDs don't collide with other IDs. If you leave them blank then the service will assign the IDs for you.
There is a similar method for updating update_annotation_df()
"},{"location":"tutorials/annotation/#staged-annotations","title":"Staged Annotations","text":"Staged annotations help ensure that the annotations you post follow the appropriate schema, both by providing guides to the field names and locally validating against a schema before uploading. The most common use case for staged annotations is to create a StagedAnnotation object for a given table, then add annotations to it individually or as a group, and finally upload to the annotation table.
To get a StagedAnnotation object, you can start with either a table name or a schema name. Here, we'll assume that there's already a table called \"my_table\" that is running a \"cell_type_local\" schema. If we want to add new annotations to the table, we simply use the table name with stage_annotations().
stage = client.annotation.stage_annotations(\"my_table\")\n
This stage object retrieves the schema for the table and hosts a local collection of annotations. Every time you add an annotation, it is immediately validated against the schema. To add an annotation, use the add method:
stage.add(\n cell_type = \"pyramidal_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\n
The argument names derive from fields in the schema and you must provide all required fields. Any number of annotations can be added to the stage. A dataframe of annotations can also be added with stage.add_dataframe, and requires an exact match between column names and schema fields. The key difference between this and posting a dataframe directly is that annotations added to a StagedAnnotations are validated locally, allowing any issues to be caught before uploading.
You can see the annotations as a list of dictionary records with stage.annotation_list or as a Pandas dataframe with stage.annotation_dataframe. Finally, if you initialized the stage with a table name, this information is stored in the stage and you can simply upload it from the client.
client.annotation.upload_staged_annotations(stage)\n
Updating annotations requires knowing the annotation id of the annotation you are updating, which is not required in the schema otherwise. In order to stage updated annotations, set the update parameter to True when creating the stage.
update_stage = client.annotation.stage_annotations(\"my_table\", update=True)\nupdate_stage.add(\n id=1,\n cell_type = \"stellate_cell\",\n classification_system=\"excitatory\",\n pt_position=[100,100,10],\n)\n
The update also informs the framework client to treat the annotations as an update and it will use the appropriate methods automatically when uploading client.annotation.upload_staged_annotations.
If you want to specify ids when posting new annotations, id_field can be set to True when creating the StagedAnnotation object. This will enforce an id column but still post the data as new annotations.
If you might be adding spatial data in coordinates that might be different than the resolution for the table, you can also set the annotation_resolution when creating the stage. The stage will convert between the resolution you specify for your own annotations and the resolution that the table expects.
stage = client.annotation.stage_annotations(\"my_table\", annotation_resolution=[8,8,40])\nstage.add(\n cell_type='pyramidal_cell',\n classification_system=\"excitatory\",\n pt_position=[50,50,10],\n)\n
"},{"location":"tutorials/authentication/","title":"Authentication","text":"Authentication tokens are generally needed for programmatic access to our services. The AuthClient handles storing and loading your token or tokens and inserting it into requests in other clients.
We can access the auth client from client.auth. Once you have saved a token, you probably won't interact with this client very often, however it has some convenient features for saving new tokens the first time. Let's see if you have a token already. Probably not.
client = CAVEclient()\nauth = client.auth\nprint(f\"My current token is: {auth.token}\")\n
"},{"location":"tutorials/authentication/#getting-a-new-token","title":"Getting a new token","text":"To get a new token, you will need to manually acquire it. For convenience, the function client.auth.get_new_token() provides instructions for how to get and save the token.
By default, the token is saved to ~/.cloudvolume/secrets/cave-secret.json as a string under the key token. This makes it compatible by default with Cloudvolume projects, which can come in handy. The following steps will save a token to the default location.
auth.get_new_token()\n
new_token = 'abcdef1234567890' #This is the text you see after you visit the website.\nauth.save_token(token=new_token)\nprint(f\"My token is now: {auth.token}\")\n
Note that requesting a new token will invalidate your previous token on the same project. If you want to use the same token across different computers, you will need to share the same token information.
"},{"location":"tutorials/authentication/#loading-saved-tokens","title":"Loading saved tokens","text":"Try opening ~/.cloudvolume/secrets/cave-secret.json to see what we just created.
If we had wanted to use a different file or a different json key, we could have specified that in auth.save_token.
Because we used the default values, this token is used automatically when we initialize a new CAVEclient. If we wanted to use a different token file, token key, or even directly specify a token we could do so here.
client = CAVEclient(datastack_name)\nprint(f\"Now my basic token is: {client.auth.token}\")\n\nclient_direct = CAVEclient(datastack_name, auth_token='another_fake_token_678')\nprint(f\"A directly specified token is: {client_direct.auth.token}\")\n
If you use a CAVEclient, the AuthClient and its token will be automatically applied to any other services without further use.
"},{"location":"tutorials/chunkedgraph/","title":"Chunked Graph","text":"The chunkedgraph is a dynamic oct-tree connected components supervoxel graph.
A visual representation of an oct-tree (Wikipedia (WhiteTimberwolf) CC BY-SA 3.0)
As with any oct-tree, it is organized in hierarchical levels, with the bottom level 1 corresponding to the supervoxels of the segmentations, and the top level being the unique connected components of the supervoxel graph.
A figure illustrating the spatial chunking and editing of a the chunkedgraph. From Dorkenwald et. al 2021
The ChunkedGraph client allows one to interact with the ChunkedGraph service, which stores and updates the supervoxel agglomeration graph. This is most often useful for looking up an object root id of a supervoxel or looking up supervoxels belonging to a root id. The ChunkedGraph client is at client.chunkedgraph.
"},{"location":"tutorials/chunkedgraph/#look-up-a-supervoxel","title":"Look up a supervoxel","text":"Usually in Neuroglancer, one never notices supervoxel ids, but they are important for programmatic work. In order to look up the root id for a location in space, one needs to use the supervoxel segmentation to get the associated supervoxel id. The ChunkedGraph client makes this easy using the client.chunkedgraph.get_root_id() method.
sv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id)\n
However, as proofreading occurs, the root id that a supervoxel belongs to can change. By default, this function returns the current state, however one can also provide a UTC timestamp to get the root id at a particular moment in history. This can be useful for reproducible analysis. Note below that the root id for the same supervoxel is different than it is now.
import datetime\n\n# I looked up the UTC POSIX timestamp from a day in early 2019.\ntimestamp = datetime.datetime.utcfromtimestamp(1546595253)\n\nsv_id = 104200755619042523\nclient.chunkedgraph.get_root_id(supervoxel_id=sv_id, timestamp=timestamp)\n
If you are doing this across lots of supervoxels (or any nodes) then you can do it more efficiently in one request with client.chunkedgraph.get_roots()
node_ids = [104200755619042523, 104200755619042524,104200755619042525]\nroot_ids = client.chunkedgraph.get_roots(node_ids)\n
"},{"location":"tutorials/chunkedgraph/#getting-supervoxels-for-a-root-id","title":"Getting supervoxels for a root id","text":"A root id is associated with a particular agglomeration of supervoxels, which can be found with the client.chunkedgraph.get_leaves() method. A new root id is generated for every new change in the chunkedgraph, so time stamps do not apply.
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id)\n
You can also query the chunkedgraph not all the way to the bottom, using the stop_layer option
root_id = 648518346349541252\nclient.chunkedgraph.get_leaves(root_id,stop_layer=2)\n
This will get all the level 2 IDs for this root, which correspond to the lowest chunk of the hierarchy. An analogous option exists for client.chunkedgraph.get_roots(). This is useful to help find nodes to query within the l2cache, amongst other things.
"},{"location":"tutorials/chunkedgraph/#other-functions","title":"Other functions","text":"There are a variety of other interesting functions to explore in client.chunkedgraph.
"},{"location":"tutorials/framework/","title":"Introduction","text":""},{"location":"tutorials/framework/#caveclient-one-client-for-all-services","title":"CAVEclient: one client for all services","text":"The CAVE Framework consists of a number of different services, each with a specific set of tasks that it can perform through REST endpoints. The CAVEclient is designed to ease programmatic interaction with all of the various endpoints. In addition, most programmatic access requires the use of authentication tokens. In order to collect a given server, datastack name, and user token together into a coherent package that can be used on multiple endpoints, the CAVEclient builds appropriately configured clients for each of the specific services. Each of the individual services has their own specific documentation as well.
"},{"location":"tutorials/framework/#global-and-local-services","title":"Global and Local Services","text":"There are two categories of data in CAVE: Global and local. Local services are associated with a single so-called datastack, which refers to a precise collection of imagery and segmentation data that function together. For example, EM imagery and a specific pychunkedgraph segmentation would be one datastack, while the same EM imagery but an initial static segmentation would be another. Datastacks are referred to by a short name, for instance pinky100_public_flat_v185.
Global services are those that are potentially shared across multiple different specific datastacks. These include the info service, which can describe the properties of all available datastacks, the authentication service, and the state service that hosts neuroglancer states. Global services are associated with a particular URL (by default http://globalv1.daf-apis.com), but not a single datastack.
"},{"location":"tutorials/framework/#initializing-a-caveclient","title":"Initializing a CAVEclient","text":"Assuming that the services are on http://globalv1.daf-apis.com and authentication tokens are either not being used or set up with default values (see Authentication), a simple CAVEclient that can only access global services can be initialized:
from caveclient import CAVEclient\n\nclient = CAVEclient()\n
Just to confirm that this works, let's see if we can get the EM image source from the InfoService. If you get a list of names of datastacks, all is good. If you have not yet set up an authentication token or you get an authentication error, look at Getting a new token for information about how to set up your auth token.
client.info.get_datastacks()\n
If you have a specific datastack you want to use, you can initialize your CAVEclient with it. This gives you access to the full range of client functions.
client = CAVEclient(datastack_name='my_datastack')\n
"},{"location":"tutorials/framework/#using-other-server-addresses","title":"Using Other Server Addresses","text":"If your data is hosted by a different global server, you specify its address when initializing the client.
client = CAVEclient(datastack_name='my_datastack', server_address='http://global.myserver.com')\n
By default, if you pass both a server address and a datastack, the client will store the mapping from datastack to server address in the same location as the default for authentication tokens. Once stored, the client will automatically use the correct server address for the datastack if none is provided. You can override storing the server address by passing write_server_address=False. Datastacks can be removed from the cache using
caveclient.datastack_lookup.reset_server_address_cache(datastack_name).
"},{"location":"tutorials/framework/#accessing-specific-clients","title":"Accessing specific clients","text":"Each client can be accessed as a property of the main client. See the documentation at left for the capabilities of each. Assuming your client is named client, the subclients for each service are:
- Authentication Service :
client.auth - AnnotationEngine :
client.annotation - PyChunkedGraph :
client.chunkedgraph - InfoService :
client.info - EM Annotation Schemas :
client.schemas - JSON Neuroglancer State Service :
client.state - Skeleton Service :
client.skeletonservice
"},{"location":"tutorials/info/","title":"Info Service","text":"A datastack has a number of complex paths to various data sources that together comprise a datastack. Rather than hardcode these paths, the InfoService allows one to query the location of each data source. This is also convenient in case data sources change.
An InfoClient is accessed at client.info.
client = CAVEclient(datastack_name)\nprint(f\"This is an info client for {client.info.datastack_name} on {client.info.server_address}\")\n
"},{"location":"tutorials/info/#accessing-datastack-information","title":"Accessing datastack information","text":"All of the information accessible for the datastack can be seen as a dict using get_datastack_info().
info.get_datastack_info()\n
Individual entries can be found as well. Use tab autocomplete to see the various possibilities.
info.graphene_source()\n
"},{"location":"tutorials/info/#adjusting-formatting","title":"Adjusting formatting","text":"Because of the way neuroglancer looks up data versus cloudvolume, sometimes one needs to convert between gs:// style paths to https://storage.googleapis.com/ stype paths. All of the path sources in the info client accept a format_for argument that can handle this, and correctly adapts to graphene vs precomputed data sources.
neuroglancer_style_source = info.image_source(format_for='neuroglancer')\nprint(f\"With gs-style: { neuroglancer_style_source }\")\n\ncloudvolume_style_source = info.image_source(format_for='cloudvolume')\nprint(f\"With https-style: { cloudvolume_style_source }\")\n
"},{"location":"tutorials/l2cache/","title":"Level 2 Cache","text":"To understand the level 2 cache, you must understand the structure of the chunkedgraph so see the chunkedgraph tutorial.
Nodes on the second level or layer of the graph, corresponds to all the supervoxels that are locally connected to one another within a single level 2 spatial \"chunk\" of the data. The Level 2 Cache, is a service whose job it is to track and update relevant statistics about every level 2 node within the a chunkedgraph. The source code of this service can be found here.
"},{"location":"tutorials/l2cache/#finding-level-2-nodes","title":"Finding Level 2 Nodes","text":"The chunkedgraph can be used to find the level2 nodes of a rootID using a stop_layer=2 keyword argument on the client.chunkedgraph.get_leaves(). Conversely the level 2 node of a supervoxel can be found using the same keyword argument of client.chunkedgraph.get_roots(). Note if you don't specify a timestamp it will give you the level2 node that is presently associated with the object.
"},{"location":"tutorials/l2cache/#statistics","title":"Statistics","text":"The statistics that are available are:
- area_nm2: The surface area of the object in square nanometers. Does not include border touching voxels
- size_nm3: The volume of the object in cubic nanometers, based on counting voxels in the object.
- max_dt_nm: The maximum edge distance transform of that object in nanometers. Meant to capture the maximum \"thickness\" of the voxels in the node.
- mean_dt_nm: The average edge distance transform of that object in nanometers. Meant to capture the average \"thickness\" of voxels in that node.
- rep_coord_nm: A list of x,y,z coordinates in nanometers that represent a point within the object that is designed to be close to the \"center\" of the object. This is the location of the max_dt_nm value.
- chunk_intersect_count: A 2 x 3 matrix representing the 6 sides of the chunk, and whose values represent how many voxels border that side of the chunk. Meant to help understand significant the borders with other chunks are. Ordering is the [[x_bottom, y_bottom, z_bottom],[x_top, y_top, z_top]] where {xyz}_bottom refers to the face which has the smallest values for that dimension, and {xyz}_top refers to the face which has the largest.
- pca A 3x3 matrix representing the principal components of the xyz point cloud of voxels for this object. Ordering is NxD where N is the components and D are the xyz dimensions. Meant to help desribe the orientation of the level 2 chunk. Note that this is not calculated for very small objects and so might not be present for all level 2 nodes. You will see that its availability correlates strongly with size_nm3.
- pca_val The 3 principal component values for the PCA components.
"},{"location":"tutorials/l2cache/#retrieving-level-2-statistics","title":"Retrieving Level 2 Statistics","text":"Level 2 stats about nodes can be retreived using the client.l2cache.get_l2data() method. It simply takes a list of level 2 nodes you want to retrieve. Optionally you can specify only the attributes that you are interested in retrieving which will speed up the request.
"},{"location":"tutorials/l2cache/#missing-data","title":"Missing Data","text":"The service is constantly watching for changes made to objects and recalculating stats on new level2 nodes that are created, in order to keep its database of statistics current. This however takes some time, and is subject to sporadic rare failures. If you request stats on a level 2 node which are not in the database, you will receive an empty dictionary for that node. This will immediately trigger the system to recalculate the statistics of that missing data, and so it should be available shortly (on the order of seconds) if systems are operational. Please note that PCA is not calculated for very small objects because it is not meaningful. So if you are interested in differentiating whether PCA is not available because it hasn't been calculated, vs when its not available because it is not possible to calculate, you should ask for at least one other non PCA statistic as well. You will see that its availability correlates strongly with size_nm3.
"},{"location":"tutorials/l2cache/#use-cases","title":"Use Cases","text":""},{"location":"tutorials/l2cache/#calculate-total-area-and-volume-of-cells","title":"Calculate Total Area and Volume of Cells","text":"Say you want to calculate the total surface area and volume of a object in the dataset. The areas and volume of each component can simply be added together to do this.
import pandas as pd\nroot_id = 648518346349541252\nlvl2nodes = client.chunkedgraph.get_leaves(root_id,stop_layer=2)\nl2stats = client.l2cache.get_l2data(lvl2nodes, attributes=['size_nm3','area_nm2'])\nl2df = pd.DataFrame(l2stats).T\ntotal_area_um2=l2df.area_nm2.sum()/(1000*1000)\ntotal_volume_um3 = l2df.size_nm3.sum()/(1000*1000*1000)\n
By utilizing the bounds argument of get_leaves, you can also do simple spatially restricted analysis of objects. In fact, because you have data on each level2 node individually, you can segregate the neuron using any labelling of its topology.
"},{"location":"tutorials/l2cache/#skeletonization","title":"Skeletonization","text":"Level 2 nodes have \"cross chunk\" edges within the chunkedgraph which represent what level 2 nodes that object is locally connected to. This forms a graph between the level 2 nodes of the object that can be retrieved using the chunkedgraph function client.chunkedgraph. This graph represents a topological representation of the neuron at the resolution of individual chunks, and is guaranteed to be fully connected, unlike a voxel or mesh representation of the neuron which can have gaps where there are defects in the segmentation volume or incorrectly inferred edges at self contact locations.
The level 2 graph can be turned into a skeleton representation of a neuron using a graph based TEASAR like algorithm as described for skeletonizing meshes in this MeshParty Documentation. There is an implementation of this approach that utilizes the chunkedgraph and the L2cache if available here and on pypi as pcg-skel. In this implementation the l2cache is used to more accurately place the level 2 nodes in space using the rep_coord_nm value.
"},{"location":"tutorials/l2cache/#trajectory-distributions","title":"Trajectory Distributions","text":"If one is interested in the bulk direction of processes in a region of the brain, one can start with supervoxels in a region, find level 2 nodes that correspond to them, filter out components based on size, (or other criteria such as whether they are part of objects that have components in some other brain area) and look at the distribution of PCA components to understand the directions that those processes are moving within that region of space.
"},{"location":"tutorials/materialization/","title":"Materialization","text":"The Materialization client allows one to interact with the materialized annotation tables, that were posted to the annotation service (the annotations tutorial).
To see the entire class visit the API doc.
The service regularly looks up all annotations and the segids underneath all the boundspatialpoints. You can then query these tables to find out the IDs that underlie the annotations, or the annotations that now intersect with certain IDs.
For example, one common pattern is that you have identified a cell based on the location of its cell body, and you have an annotation there.
You want to know what are the inputs onto the cell, so you first query the annotation table with your soma annotation, asking for the current ID underneath that soma. Then you query a synapse table for all synapse annotations that have a post-synaptic ID equal to the ID from your soma annotation.
In this way your code stays the same, as the proofreading changes and you can track the connectivity of your cell over time.
"},{"location":"tutorials/materialization/#initializing-the-client","title":"Initializing the client","text":"By default when you initialize the overall client, it will choose the most recent materialization version available. This may or may not be desirable depending on your use case. If your code involves using specific IDs then you should be using a specific version that is tied to a timepoint where those IDs are valid.
To see what versions are available, use the client.materialize.get_versions() function.
client.materialize.get_versions()\n
Each version has a timestamp it was run on as well as a date when it will expire. You can query all this metadata for a specific version using client.materialize.get_version_metadata() or all versions using client.materialize.get_versions_metadata().
To change the default version, alter the .version property of the client. This will change the version for all subsequent calls which expect one, unless you specify a different version in the method call. Note that this also sets the timestamp property of the client to the timestamp of the version for methods which expect a timestamp.
client.version = 9\n
You can also specify the version when you initialize the client, e.g.:
client = CAVEclient('minnie65_public', version=661)\n
Or, you can specify the version when making a particular method call.
"},{"location":"tutorials/materialization/#browsing-versions","title":"Browsing versions","text":"To see what tables are available in a version you can use client.materialize.get_tables().
If you want to read about the description of what that table is, use the annotationengine client client.materialize.get_table_metadata().
If you want to read more about the schema for the annotation table use the schema service client.schema.schema_definition().
Note, the materialization service has a human readable webpage that links to the other services that might be more convenient for you to browse, to get a link there in ipython display client.materialize.homepage
for some important tables, the info service has a pointer to which table you should use in the metadata for the datastack. client.info.get_datastack_info()['synapse_table'] and client.info.get_datastack_info()['soma_table'].
To see how many annotations are in a particular table use
nannotations=client.materialize.get_annotation_count('my_table')\n
"},{"location":"tutorials/materialization/#querying-tables","title":"Querying tables","text":"To query a small table, you can just download the whole thing using client.materialize.query_table() which will return a dataframe of the table.
Note however, some tables, such as the synapse table might be very large 200-300 million rows and the service will only return the first 200,000 results, and not in a deterministic manner. NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
To just get a preview, use the limit argument (but note again that this won't be a reproducible set)
df=client.materialize.query_table('my_table', limit=10)\n
For most applications, you will want to filter the query in some way.
We offer seven kinds of filters you can apply: filter_in_dict filter_out_dict, filter_equal_dict, filter_greater_dict, filter_less_dict, filter_greater_equal_dict, and filter_less_equal_dict. For query_table each is specified as a dictionary where the keys are column names, and the values are a list of values (or single value in the case of filter_equal).
So for example to query a synapse table for all synapses onto a neuron in flywire you would use
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID})\n
The speed of querying is affected by a number of factors, including the size of the data. To improve the performance of results, you can reduce the number of columns returned using select_columns.
So for example, if you are only interested in the root_ids and locations of pre_synaptic terminals you might limit the query with select_columns. Also, it is convenient to return the with positions as a column of np.array([x,y,z]) coordinates for many purposes. However, sometimes you might prefer to have them split out as separate x, y, z columns. To enable this option use split_columns=True. split_columns=True is faster, as combining them is an extra step.
synapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n select_columns=['id','pre_pt_root_id', 'pre_pt_position'],\n split_columns=True)\n
"},{"location":"tutorials/materialization/#desired-resolution","title":"Desired Resolution","text":"Often you want to have position information in different units. For example, to consider synapse locations or soma locations, you might want to have positions in nanometers or microns.
To create neuroglancer views, you might want positions in integer voxels of a size that aligns with the resolution you are used to using Neuroglancer at.
Annotation tables can be created and uploaded in varying resolutions according to whatever the user of the table felt was natural. This information is available in the metadata for that table. In addition, you may pass desired_resolution as a keyword argument which will automatically convert all spatial positions into voxels of that size in nanometers.
So if you want positions in nanometers, you would pass desired_resolution=[1,1,1]. If you want positions in microns you would pass desired_resolution=[1000,1000,1000]. If you want positions in 4,4,40nm voxel coordinates to use with cloud-volume or neuroglancer you would pass desired_resolution=[4,4,40].
"},{"location":"tutorials/materialization/#spatial-filters","title":"Spatial Filters","text":"You can also filter columns that are associated with spatial locations based upon being within a 3d bounding box.
This is done by adding a filter_spatial_dict argument to query_table. The units of the bounding box should be in the units of the voxel_resolution of the table (which can be obtained from client.materialize.get_table_metadata()).
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nsynapse_table = client.info.get_datastack_info('synapse_table')\ndf=client.materialize.query_table(synapse_table,\n filter_equal_dict = {'post_pt_root_id': MYID},\n filter_spatial_dict = {'post_pt_position': bounding_box})\n
"},{"location":"tutorials/materialization/#synapse-query","title":"Synapse Query","text":"For synapses in particular, we have a simplified method for querying them with a reduced syntax. client.materialize.synapse_query() lets you specify pre and post synaptic partners as keyword arguments and bounding boxes. The defaults make reasonable assumptions about what you want to query, namely that the synapse_table is the table that the info service advertises, and that if you specify a bounding box, that you want the post_pt_position. These can be overridden of course, but the above bounding box query is simplified to.
NOTE! This API is not designed to enable enmass downloading of the entire synapse table there are more efficient ways of doing this. Contact your dataset administrator for more information if this is what you are looking to do.
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\ndf=client.materialize.query_table(post_ids = MYID,\n bounding_box=bounding_box)\n
"},{"location":"tutorials/materialization/#live-query","title":"Live Query","text":"In order to query the materialized tables above you can only use IDs that were present at the timestamp of the materialization. If you query the tables with an ID that is not valid during the time of the materialization you will get empty results.
To check if root_ids are valid at your materialization's timestamp, you can use client.chunkedgraph.is_latest_roots()
import numpy as np\nmat_time = client.materialize.get_timestamp()\nis_latest = client.chunkedgraph.is_latest_roots([MYID], timestamp=mat_time)\nassert(np.all(is_latest))\n
If you need to lookup what happened to that ID, you can use the chunkedgraph lineage tree, to look into the future or the past, depending on your application you can use client.chunkedgraph.get_lineage_graph().
Again, the ideal situation is that you have an annotation in the database which refers to your objects of interest, and querying that table by the id column will return the object in the most recent materialization.
However, sometimes you might be browsing and proofreadding the data and get an ID that is more recent that the most recent version available. For convenience, you can use client.materialize.live_query().
to automatically update the results of your query to a time in the future, such as now. For example, to pass now, use datetime.datetime.now(datetime.timezone.utc). Note all timestamps are in UTC throughout the codebase.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.live_query(synapse_table,\n datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\n
This will raise an ValueError exception if the IDs passed in your filters are not valid at the timestamp given
You can also pass a timestamp directly to query_table and it will call live_query automatically.
import datetime\nsynapse_table = client.info.get_datastack_info()['synapse_table']\ndf=client.materialize.query_table(synapse_table,\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n filter_equal_dict = {'post_pt_root_id': MYID})\n
Also, keep in mind if you run multiple queries and at each time pass datetime.datetime.now(datetime.timezone.utc), there is no guarantee that the IDs will be consistent from query to query, as proofreading might be happening at any time. For larger scale analysis constraining oneself to a materialized version will ensure consistent results.
Versions have varying expiration times in order to support the tradeoff between recency and consistency, so before undertaking an analysis project consider what version you want to query and what your plan will be to update your analysis to future versions.
"},{"location":"tutorials/materialization/#content-aware-interface-experimental","title":"Content-aware Interface (Experimental)","text":"As of version 5.8.0, we have introduced a new interface to query tables and views. This interface might have small but breaking changes in the near future. :::
In order to make the querying interface more consistent across tables, we have introduced an additional alternative interface to filtering and querying data via the client.materialize.tables object. When you instantiate this object, this object finds all of the existing tables and the list of their columns and lets you filter the tables as arguments in the function with suggestions. Moreover, the filtering arguments and the querying arguments are separated into two.
Let's see how this works with a simplest example --- downloading a table called nucleus_detection_v0. First, we reference the table as a function and then we run the query --- this is exactly the same as client.materialize.query_table('nucleus_detection_v0').
client = CAVEclient('minnie65_public')\nnuc_df = client.materialize.tables.nucleus_detection_v0().query()\n
Where things differ is when we add filters. If we want to query based on a set of values for the field \"id\", for example, we add that as an argument:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(id=my_ids).query()\n
Where in this example the id= queries the column id based on the schema. These values can be either individual elements (i.e. an integer or a string) or a list/array of elements, and any field can be used. The tooling will automatically sort out how to format the filtering appropriately when running the query. Importantly, the filtering is identical between querying all types of tables and queries. To see the complete list of fields that can be queried, you can tab-autocomplete or in Jupyter or IPython glance at the docstring with client.materialize.tables.nucleus_detection_v0?.
In addition to filtering by one or many values, you can also do spatial queries. To find only annotations within a particular bounding box, you need to specify the spatial point (e.g. pt_position). For each such spatial point, there will be an argument {spatial_point_position}_bbox (e.g. pt_position_bbox) that accepts a 2x3 list of coordinates specifying the upper and lower bounds for the query. For example, to find all nucleus centroids within a particular bounding box, it would be:
bounding_box = [[min_x, min_y, min_z], [max_x, max_y, max_z]]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n pt_position_bbox=bounding_box\n).query()\n
If you are not using any filters, you can omit the parenthesis and use the get_all or get_all_live functions directly, which act similarly to the query and live_query functions respectively. The first example could be rewritten as:
nuc_df = client.materialize.tables.nucleus_detection_v0.get_all()\n
If you want to list all available fields, you can use the .fields attribute. Similarly, you can get all numeric fields with the .numeric_fields attribute and all spatial fields (allowing bounding box queries) with .spatial_fields.
nuc_df = client.materialize.tables.nucleus_detection_v0.spatial_fields\n
If you need to specify the table programmatically, you can also use a dictionary-style approach to getting the table filtering function. For example, an equivalent version of the above line would be:
my_ids = [373879, 111162]\nmy_table = 'nucleus_detection_v0'\nnuc_df = client.materialize.tables[my_table](id=my_ids).query()\n
The query function can also take arguments relating to timestamps or formatting where they act just like in the other query method. In particular, the arguments that apply to query are: select_columns, offset, limit, split_positions, materialization_version, timestamp, metadata, desired_resolution, and get_counts. For example, to add a desired resolution and split positions in the above query, it would look like:
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).query(\n split_positions=True,\n desired_resolution=[1,1,1],\n)\n
Inequalities can also be used in filtering numeric columns. Here, you can pass a dictionary instead of a list of values, with the keys being inequality operators (\">\", \">=\", \"<\", and \"<=\") and the values being the comparison. For example, to query for all nuclei with a volume greater than 1000:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 1000}\n).query()\n
You can also use multiple inequalities in the same dictionary to filter within a range. For example, to query for all nuclei with a volume between 500 and 750:
client.materialize.tables.nucleus_detection_v0(\n volume={\">\": 500, \"<\": 750}\n).query()\n
If you want to do a live query instead of a materialized query, the filtering remains identical but we use the live_query function instead. The one required argument for live_query is the timestamp.
my_ids = [373879, 111162]\nnuc_df = client.materialize.tables.nucleus_detection_v0(\n id=my_ids\n).live_query(\n timestamp=datetime.datetime.now(datetime.timezone.utc),\n)\n
The live query functions have similar but slightly different arguments: timestamp (required), offset, limit, split_positions, metadata, desired_resolution, and allow_missing_lookups.
Note that way that IPython handles docstrings means that while you can use ? to get the docstring of the filtering part of the function, you can't simply do something like client.materialize.tables.nucleus_detection_v0().query?. It will tell you the function can't be found, because technically the query function does not yet exist until the table filtering function is called.
Instead, if you want to glimpse the docstring of the query or live_query functions, you need to split it into two lines:
qry_func = client.materialize.tables.nucleus_detection_v0().query\nqry_func?\n
Finally, if the project you are working with has views, a similar interface is available to them via client.materialize.views. Currently views are not compatible with live query, and so only the .query function is available.
"},{"location":"tutorials/schemas/","title":"EM Annotation Schemas","text":"The EMAnnotationSchemas client lets one look up the available schemas and how they are defined. This is mostly used for programmatic interactions between services, but can be useful when looking up schema definitions for new tables.
"},{"location":"tutorials/schemas/#get-the-list-of-schema","title":"Get the list of schema","text":"One can get the list of all available schema with the schema method. Currently, new schema have to be generated on the server side, although we aim to have a generic set available to use.
client.schema.get_schemas()\n
"},{"location":"tutorials/schemas/#view-a-specific-schema","title":"View a specific schema","text":"The details of each schema can be viewed with the schema_definition method, formatted as per JSONSchema.
example_schema = client.schema.schema_definition('microns_func_coreg')\nexample_schema\n
This is mostly useful for programmatic interaction between services at the moment, but can also be used to inspect the expected form of an annotation by digging into the format.
example_schema['definitions']['FunctionalCoregistration']\n
"},{"location":"tutorials/state/","title":"JSON Neuroglancer State Service","text":"We store the JSON description of a Neuroglancer state in a simple database at the JSON Service. This is a convenient way to build states to distribute to people, or pull states to parse work by individuals. The JSON Client is at client.state
client.state\n
"},{"location":"tutorials/state/#retrieving-a-state","title":"Retrieving a state","text":"JSON states are found simply by their ID, which you get when uploading a state. You can download a state with get_state_json.
example_id = 4845531975188480\nexample_state = client.state.get_state_json(test_id)\nexample_state['layers'][0]\n
"},{"location":"tutorials/state/#uploading-a-state","title":"Uploading a state","text":"You can also upload states with upload_state_json. If you do this, the state id is returned by the function. Note that there is no easy way to query what you uploaded later, so be VERY CAREFUL with this state id if you wish to see it again.
Note: If you are working with a Neuroglancer Viewer object or similar, in order to upload, use viewer.state.to_json() to generate this representation.
example_state['layers'][0]['name'] = 'example_name'\nnew_id = client.state.upload_state_json(example_state)\n
test_state = client.state.get_state_json(new_id)\ntest_state['layers'][0]['name']\n
"},{"location":"tutorials/state/#generating-a-neuroglancer-url","title":"Generating a Neuroglancer URL","text":"Once you have a state ID, you want to turn it into a well-formatted link. So you don\\'t have to remember all the endpoints, we can do this from the state client.
ngl_base = 'neuromancer-seung-import.appspot.com'\nclient.state.build_neuroglancer_url(new_id, ngl_base)\n
Note that the neuroglancer base can be found in the info service under client.info.viewer_site().
"}]}
\ No newline at end of file
diff --git a/sitemap.xml b/sitemap.xml
index 764f6591..00201226 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -2,118 +2,118 @@
https://caveconnectome.github.io/CAVEclient/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/changelog/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/contributing/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/glossary/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/installation/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/annotation/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/auth/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/chunkedgraph/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/client/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/config/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/datastack_lookup/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/info/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/l2cache/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/materialize/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/schema/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/skeleton/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/api/state/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/advanced/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/annotation/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/authentication/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/chunkedgraph/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/framework/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/info/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/l2cache/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/materialization/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/schemas/
- 2024-10-24
+ 2024-10-29
https://caveconnectome.github.io/CAVEclient/tutorials/state/
- 2024-10-24
+ 2024-10-29
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 08ee8eec..b8beca01 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ