Merge pull request #59 from twosixlabs/21-document-armory-library

API documentation with prefatory annotations

docs/reference/data.md

@@ -1,3 +1,7 @@
 # Dataset Ingestion and Adaptation
+A dataset is a collection of images, i.e. the sample set, in a sequence-like structure. This class is used in the initial uploading and manipulation of a dataset and supports converting 2 different types of data into a structure compatible with armory-library:
+
+- Tuples datasets which can be turned into a map
+- PyTorch datasets needing to be turned into numpy arrays
 
 ::: charmory.data

docs/reference/engine.md

@@ -1,4 +1,7 @@
 # Armory Engines
+An engine is the core of the Armory library. There are two different types of engines which the user should choose from based on their overall objective:
+- The Evaluation Engine performs model robustness evaluations as pertains to adversarial attacks. Can optionally be recorded in MLflow.
+- The Adversarial Dataset Engine creates the adversarial dataset by applying an attack to each sample in the original dataset, outputting the results into a directory. Additional modifications to the samples may also be preformed.
 
 ::: charmory.engine.EvaluationEngine
 

docs/reference/evaluation.md

@@ -1,3 +1,4 @@
 # Evaluation Configuration
+An evaluation is the complete testing of a model/attack. This class contains the configuration of the entire evaluation, to include details about the attack, dataset, evaluation, collected metrics, model, and host system configuration.
 
 ::: charmory.evaluation

docs/reference/labels.md

@@ -1,3 +1,13 @@
 # Label Targeters
+A label is the target/ground truth associated with a single sample. This class is a utility containing algorithms for generating or updating target labels in a variety of formats to include:
+
+- Single, fixed integer
+- Single, fixed string
+- Random label from a predefined list
+- Fixed integer offset
+- Fixed values as specified by an ordered list
+- Exact value as the input label
+- Replacement of object detection labels with fixed integer
+- Transcript from a fixed list with the length closest to that of the input label
 
 ::: charmory.labels

docs/reference/model.md

@@ -1,4 +1,5 @@
 # Model Ingestion and Adaptation
+Models help adapt the input and output data of the evaluation to a user's needs. The pre- and post-adapters can be specified on the base ArmoryModel. The JATIC Image Classiifcation model class has a pre-defined output adapter specific to that task which includes logits, probabilities, and scores. The JATIC Object Detection model also has a pre-defined output adapter which includes boxes, logits, probabilities, and scores. The YOLOS Transformer has pre-defined input and output adapters for HuggingFace transformer YOLOS models. On input, the "labels" property becomes "class_labels". On output, the loss components are returned instead of the predictions since the model is put into training mode during attack generation.
 
 ::: charmory.model.base
 

docs/reference/profiler.md

@@ -1,3 +1,4 @@
 # Computational Performance Profilers
+Profilers are responsible for generating benchmarking metrics for the evaluation. This class contains a choice of 3 profilers to collect computational metrics for evaluations: Basic, Deterministic, and Null. The first 2 provide an average CPU time for each type of computation.
 
 ::: armory.metrics.compute

docs/reference/task.md

@@ -1,4 +1,11 @@
 # Evaluation Tasks
+A task should match what the model is intended to do. Right now, the options are image classification and object detection, but more could be supported in the future.
+
+The class holds details of the entire task to include the evaluation configuration, whether to skip the benign and/or attack datasets, an optional adapter to be applied to the inference data prior to exporting to MLflow, and a frequency at which batches will be exported to MLflow, if at all.
+
+The Image Classification task can additionally be created with the total number of classes the model is capable of predicting, and the L-norm order for the perturbation distance metrics.
+
+The Object Detection task can additionally be created with an option to track Mean Average Precision (MAP) metrics per class, with a minimum prediction score for a detection bounding box to be drawn on the exported sample, a maximum intersection-over-union value for non-maximum suppression filtering of detection bounding boxes, and a minimum prediction score, with all predictions lower than this being ignored.
 
 ::: charmory.tasks.base.BaseEvaluationTask
 

docs/reference/track.md

@@ -1,3 +1,4 @@
 # Evaluation Tracking
+Utilities supporting the tracking of evaluations within armory-library. Contains functions to see/choose/clear which parameters to be logged with MLflow, the ability to set the MLflow tracking URI, name and describe the experiment, and actually track the evaluation and system metrics (CPU/GPU, disk, and network utilization).
 
 ::: charmory.track

docs/reference/transforms.md

@@ -1,3 +1,4 @@
 # Dataset Transforms
+A transform is a (preprocessing) function on a model. This class contains general data transform functions for experimentation, to include Bounded Boxes (and the conversion between different formats of these), the resizing, padding, rescaling, and normalization (into z-scores) of images, image classification transforms, and object detection transforms.
 
 ::: charmory.experimental.transforms

docs/reference/utils.md

@@ -1,3 +1,10 @@
 # Utilities
+A variety of utilities to aid in evaluation tasks. Includes:
+
+- A custom torchvision transform which converts PIL images to numpy arrays
+- A customizable transform which serves as the inverse of torchvision.transforms.Normalize
+- The ability to apply a given ART pre- or post-processor defense to a model
+- A transform that can be applied to JATIC-wrapped datasets using a preprocessor from a JATIC-wrapped model
+- A check for whether a given estimator has any pre- or post-processor defenses applied to it
 
 ::: charmory.utils

library/src/charmory/model/base.py

@@ -20,7 +20,8 @@ class ArmoryModel(nn.Module):
     Wrapper around a model to apply an adapter to inputs and outputs of the
     model.
 
-    Example::
+    Example:
+
         from charmory.model import ArmoryModel
 
         def preadapter(images, *args, **kwargs):