diff --git a/docs/reference/data.md b/docs/reference/data.md index 5151b07d..cd04d42e 100644 --- a/docs/reference/data.md +++ b/docs/reference/data.md @@ -1,3 +1,7 @@ # Dataset Ingestion and Adaptation +A dataset is a collection of images, i.e. the sample set, in a sequence-like structure. This class is used in the initial uploading and manipulation of a dataset and supports converting 2 different types of data into a structure compatible with armory-library: + +- Tuples datasets which can be turned into a map +- PyTorch datasets needing to be turned into numpy arrays ::: charmory.data diff --git a/docs/reference/engine.md b/docs/reference/engine.md index e90b9431..30b94b1e 100644 --- a/docs/reference/engine.md +++ b/docs/reference/engine.md @@ -1,4 +1,7 @@ # Armory Engines +An engine is the core of the Armory library. There are two different types of engines which the user should choose from based on their overall objective: +- The Evaluation Engine performs model robustness evaluations as pertains to adversarial attacks. Can optionally be recorded in MLflow. +- The Adversarial Dataset Engine creates the adversarial dataset by applying an attack to each sample in the original dataset, outputting the results into a directory. Additional modifications to the samples may also be preformed. ::: charmory.engine.EvaluationEngine diff --git a/docs/reference/evaluation.md b/docs/reference/evaluation.md index 93b20601..e796d40b 100644 --- a/docs/reference/evaluation.md +++ b/docs/reference/evaluation.md @@ -1,3 +1,4 @@ # Evaluation Configuration +An evaluation is the complete testing of a model/attack. This class contains the configuration of the entire evaluation, to include details about the attack, dataset, evaluation, collected metrics, model, and host system configuration. ::: charmory.evaluation diff --git a/docs/reference/labels.md b/docs/reference/labels.md index 081ee696..3d90fa80 100644 --- a/docs/reference/labels.md +++ b/docs/reference/labels.md @@ -1,3 +1,13 @@ # Label Targeters +A label is the target/ground truth associated with a single sample. This class is a utility containing algorithms for generating or updating target labels in a variety of formats to include: + +- Single, fixed integer +- Single, fixed string +- Random label from a predefined list +- Fixed integer offset +- Fixed values as specified by an ordered list +- Exact value as the input label +- Replacement of object detection labels with fixed integer +- Transcript from a fixed list with the length closest to that of the input label ::: charmory.labels \ No newline at end of file diff --git a/docs/reference/model.md b/docs/reference/model.md index 8066f76a..994ffa95 100644 --- a/docs/reference/model.md +++ b/docs/reference/model.md @@ -1,4 +1,5 @@ # Model Ingestion and Adaptation +Models help adapt the input and output data of the evaluation to a user's needs. The pre- and post-adapters can be specified on the base ArmoryModel. The JATIC Image Classiifcation model class has a pre-defined output adapter specific to that task which includes logits, probabilities, and scores. The JATIC Object Detection model also has a pre-defined output adapter which includes boxes, logits, probabilities, and scores. The YOLOS Transformer has pre-defined input and output adapters for HuggingFace transformer YOLOS models. On input, the "labels" property becomes "class_labels". On output, the loss components are returned instead of the predictions since the model is put into training mode during attack generation. ::: charmory.model.base diff --git a/docs/reference/profiler.md b/docs/reference/profiler.md index 2b404320..a6992278 100644 --- a/docs/reference/profiler.md +++ b/docs/reference/profiler.md @@ -1,3 +1,4 @@ # Computational Performance Profilers +Profilers are responsible for generating benchmarking metrics for the evaluation. This class contains a choice of 3 profilers to collect computational metrics for evaluations: Basic, Deterministic, and Null. The first 2 provide an average CPU time for each type of computation. ::: armory.metrics.compute \ No newline at end of file diff --git a/docs/reference/task.md b/docs/reference/task.md index 2d7898e9..ed1135d2 100644 --- a/docs/reference/task.md +++ b/docs/reference/task.md @@ -1,4 +1,11 @@ # Evaluation Tasks +A task should match what the model is intended to do. Right now, the options are image classification and object detection, but more could be supported in the future. + +The class holds details of the entire task to include the evaluation configuration, whether to skip the benign and/or attack datasets, an optional adapter to be applied to the inference data prior to exporting to MLflow, and a frequency at which batches will be exported to MLflow, if at all. + +The Image Classification task can additionally be created with the total number of classes the model is capable of predicting, and the L-norm order for the perturbation distance metrics. + +The Object Detection task can additionally be created with an option to track Mean Average Precision (MAP) metrics per class, with a minimum prediction score for a detection bounding box to be drawn on the exported sample, a maximum intersection-over-union value for non-maximum suppression filtering of detection bounding boxes, and a minimum prediction score, with all predictions lower than this being ignored. ::: charmory.tasks.base.BaseEvaluationTask diff --git a/docs/reference/track.md b/docs/reference/track.md index 9da9a8fb..a994946f 100644 --- a/docs/reference/track.md +++ b/docs/reference/track.md @@ -1,3 +1,4 @@ # Evaluation Tracking +Utilities supporting the tracking of evaluations within armory-library. Contains functions to see/choose/clear which parameters to be logged with MLflow, the ability to set the MLflow tracking URI, name and describe the experiment, and actually track the evaluation and system metrics (CPU/GPU, disk, and network utilization). ::: charmory.track \ No newline at end of file diff --git a/docs/reference/transforms.md b/docs/reference/transforms.md index 09b1c875..7923e8bd 100644 --- a/docs/reference/transforms.md +++ b/docs/reference/transforms.md @@ -1,3 +1,4 @@ # Dataset Transforms +A transform is a (preprocessing) function on a model. This class contains general data transform functions for experimentation, to include Bounded Boxes (and the conversion between different formats of these), the resizing, padding, rescaling, and normalization (into z-scores) of images, image classification transforms, and object detection transforms. ::: charmory.experimental.transforms \ No newline at end of file diff --git a/docs/reference/utils.md b/docs/reference/utils.md index 88aacdf1..9a4c6df7 100644 --- a/docs/reference/utils.md +++ b/docs/reference/utils.md @@ -1,3 +1,10 @@ # Utilities +A variety of utilities to aid in evaluation tasks. Includes: + +- A custom torchvision transform which converts PIL images to numpy arrays +- A customizable transform which serves as the inverse of torchvision.transforms.Normalize +- The ability to apply a given ART pre- or post-processor defense to a model +- A transform that can be applied to JATIC-wrapped datasets using a preprocessor from a JATIC-wrapped model +- A check for whether a given estimator has any pre- or post-processor defenses applied to it ::: charmory.utils \ No newline at end of file diff --git a/library/src/charmory/model/base.py b/library/src/charmory/model/base.py index dfd9b600..d2e92467 100644 --- a/library/src/charmory/model/base.py +++ b/library/src/charmory/model/base.py @@ -20,7 +20,8 @@ class ArmoryModel(nn.Module): Wrapper around a model to apply an adapter to inputs and outputs of the model. - Example:: + Example: + from charmory.model import ArmoryModel def preadapter(images, *args, **kwargs):