v3.0.0

Added

• Support for asyncio within the framework and plugins
• Better type checking support
• self.log may be leveraged from within all plugin classes
• Add new Error() class for standardizing errors from stoQ and plugins
  Error() will track plugin name, error message, and payload_id (optional)
• Add configuration properties from [Core] and [Documentation] to each plugin object when loaded
• PayloadMeta now has a should_scan boolean.
  Allows payloads to be logged and archived, but not scanned by worker plugin.
• Payload is now updated as results are completed.
  Results from completed scans will be available to other plugins instantly
• Request() class is passed to all dispatchers, workers, and archiver plugins.
  The Request object contains all payloads, request metadata, results, and errors from
  all other completed plugins. This will allow for all neccessary plugins to have a full
  understanding of the current state of the complete Request.
• WorkerPlugins now have a configuration option of required_workers.
  This allows for chained worker dependencies. If required_workers is defined, the
  parent plugin will not be run until all required plugins are completed successfully. The
  parent plugin may then use results from other completed plugins for their respective
  scanning tasks.
• Duplicate extracted payloads are no longer simply skipped, they are appended to
  Payload.results[].extracted_by and Payload.results[].extracted_from
• Add StoqConfigParser to stoq.helpers to extend options for Stoq and plugin configurations.
• Parallelization is performed across all of the plugins that can run in a given round,
  instead of parallelizing across all of the plugins to perform on a given payload (#147)
• Ensure plugin_name is set to the name of the plugin class in case Name is not defined in
  the plugin's configuration.

Changed

• PayloadResults is now an object of Payload.results, rather than an independent object
• Most objects have been removed from Payload and are now availabe in Payload.results,
  namely extracted_by, extracted_from, payload_id, size, payload_meta
• Payload.plugins_run moved to PayloadResults.plugins_runs and is now a Dict[str, List[str]]
  rather than Dict[str, List[List[str]]]
• PayloadResults.workers is now a Dict[str, Dict] rather than List[Dict[str, Dict]]
• PayloadMeta is now an object of PayloadResults.payload_meta
• PayloadResults.extracted_by is now a List[str] rather than str
• PayloadResults.extracted_from is now a List[str] rather than str
• Dispatchers run on each payload every round, instead of once per payload. This allows
  the dispatcher to take advantage of the request state model. (#147)
• Worker plugins can specify additional plugins to run on the payload they scan, effectively giving them dispatch capability.
  With YARA, for example, this allows us to directly scan with YARA and dispatch
  to other plugins by running YARA once. Otherwise, we would run YARA as a dispatcher,
  and then immediately run YARA again as a worker plugin. (#147)
• Archivers run at the very end along with connectors and decorators because we no
  longer scan a payload to completion at once. (#147)
• The default value for max_recursion has increased because the average number of
  worker rounds taken to complete a scan is expected to increase. (#147)

Deprecated

• DeepDispatcher plugin class has been removed
• Payload.plugins_run has been removed in favor of PayloadResults.plugins_run)
• Payload.worker_results has been removed in favor of PayloadResults.workers
• RequestMeta is no longer passed to plugins, in favor of the Request object
• plugins_opts has been removed from plugin __init__ function. All plugin configuration options
  are only available in self.config