Gatherer

Docker Image

This is a Docker container that uses the cisagov fork of 18F's domain-scan to gather domains as a precursor to scanning by pshtt, trustymail, and sslyze.

This Docker container is intended to be run via orchestrator.

N.B.: The secrets in the src/secrets directory are only used when testing via the docker-compose.yml composition. Normally this Docker container is run via the Docker composition in cisagov/orchestrator, which expects the secrets in a different location.

Running

Running with Docker

To run the cisagov/gatherer image via Docker:

docker run cisagov/gatherer:1.5.18

Running with Docker Compose

1. Create a docker-compose.yml file similar to the one below to use Docker Compose.

   ---
   version: "3.7"
   
   services:
     gatherer:
       image: cisagov/gatherer:1.5.18
       volumes:
         - type: bind
           source: <your_output_dir>
           target: /home/cisa/shared

2. Start the container and detach:

   docker compose up --detach

Using secrets with your container

This container also supports passing sensitive values via Docker secrets. Passing sensitive values like your credentials can be more secure using secrets than using environment variables. See the secrets section below for a table of all supported secret files.

1. To use secrets, create a database_creds.yml file in this format:

   ---
   version: '1'
   
   database:
     name: cyhy
     uri: mongodb://readonly:[email protected]:27017/cyhy
   

2. Then add the secret to your docker-compose.yml file:

   ---
   version: "3.7"
   
   secrets:
     database_creds:
       file: database_creds.yml
   
   services:
     gatherer:
       image: cisagov/gatherer:1.5.18
       volumes:
         - type: bind
           source: <your_output_dir>
           target: /home/cisa/shared
       secrets:
         - source: database_creds
           target: database_creds.yml

Updating your container

Docker Compose

1. Pull the new image from Docker Hub:

   docker compose pull

2. Recreate the running container by following the previous instructions:

   docker compose up --detach

Docker

1. Stop the running container:

   docker stop <container_id>

2. Pull the new image:

   docker pull cisagov/gatherer:1.5.18

3. Recreate and run the container by following the previous instructions.

Image tags

The images of this container are tagged with semantic versions of the underlying gatherer project that they containerize. It is recommended that most users use a version tag (e.g. :1.5.18).

Image:tag	Description
cisagov/gatherer:1.5.18	An exact release version.
cisagov/gatherer:1.5	The most recent release matching the major and minor version numbers.
cisagov/gatherer:1	The most recent release matching the major version number.
cisagov/gatherer:edge	The most recent image built from a merge into the develop branch of this repository.
cisagov/gatherer:nightly	A nightly build of the develop branch of this repository.
cisagov/gatherer:latest	The most recent release image pushed to a container registry. Pulling an image using the :latest tag should be avoided.

See the tags tab on Docker Hub for a list of all the supported tags.

Volumes

Mount point	Purpose
/home/cisa/shared	Output

Ports

There are no ports exposed by this container.

Environment variables

Required

There are no required environment variables.

Optional

There are no optional environment variables.

Secrets

Filename	Purpose
database_creds.yml	Cyber Hygiene database credentials in this format

Building from source

Build the image locally using this git repository as the build context:

docker build \
  --build-arg VERSION=1.5.18 \
  --tag cisagov/gatherer:1.5.18 \
  https://github.com/cisagov/gatherer.git#develop

Cross-platform builds

To create images that are compatible with other platforms, you can use the buildx feature of Docker:

1. Copy the project to your machine using the Code button above or the command line:

   git clone https://github.com/cisagov/gatherer.git
   cd gatherer

2. Create the Dockerfile-x file with buildx platform support:

   ./buildx-dockerfile.sh

3. Build the image using buildx:

   docker buildx build \
     --file Dockerfile-x \
     --platform linux/amd64 \
     --build-arg VERSION=1.5.18 \
     --output type=docker \
     --tag cisagov/gatherer:1.5.18 .

Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

License

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.