Skip to content
/ snyky Public

A known vulnerable Flask app with an excessive amount of automated testing

License

Notifications You must be signed in to change notification settings

garethr/snyky

This branch is 1 commit ahead of master.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

400a088 · Jun 9, 2024

History

99 Commits
Dec 13, 2019
Nov 2, 2019
Feb 29, 2020
Nov 2, 2019
Nov 9, 2019
Oct 19, 2019
Dec 14, 2019
Oct 31, 2019
May 10, 2019
Jun 9, 2024
Oct 31, 2019
Nov 2, 2019
Dec 14, 2019
Dec 14, 2019
Dec 14, 2019
Feb 29, 2020
Oct 30, 2019
Oct 19, 2019
Dec 13, 2019

Repository files navigation

Snyky

The following application is used for demonstration purposes only. It contains a large number of overlapping integrations described below.

Policy

Snyky makes extensive use of Open Policy Agent to validate that various policies are met. This includes:

  • Checking properties of the pytest.ini file
  • Checking the build instructions in the Dockerfile
  • Checking dependencies in Pipfile
  • Checking the Kubernetes configuration in the Helm Chart

These policies can be applied in a variety of different ways. Note this is for demonstration purposes only, it's likely that you would only use one or two of these in a real application.

  1. Using Conftest
  2. Using GitHub Actions
  3. In CircleCI
  4. In a Tekton Pipeline
  5. Using Docker
  6. As part of a Python unit test suite
  7. Using Gatekeeper

1. Using Conftest

Conftest provides a developer focused user interface for Open Policy Agent. Let's first run the tests for our policies to make sure everything is in order:

$ conftest verify
PASS - policy/policy/pytest_test.rego - data.pytest.test_require_black
PASS - policy/policy/pytest_test.rego - data.pytest.test_require_isort
PASS - policy/policy/pytest_test.rego - data.pytest.test_require_isort_and_black
PASS - policy/policy/pytest_test.rego - data.pytest.test_recommend_coverage
PASS - policy/policy/pytest_test.rego - data.pytest.test_recommend_type_checker
PASS - policy/policy/pytest_test.rego - data.pytest.test_valid_with_required_options
PASS - policy/policy/pytest_test.rego - data.pytest.test_no_warnings_with_recommended_option

Then let's demontrate running some of our tests to verify our Pytest configuration meets with our defined policy.

$ conftest test --namespace pytest pytest.ini
WARN - pytest.ini - Consider enforcing type checking when running tests
WARN - pytest.ini - Consider enabling coverage reporting for test

You can see the policy in policy/pytest.rego.

The application is packaged as a Helm chart, and you can use the Conftest Helm plugin to render the chart template and run the resulting manifests through the local policy:

$ helm conftest snyky
FAIL - snyky in the Deployment garethr/snyky has an image, snyky, using the latest tag
FAIL - snyky in the Deployment snyky does not have a memory limit set
FAIL - snyky in the Deployment snyky does not have a CPU limit set
FAIL - snyky in the Deployment snyky doesn't drop all capabilities
FAIL - snyky in the Deployment snyky is not using a read only root filesystem
FAIL - snyky in the Deployment snyky allows priviledge escalation
FAIL - snyky in the Deployment snyky is running as root
Error: plugin "conftest" exited with erro

2. Using GitHub Actions

Conftest has a GitHub Action which makes integrating policy testing into GitHub easier. This includes Actions for using Conftest and a separate action for using the Conftest Helm plugin. You can see these running in this repository.

Policy

For the workflow definition see .github/workflows/policy.yml.

3. In CircleCI

Conftest has a CircleCI Orb which makes setting up Conftest in a CircleCI build straighforward. The Orb provides a number of different commands and you can see some of them in use in this repository.

CircleCI

For the build configuration see .circleci/config.yml.

4. In a Tekton Pipeline

Tekton provides a Kubernetes-native pipeline. The following requires you to have a Kubernetes cluster running but will install the latest version of Tekton, as well as a custom pipeline for this project.

$ make tekton-init
namespace/tekton-pipelines unchanged
podsecuritypolicy.policy/tekton-pipelines configured
clusterrole.rbac.authorization.k8s.io/tekton-pipelines-admin unchanged
serviceaccount/tekton-pipelines-controller unchanged
...
$ make tekton-pipeline
pipeline.tekton.dev/snyky-pipeline created
pipelineresource.tekton.dev/snyky-git created
task.tekton.dev/conftest-verify create

We can use the Tekton CLI to start a run of our pipeline:

$ tkn pipeline start snyky-pipelin
? Choose the git resource to use for source-repo: snyky-git (https://github.com/garethr/snyky.git)
Pipelinerun started: snyky-pipeline-run-xrg96

In order to track the pipelinerun progress run:
tkn pipelinerun logs snyky-pipeline-run-xrg96 -f -n defaul

We can also use tkn to grab the logs.

$ tkn pipelinerun logs snyky-pipeline-run-xrg96 -f -n default
...
[pytest-conftest : conftest] WARN - pytest.ini - Consider enforcing type checking when running tests
[pytest-conftest : conftest] WARN - pytest.ini - Consider enabling coverage reporting for tests

[conftest-verify : conftest-verify] PASS - policy/policy/pytest_test.rego - data.pytest.test_require_black
[conftest-verify : conftest-verify] PASS - policy/policy/pytest_test.rego - data.pytest.test_require_isort
[conftest-verify : conftest-verify] PASS - policy/policy/pytest_test.rego - data.pytest.test_require_isort_and_black
[conftest-verify : conftest-verify] PASS - policy/policy/pytest_test.rego - data.pytest.test_recommend_coverage
[conftest-verify : conftest-verify] PASS - policy/policy/pytest_test.rego - data.pytest.test_recommend_type_checker
[conftest-verify : conftest-verify] PASS - policy/policy/pytest_test.rego - data.pytest.test_valid_with_required_options
[conftest-verify : conftest-verify] PASS - policy/policy/pytest_test.rego - data.pytest.test_no_warnings_with_recommended_options
...

If you prefer a graphical tool then run the Tekton dashboard:

make tekton-dashboard

For the full pipeline configuration see tekton/pipeline.yaml.

5. Using Docker

There are two approaches to using Conftest with Docker. The simplest is just mounting the project and running the Conftest Docker image like so.

$ docker run --rm -v (pwd):/project instrumenta/conftest test snyky.yaml
FAIL - snyky.yaml - snyky in the Deployment garethr/snyky has an image, snyky, using the latest tag
FAIL - snyky.yaml - snyky in the Deployment snyky does not have a memory limit set
FAIL - snyky.yaml - snyky in the Deployment snyky does not have a CPU limit set
FAIL - snyky.yaml - snyky in the Deployment snyky doesn't drop all capabilities
FAIL - snyky.yaml - snyky in the Deployment snyky is not using a read only root filesystem
FAIL - snyky.yaml - snyky in the Deployment snyky is running as roo

A more advanced pattern is to add Conftest to a Docker image like so:

COPY --from=instrumenta/conftest /conftest /usr/local/bin/conftest

And then use it as part of a Docker build.

$ docker build --target Policy .
[+] Building 3.6s (18/18) FINISHED
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 37B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 2B                                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.7-alpine3.8                                                  0.0s
 => [internal] load build context                                                                                        0.1s
 => => transferring context: 36.87kB                                                                                     0.1s
 => FROM docker.io/instrumenta/conftest:latest                                                                           0.0s
 => [pipenv 1/2] FROM docker.io/library/python:3.7-alpine3.8                                                             0.0s
 => CACHED [pipenv 2/2] RUN pip3 install pipenv                                                                          0.0s
 => CACHED [parent 1/4] WORKDIR /app                                                                                     0.0s
 => CACHED [parent 2/4] COPY Pipfile /app/                                                                               0.0s
 => CACHED [parent 3/4] COPY Pipfile.lock /app/                                                                          0.0s
 => CACHED [parent 4/4] RUN apk add --no-cache --update git=2.18.1-r0                                                    0.0s
 => CACHED [dev-base 1/3] COPY --from=instrumenta/conftest /conftest /usr/local/bin/conftest                             0.0s
 => CACHED [dev-base 2/3] RUN pipenv install --dev                                                                       0.0s
 => CACHED [dev-base 3/3] COPY . /app                                                                                    0.0s
 => [policy 1/4] RUN conftest test --namespace pytest pytest.ini                                                         0.8s
 => [policy 2/4] RUN conftest test --namespace pipfile --input toml Pipfile                                              0.8s
 => [policy 3/4] RUN conftest test --namespace docker Dockerfile                                                         0.9s
 => ERROR [policy 4/4] RUN conftest test snyky.yaml                                                                      0.9s
------
 > [policy 4/4] RUN conftest test snyky.yaml:
#18 0.631 FAIL - snyky.yaml - snyky in the Deployment garethr/snyky has an image, snyky, using the latest tag
#18 0.631 FAIL - snyky.yaml - snyky in the Deployment snyky does not have a memory limit set
#18 0.631 FAIL - snyky.yaml - snyky in the Deployment snyky does not have a CPU limit set
#18 0.631 FAIL - snyky.yaml - snyky in the Deployment snyky doesn't drop all capabilities
#18 0.631 FAIL - snyky.yaml - snyky in the Deployment snyky is not using a read only root filesystem
#18 0.631 FAIL - snyky.yaml - snyky in the Deployment snyky is running as root
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c conftest test snyky.yaml]: runc did not terminate sucessfully

You can also use the Conftest Helm plugin via Docker as well:

$ docker run --rm -it -v (pwd):/chart instrumenta/helm-conftest conftest snyky
FAIL - snyky in the Deployment garethr/snyky has an image, snyky, using the latest tag
FAIL - snyky in the Deployment snyky does not have a memory limit set
FAIL - snyky in the Deployment snyky does not have a CPU limit set
FAIL - snyky in the Deployment snyky doesn't drop all capabilities
FAIL - snyky in the Deployment snyky is not using a read only root filesystem
FAIL - snyky in the Deployment snyky allows priviledge escalation
FAIL - snyky in the Deployment snyky is running as root
Error: plugin "conftest" exited with erro

6. As part of a Python unit test suite

Using Policykit it's possible to integrate Conftest output with Python, and to use it with a unit testing framework, in this case pytest.

$ pipenv run pytest src/test_policy.py
========================================= test session starts ==========================================
platform darwin -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
cachedir: .pytest_cache
rootdir: /Users/garethr/Documents/snyky, inifile: pytest.ini
plugins: isort-0.3.1, black-0.3.7, flask-0.15.0
collected 7 items

src/test_policy.py::BLACK SKIPPED                                                                 [ 14%]
src/test_policy.py::ISORT SKIPPED                                                                 [ 28%]
src/test_policy.py::TestPolicy::test_policy PASSED                                                [ 42%]
src/test_policy.py::TestPolicy::test_pytest_config PASSED                                         [ 57%]
src/test_policy.py::TestPolicy::test_pipfile PASSED                                               [ 71%]
src/test_policy.py::TestPolicy::test_dockerfile PASSED                                            [ 85%]
src/test_policy.py::TestPolicy::test_kubernetes_manifest_for_warnings PASSED                      [100%]

===================================== 5 passed, 2 skipped in 0.47s =====================================

You can see the unit tests in src/test_policy.py.

7. Using Gatekeeper

The repository is also setup to make using Gatekeeper possible as well.

First we need to generate a Gatekeeper ConstraintTemplate from our rego policies. This is done using the GitHub Action from Policykit.

Gatekeeper

This generates policy/SecurityControls.yaml.

(Note that currently this requires the new lib functionality currently in HEAD.)

kubectl apply -f policy/SecurityControls.yaml

With the ConstraintTemplate configured we can create a constraint:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: SecurityControls
metadata:
  name: enforce-deployment-and-pod-security-controls
spec:
  match:
    kinds:
      - apiGroups: ["batch", "extensions", "apps", ""]
        kinds: ["Deployment", "Pod", "CronJob", "Job", "StatefulSet", "DaemonSet", "ConfigMap", "Service"]

As configured above this will use the admission controller to block requests that do not meet our policies.

$ kubectl apply -f gatekeeper/deployment.yaml
Error from server ([denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment does not have a memory limit set
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment does not have a CPU limit set
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment doesn't drop all capabilities
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment is not using a read only root filesystem
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment is running as root): error when creating "deployment.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment does not have a memory limit set
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment does not have a CPU limit set
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment doesn't drop all capabilities
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment is not using a read only root filesystem
[denied by enforce-deployment-and-pod-security-controls] nginx in the Deployment nginx-deployment is running as roo

We can also set the policies up in dryrun mode and view any violations in the status field of the constraint.

$ kubectl get SecurityControls audit-deployment-and-pod-security-controls -o yaml
...
  - enforcementAction: dryrun
    kind: Deployment
    message: nginx in the Deployment nginx-deployment doesn't drop all capabilities
    name: nginx-deployment
    namespace: audit
  - enforcementAction: dryrun
    kind: Deployment
    message: nginx in the Deployment nginx-deployment is not using a read only root
      filesystem
    name: nginx-deployment
    namespace: audit
  - enforcementAction: dryrun
    kind: Deployment
    message: nginx in the Deployment nginx-deployment allows priviledge escalation
    name: nginx-deployment
    namespace: audit
  - enforcementAction: dryrun
    kind: Deployment
    message: nginx in the Deployment nginx-deployment is running as root
    name: nginx-deployment
    namespace: audit

Gatekeeper is opinionated about input and output, if you are interested in writing policies compatible with Gatekeeper and other OPA tools like Conftest then note the is_gatekeeper and gatekeeper_format rules in policy/lib/kubernetes.rego.

Vulnerabilities

Snyky also demonstrates different ways of integrating vulnerability scanning with a Python project using Snyk.

  1. Using Snyk locally
  2. Using GitHub Actions
  3. In CircleCI
  4. In a Tekton Pipeline
  5. Using Docker

In the Snyk dashboard this looks like:

Snyky in Snyk

1. Using Snyk locally

Snyk can be installed locally, using NPM, or using Homebrew or Scoop.

$ snyk test

Tested 7 dependencies for known issues, found 2 issues, 2 vulnerable paths.


Issues with no direct upgrade or patch:
  ✗ Improper Input Validation [High Severity][https://snyk.io/vuln/SNYK-PYTHON-FLASK-42185] in [email protected]
  This issue was fixed in versions: 0.12.3
  ✗ Denial of Service (DOS) [High Severity][https://snyk.io/vuln/SNYK-PYTHON-FLASK-451637] in [email protected]
  This issue was fixed in versions: 1.0



Organization:      garethr
Package manager:   pip
Target file:       Pipfile
Open source:       no
Project path:      /Users/garethr/Documents/snyky
Licenses:          enable

2. Using GitHub Actions

Snyk has a set of GitHub Actions which can be used to check for vulnerabilities in appications and Docker images.

Snyk

For the workflow definition see .github/workflows/snyk.yml.

3. In CircleCI

Snyk has a CircleCI Orb which can be used to check for vulnerabilities in your CI builds.

4. In a Tekton Pipeline

Snyk has a set of Tekton Tasks which can be used to check for vulnerabilities in your pipelines. Configuration is as simple as adding a step to your pipeline definition and setting a secret with the SNYK_TOKEN.

- name: snyk
  taskRef:
    name: snyk-python
  resources:
    inputs:
      - name: source
        resource: source-rep

From the pipeline above you should see the Snyk output in the logs:

$ tkn pipelinerun logs snyky-pipeline-run-xrg96 -f -n default
...
[snyk : snyk] All dependencies are now up-to-date!
[snyk : snyk]
[snyk : snyk] Testing /workspace/source...
[snyk : snyk]
[snyk : snyk] Tested 7 dependencies for known issues, found 2 issues, 2 vulnerable paths.
[snyk : snyk]
[snyk : snyk]
[snyk : snyk] Issues with no direct upgrade or patch:
[snyk : snyk]   ✗ Improper Input Validation [High Severity][https://snyk.io/vuln/SNYK-PYTHON-FLASK-42185] in [email protected]
[snyk : snyk]   This issue was fixed in versions: 0.12.3
[snyk : snyk]   ✗ Denial of Service (DOS) [High Severity][https://snyk.io/vuln/SNYK-PYTHON-FLASK-451637] in [email protected]
[snyk : snyk]   This issue was fixed in versions: 1.0
[snyk : snyk]
[snyk : snyk]
[snyk : snyk]
[snyk : snyk] Organization:      garethr
[snyk : snyk] Package manager:   pip
[snyk : snyk] Target file:       Pipfile
[snyk : snyk] Open source:       no
[snyk : snyk] Project path:      /workspace/source
[snyk : snyk] Licenses:          enabled
[snyk : snyk

5. Using Docker

You'll need a valid SNYK_TOKEN environment variable set to use Snyk via the Snyk Docker images.

docker run --rm -it --env SNYK_TOKEN -v $(pwd):/app snyk/snyk:python

For more advanced usecases you can copy Snyk into your image and use it as part of a build. You can use:

COPY --from=snyk/snyk:linux /usr/local/bin/snyk /usr/local/bin/snyk

Or if using an Alpine image use:

RUN apk add --no-cache libstdc++
COPY --from=snyk/snyk:alpine /usr/local/bin/snyk /usr/local/bin/snyk

You can see an example of this pattern in the Dockerfile, and you can run it like so:

$ docker build --build-arg SNYK_TOKEN --target Security .
[+] Building 21.2s (19/19) FINISHED
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 37B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 2B                                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.7-alpine3.8                                                  0.0s
 => CACHED FROM docker.io/snyk/snyk:alpine                                                                               0.0s
 => => resolve docker.io/snyk/snyk:alpine                                                                                1.2s
 => FROM docker.io/instrumenta/conftest:latest                                                                           0.0s
 => [pipenv 1/2] FROM docker.io/library/python:3.7-alpine3.8                                                             0.0s
 => [internal] load build context                                                                                        0.1s
 => => transferring context: 116.61kB                                                                                    0.1s
 => CACHED [pipenv 2/2] RUN pip3 install pipenv                                                                          0.0s
 => CACHED [parent 1/4] WORKDIR /app                                                                                     0.0s
 => CACHED [parent 2/4] COPY Pipfile /app/                                                                               0.0s
 => CACHED [parent 3/4] COPY Pipfile.lock /app/                                                                          0.0s
 => CACHED [parent 4/4] RUN apk add --no-cache --update git=2.18.1-r0                                                    0.0s
 => CACHED [dev-base 1/3] COPY --from=instrumenta/conftest /conftest /usr/local/bin/conftest                             0.0s
 => CACHED [dev-base 2/3] RUN pipenv install --dev                                                                       0.0s
 => [dev-base 3/3] COPY . /app                                                                                           0.2s
 => [security 1/4] RUN apk add --no-cache libstdc++                                                                      1.5s
 => [security 2/4] COPY --from=snyk/snyk:alpine /usr/local/bin/snyk /usr/local/bin/snyk                                  0.2s
 => [security 3/4] RUN pipenv update                                                                                    16.0s
 => ERROR [security 4/4] RUN snyk test                                                                                   3.1s
------
 > [security 4/4] RUN snyk test:
#19 2.716
#19 2.716 Testing /app...
#19 2.716
#19 2.716 Tested 7 dependencies for known issues, found 2 issues, 2 vulnerable paths.
#19 2.716
#19 2.716
#19 2.716 Issues with no direct upgrade or patch:
#19 2.716   ✗ Improper Input Validation [High Severity][https://snyk.io/vuln/SNYK-PYTHON-FLASK-42185] in [email protected]
#19 2.716   This issue was fixed in versions: 0.12.3
#19 2.716   ✗ Denial of Service (DOS) [High Severity][https://snyk.io/vuln/SNYK-PYTHON-FLASK-451637] in [email protected]
#19 2.716   This issue was fixed in versions: 1.0
#19 2.716
#19 2.716
#19 2.716
#19 2.716 Organization:      garethr
#19 2.716 Package manager:   pip
#19 2.716 Target file:       Pipfile
#19 2.716 Open source:       no
#19 2.716 Project path:      /app
#19 2.716 Licenses:          enabled
#19 2.717
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c snyk test]: runc did not terminate sucessfull

Installation

The full set of examples requires several tools to be installed:

About

A known vulnerable Flask app with an excessive amount of automated testing

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published