Merge pull request #186 from Santandersecurityresearch/develop

Updates to header rules along with bug fixes and general code and test improvements

CONTRIBUTING.md

@@ -92,10 +92,11 @@ Ready to contribute? Here's how to set up
 
 ## Pull Request Guidelines
 
-Before you submit a pull request, check that it meets these guidelines:
-
-1.  The pull request should include tests.
-2.  If the pull request adds functionality, the docs should be updated.
-    Put your new functionality into a function with a docstring, and add
-    the feature to the list in README.md.
-3.  The pull request should work for Python 3.5, 3.6 and 3.7
+When submitting a pull request, please ensure that:
+
+1.  You submit it to 'develop' branch and there's no conflicts.
+2.  You check all tests are passing and have created new ones if change not covered in current test suite. 
+3.  You update `README.md` if functionality has been added or modified. If you are creating new classes or methods, please use docstring to document the code.
+4.  You update `RULES.md` when extending or modifying the way rules can be used, adding documentation and examples for the new/modified feature. 
+5.  Code works for Python >= 3.7
+6.  Once PR is submitted, workflow steps are successful (e.g.: Flake8, Bandit, Safety, etc.)

README.md

@@ -1,9 +1,9 @@
 [![GitHub release](https://img.shields.io/github/release/Santandersecurityresearch/DrHeader.svg)](https://GitHub.com/Santandersecurityresearch/DrHeader/releases/)
 [![Github all releases](https://img.shields.io/github/downloads/Santandersecurityresearch/DrHeader/total.svg)](https://GitHub.com/Santandersecurityresearch/DrHeader/releases/)
-[![HitCount](http://hits.dwyl.io/Santandersecurityresearch/DrHeader.svg)](http://hits.dwyl.io/Santandersecurityresearch/DrHeader)
+[![HitCount](https://hits.dwyl.com/Santandersecurityresearch/DrHeader.svg)](https://hits.dwyl.com/Santandersecurityresearch/DrHeader)
 [![Total alerts](https://img.shields.io/lgtm/alerts/g/Santandersecurityresearch/DrHeader.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/Santandersecurityresearch/DrHeader/alerts/)
 [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/Santandersecurityresearch/DrHeader.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/Santandersecurityresearch/DrHeader/context:python)
-[![MIT license](http://img.shields.io/badge/license-MIT-brightgreen.svg)](http://opensource.org/licenses/MIT)
+[![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://opensource.org/licenses/MIT)
 
 ![drHEADer](assets/img/hero.png)
 
@@ -62,7 +62,6 @@ $ drheader scan bulk -ff targets.txt
 There are a number of parameters you can specify during bulk scans, these are:
 | Option            | Description                                            |
 | :---------------- | :----------------------------------------------------- |
-| -p, --post        | Use a post request to obtain headers                   |
 | --json            | Output report as json                                  |
 | --debug           | Show error messages                                    |
 | --rules FILENAME  | Use custom rule set                                    |
@@ -102,7 +101,7 @@ By default, the tool uses **GET** method when making a request, but you can chan
 drheader_instance = Drheader(url="http://test.com", method="POST")
 ```
 
-Remember you can use any method supported by `requests` such as POST, PUT, GET and DELETE.
+Remember you can use any method supported by `requests` such as POST, PUT, GET and DELETE. This feature is not currently available in the CLI. It will be added in an upcoming release
 
 At the same time, you can customize the headers sent by the request. For that, you just have to use the `request_headers` argument:
 

RULES.md

@@ -1,6 +1,6 @@
 # Introduction
 
-This document describes the format of the `rules.yml` file. This file defines the policy drHEADer relies on to audit security headers. It also documents how to make changes to it so that you can configure your custom policy based on your particular requirements. 
+This document describes the format of the `rules.yml` file. This file defines the policy drHEADer relies on to audit security headers. It also documents how to make changes to it so that you can configure your custom policy based on your particular requirements.
 
 # File Format
 
@@ -33,6 +33,18 @@ Headers:
     Must-Avoid:
     - unsafe-inline
     - unsafe-eval
+    Directives:
+      script-src:
+        Required: True
+        Enforce: True
+        Value:
+        - self
+      style-src:
+        Required: True
+        Enforce: False
+        Value:
+        Must-Avoid:
+        - http://www.example.com
   Set-Cookie:
     Required: Optional
     Enforce: False
@@ -53,22 +65,24 @@ Headers:
 The yaml file structure for drHEADwe is as follows:
 
 * There must always be a root element with name 'Headers:'
-* Inside the root element, there must be as many elements as headers you want drHEADer to audit (ie: Content-Security-Policy, Set-Cookie, etc.)
+* Inside the root element, there must be as many elements as headers you want drHEADer to audit (ie: Content-Security-Policy, Set-Cookie, etc.).
+* Within each header, rules can be set for individual directives (directives are only applicable if they are set using a key-value format, such as for those in the Content-Security-Policy).
+  The rules for the directives must be defined under a root element 'Directives' under the relevant header
 * For each of these elements (or security headers to audit), the following flags can be set based on the specific requirements for that header:
     * Required:
         * 'True' if header is required to be present in the HTTP response
-        * 'False' if header is not required to be present in the HTTP response
+        * 'False' if header is required not to be present in the HTTP response
         * 'Optional' if header can be present in the HTTP response but is not mandatory
     * Enforce:
         * 'True' if the policy enforces a value for that header (full match)
         * 'False' if the policy does not enforce a value for that header
     * Value:
         * It must be empty if 'Enforce' is set to False, otherwise
         * It must be set to a list of values that would be accepted for that header. The validation will be successful if there is a full match (value in header matches with value in policy) with one of the values defined
-    * Delimiter: To be used when a header is enforced and the value specified contains multiple values that would be valid in any order (see example for Cache-Control). Default delimiter is ';'.  
-    * Must-Contain: To be used when 'Required' is set to True or Optional and 'Enforce' is set to False.  
+    * Delimiter: To be used when a header is enforced and the value specified contains multiple values that would be valid in any order (see example for Cache-Control). Default delimiter is ';'.
+    * Must-Contain: To be used when 'Required' is set to True or Optional and 'Enforce' is set to False.
         * It can be set to a list of values that should be part of the header value. The validation will be successful if all values specified are found in the value set for that header (ie: for set-cookie the policy specifies that httponly AND secure should be part of the header value)
-    * Must-Contain-One: To be used when 'Required' is set to True or Optional and 'Enforce' is set to False.  
+    * Must-Contain-One: To be used when 'Required' is set to True or Optional and 'Enforce' is set to False.
         * It can be set to a list of values where at least one should be part of the header. The validation will be successful if at least one value is found in the value set for that header (ie: for Content-Security-Policy the policy specifies that either "default-src 'none'" OR "default-src 'self'" should be part of the header value)
     * Must-Avoid: To be used when 'Required' is set to True or Optional and 'Enforce' is set to False.
         * It can be set to a list of values that should not be part of the header. The validation will be successful if none of the values are found in the value set for that header (ie: for Content-Security-Policy the policy specifies that "unsafe-inline" AND "unsafe-eval" should not be part of the header value)

drheader/__init__.py

@@ -2,6 +2,6 @@
 
 """Top-level package for drHEADer core."""
 
-__version__ = '1.5.3'
+__version__ = '1.6.0'
 
 from drheader.core import Drheader  # noqa

drheader/cli.py

@@ -4,11 +4,11 @@
 
 import json
 import logging
+import os
+import sys
 
 import click
 import jsonschema
-import sys
-import os
 import validators
 from tabulate import tabulate
 
@@ -17,7 +17,6 @@
 from drheader.cli_utils import echo_bulk_report, file_junit_report
 from drheader.utils import load_rules, get_rules_from_uri
 
-
 EXIT_CODE_NO_ERROR = os.EX_OK
 EXIT_CODE_FAILURE = os.EX_SOFTWARE
 
@@ -219,14 +218,13 @@ def single(ctx, target_url, json_output, debug, rule_file, rule_uri, merge, juni
 @click.argument('file', type=click.File(), required=True)
 @click.option('--file-format', '-ff', 'input_format', type=click.Choice(['json', 'txt']),
               help='bulk file input type. (defaults to json)')
-@click.option('--post', '-p', help='Use a post request to obtain headers', is_flag=True)
 @click.option('--json', 'json_output', help='Output report as json', is_flag=True)
 @click.option('--debug', help='Show error messages', is_flag=True)
 @click.option('--rules', 'rule_file', help='Use custom rule set', type=click.File())
 @click.option('--rules-uri', 'rule_uri', help='Use custom rule set, downloaded from URI')
 @click.option('--merge', help='Merge custom file rules, on top of default rules', is_flag=True)
 @click.pass_context
-def bulk(ctx, file, json_output, post, input_format, debug, rule_file, rule_uri, merge):
+def bulk(ctx, file, json_output, input_format, debug, rule_file, rule_uri, merge):
     """
     Scan multiple http(s) endpoints with drheader.
 
@@ -307,7 +305,7 @@ def bulk(ctx, file, json_output, post, input_format, debug, rule_file, rule_uri,
     for i, v in enumerate(urls):
         logging.debug('Querying: {}...'.format(v))
         drheader_instance = Drheader(
-            url=v['url'], post=post, params=v.get('params', None), verify=ctx.obj['verify'])
+            url=v['url'], params=v.get('params', None), verify=ctx.obj['verify'])
         logging.debug('Analysing: {}...'.format(v))
         drheader_instance.analyze(rules)
         audit.append({'url': v['url'], 'report': drheader_instance.report})

drheader/cli_utils.py

@@ -3,11 +3,9 @@
 """Utils for drheader console script."""
 
 import json
-
-import click
-
 import os
 
+import click
 from junit_xml import TestSuite, TestCase