Add support for CEL policy conditions (#316)

* Initial commit of CEL policy work Signed-off-by: nscuro <[email protected]> * Add a few custom CEL functions Signed-off-by: nscuro <[email protected]> * Make policies work with legacy way of reporting violations Signed-off-by: nscuro <[email protected]> * Implement `is_dependency_of` CEL function Signed-off-by: nscuro <[email protected]> * Support vuln aliases in CEL policies Signed-off-by: nscuro <[email protected]> * Few minor adjustments Signed-off-by: nscuro <[email protected]> * Return CEL errors in API response Signed-off-by: nscuro <[email protected]> * Fix some vulnerability fields not being fetched for policies Signed-off-by: nscuro <[email protected]> * Bump `versatile` to `0.3.0` Signed-off-by: nscuro <[email protected]> * Use AST visitor to determine which fields are accessed for any given type Signed-off-by: nscuro <[email protected]> * Cleanup Signed-off-by: nscuro <[email protected]> * Cleanup Signed-off-by: nscuro <[email protected]> * WIP: Loading of required fields; Project policy evaluation Signed-off-by: nscuro <[email protected]> * Improve violation reconciliation for projects Signed-off-by: nscuro <[email protected]> * Add test with bloated BOM to debug performance bottlenecks Signed-off-by: nscuro <[email protected]> * Disable DataNucleus L1 cache for policy reconciliation Signed-off-by: nscuro <[email protected]> * Add field mapping tests Signed-off-by: nscuro <[email protected]> * Handle implicit policy script requirements for custom functions Signed-off-by: nscuro <[email protected]> * Minor readability and code documentation improvements Signed-off-by: nscuro <[email protected]> * Fetch data for policy violation notifications in a single query DataNucleus on its own loads too much data, and does so using too many queries. Signed-off-by: nscuro <[email protected]> * Perform violation reconciliation using direct JDBC access Signed-off-by: nscuro <[email protected]> * Include strings library in CEL policy environment Signed-off-by: nscuro <[email protected]> * Cleanup; Support project properties, tags, and vulnerability aliases Signed-off-by: nscuro <[email protected]> * Add test to verify that all fields can be loaded Signed-off-by: nscuro <[email protected]> * Add remaining fields to `testWithAllFields` Signed-off-by: nscuro <[email protected]> * Add test for vuln severity evaluation Signed-off-by: nscuro <[email protected]> * Remove un-implemented `depends_on` function; Add proper logging for custom functions Signed-off-by: nscuro <[email protected]> * Handle invalid scripts and script runtime failures Signed-off-by: nscuro <[email protected]> * Add `escapeQuotes` for CEL script builders Using `escapeJson` doesn't work quite right when special characters / regular expressions are provided. All we need is prevention of "breaking out" of strings, so escaping double quotes alone is sufficient. Signed-off-by: nscuro <[email protected]> * Add tests for some legacy conditions Signed-off-by: nscuro <[email protected]> * More tests for `CelPolicyEngine` Signed-off-by: nscuro <[email protected]> * Add more tests; Implement script cache bypass for REST API interactions Signed-off-by: nscuro <[email protected]> * Add tests for hash policy (#326) * added tests for hash policy Signed-off-by: mehab <[email protected]> * updated tests Signed-off-by: mehab <[email protected]> --------- Signed-off-by: mehab <[email protected]> * Add version cel policy script builder (#324) * Add version cel policy script builder Signed-off-by: vithikashukla <[email protected]> * add version support for coordinates cel policy Signed-off-by: vithikashukla <[email protected]> * Added unit test for version policy script builder Signed-off-by: vithikashukla <[email protected]> * added coordninates condition test Signed-off-by: vithikashukla <[email protected]> * added coordinates condition test Signed-off-by: vithikashukla <[email protected]> --------- Signed-off-by: vithikashukla <[email protected]> Co-authored-by: vithikashukla <[email protected]> * Fix new UNIQUE constraint breaking existing behavior Signed-off-by: nscuro <[email protected]> * Add feature flag for CEL policy engine Signed-off-by: nscuro <[email protected]> * Add `UpgradeItem` to update type of `"POLICYCONDITION"."VALUE"` to `TEXT` Signed-off-by: nscuro <[email protected]> * Handle policy evaluation for individual components Signed-off-by: nscuro <[email protected]> * added unit tests for cwe cel policy Signed-off-by: mehab <[email protected]> * Add license condition test (#332) * Add version cel policy script builder Signed-off-by: vithikashukla <[email protected]> * add version support for coordinates cel policy Signed-off-by: vithikashukla <[email protected]> * Added unit test for version policy script builder Signed-off-by: vithikashukla <[email protected]> * added coordninates condition test Signed-off-by: vithikashukla <[email protected]> * added coordinates condition test Signed-off-by: vithikashukla <[email protected]> * added more conditions to test Signed-off-by: vithikashukla <[email protected]> * Added license condition test Signed-off-by: vithikashukla <[email protected]> * Update src/main/java/org/dependencytrack/policy/cel/CelPolicyEngine.java Co-authored-by: Niklas <[email protected]> Signed-off-by: VithikaS <[email protected]> * Added license group condition test Signed-off-by: vithikashukla <[email protected]> * updated comment Signed-off-by: vithikashukla <[email protected]> --------- Signed-off-by: vithikashukla <[email protected]> Signed-off-by: VithikaS <[email protected]> Co-authored-by: vithikashukla <[email protected]> Co-authored-by: Niklas <[email protected]> * Fix projection mapping for `Double` / `BigDecimal` fields Signed-off-by: nscuro <[email protected]> * support wildcard Signed-off-by: vithikashukla <[email protected]> * Add `buf` config and workflow Signed-off-by: nscuro <[email protected]> * Change Proto package from `hyades` to `dependencytrack` As this feature will be backported, we need to make sure policies will be compatible once folks start upgrading to Hyades. Signed-off-by: nscuro <[email protected]> * Fix failing tests due to Proto package change Signed-off-by: nscuro <[email protected]> * Un-ignore `cyclonedx.proto` from breaking changes check Signed-off-by: Niklas <[email protected]> --------- Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]> Signed-off-by: vithikashukla <[email protected]> Signed-off-by: VithikaS <[email protected]> Signed-off-by: Niklas <[email protected]> Co-authored-by: meha <[email protected]> Co-authored-by: VithikaS <[email protected]> Co-authored-by: vithikashukla <[email protected]> Co-authored-by: mehab <[email protected]>
DependencyTrack · Sep 27, 2023 · 5400bc5 · 5400bc5
1 parent 761ca26
commit 5400bc5
Show file tree

Hide file tree

Showing 65 changed files with 5,444 additions and 36 deletions.
diff --git a/.github/workflows/buf.yml b/.github/workflows/buf.yml
@@ -0,0 +1,29 @@
+name: Buf
+
+on:
+  pull_request:
+    branches: [ "main" ]
+
+permissions: { }
+
+jobs:
+  buf:
+    name: Buf
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+    - name: Checkout Repository
+      uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # tag=v4.1.0
+    - name: Setup buf
+      uses: bufbuild/buf-setup-action@eb60cd0de4f14f1f57cf346916b8cd69a9e7ed0b # tag=v1.26.1
+      with:
+        github_token: ${{ github.token }}
+    - name: Lint Protobuf
+      uses: bufbuild/buf-lint-action@bd48f53224baaaf0fc55de9a913e7680ca6dbea4 # tag=v1.0.3
+      with:
+        input: src/main/proto
+    - name: Detect Breaking Changes
+      uses: bufbuild/buf-breaking-action@a074e988ee34efcd4927079e79c611f428354c01 # tag=v1.1.3
+      with:
+        input: src/main/proto
+        against: https://github.com/${{ github.repository }}.git#branch=main,subdir=src/main/proto
diff --git a/pom.xml b/pom.xml
@@ -84,6 +84,7 @@
         <frontend.version>4.7.1</frontend.version>
         <lib.alpine.version>${project.parent.version}</lib.alpine.version>
         <lib.awaitility.version>4.2.0</lib.awaitility.version>
+        <lib.cel-tools.version>0.3.21</lib.cel-tools.version>
         <lib.cpe-parser.version>2.0.2</lib.cpe-parser.version>
         <lib.cvss-calculator.version>1.4.1</lib.cvss-calculator.version>
         <lib.owasp-rr-calculator.version>1.0.1</lib.owasp-rr-calculator.version>
@@ -102,6 +103,7 @@
         <lib.testcontainers.version>1.18.3</lib.testcontainers.version>
         <lib.resilience4j.version>2.0.1</lib.resilience4j.version>
         <lib.snappy-java.version>1.1.10.1</lib.snappy-java.version>
+        <lib.versatile.version>0.3.0</lib.versatile.version>
         <lib.woodstox.version>6.4.0</lib.woodstox.version>
         <lib.junit-params.version>1.1.1</lib.junit-params.version>
         <lib.log4j-over-slf4j.version>2.0.9</lib.log4j-over-slf4j.version>
@@ -193,6 +195,12 @@
                 </exclusion>
             </exclusions>
         </dependency>
+        <!-- Common Expression Language (CEL) -->
+        <dependency>
+            <groupId>org.projectnessie.cel</groupId>
+            <artifactId>cel-tools</artifactId>
+            <version>${lib.cel-tools.version}</version>
+        </dependency>
         <!-- CVSS Calculator -->
         <dependency>
             <groupId>us.springett</groupId>
@@ -326,6 +334,11 @@
             <artifactId>woodstox-core</artifactId>
             <version>${lib.woodstox.version}</version>
         </dependency>
+        <dependency>
+            <groupId>io.github.nscuro</groupId>
+            <artifactId>versatile</artifactId>
+            <version>${lib.versatile.version}</version>
+        </dependency>
         <dependency>
             <groupId>org.apache.maven</groupId>
             <artifactId>maven-artifact</artifactId>

diff --git a/src/main/java/org/dependencytrack/common/ConfigKey.java b/src/main/java/org/dependencytrack/common/ConfigKey.java
@@ -70,7 +70,8 @@ public enum ConfigKey implements Config.Key {
     BOM_UPLOAD_PROCESSING_TRX_FLUSH_THRESHOLD("bom.upload.processing.trx.flush.threshold", "10000"),
     WORKFLOW_RETENTION_DURATION("workflow.retention.duration", "P3D"),
     WORKFLOW_STEP_TIMEOUT_DURATION("workflow.step.timeout.duration", "PT1H"),
-    TMP_DELAY_BOM_PROCESSED_NOTIFICATION("tmp.delay.bom.processed.notification", "false");
+    TMP_DELAY_BOM_PROCESSED_NOTIFICATION("tmp.delay.bom.processed.notification", "false"),
+    CEL_POLICY_ENGINE_ENABLED("cel.policy.engine.enabled", "false");
 
     private final String propertyName;
     private final Object defaultValue;

diff --git a/src/main/java/org/dependencytrack/model/Policy.java b/src/main/java/org/dependencytrack/model/Policy.java
@@ -98,7 +98,7 @@ public enum ViolationState {
     @Column(name = "VIOLATIONSTATE", allowsNull = "false")
     @NotBlank
     @Size(min = 1, max = 255)
-    @Pattern(regexp = RegexSequence.Definition.PRINTABLE_CHARS, message = "The operator may only contain printable characters")
+    @Pattern(regexp = RegexSequence.Definition.PRINTABLE_CHARS, message = "The violation state may only contain printable characters")
     private ViolationState violationState;
 
     /**

diff --git a/src/main/java/org/dependencytrack/model/PolicyCondition.java b/src/main/java/org/dependencytrack/model/PolicyCondition.java
@@ -63,21 +63,28 @@ public enum Operator {
     }
 
     public enum Subject {
-        AGE,
+        AGE(PolicyViolation.Type.OPERATIONAL),
         //ANALYZER,
         //BOM,
-        COORDINATES,
-        CPE,
+        COORDINATES(PolicyViolation.Type.OPERATIONAL),
+        CPE(PolicyViolation.Type.OPERATIONAL),
         //INHERITED_RISK_SCORE,
-        LICENSE,
-        LICENSE_GROUP,
-        PACKAGE_URL,
-        SEVERITY,
-        SWID_TAGID,
-        VERSION,
-        COMPONENT_HASH,
-        CWE,
-        VULNERABILITY_ID
+        EXPRESSION(null),
+        LICENSE(PolicyViolation.Type.LICENSE),
+        LICENSE_GROUP(PolicyViolation.Type.LICENSE),
+        PACKAGE_URL(PolicyViolation.Type.OPERATIONAL),
+        SEVERITY(PolicyViolation.Type.SECURITY),
+        SWID_TAGID(PolicyViolation.Type.OPERATIONAL),
+        VERSION(PolicyViolation.Type.OPERATIONAL),
+        COMPONENT_HASH(PolicyViolation.Type.OPERATIONAL),
+        CWE(PolicyViolation.Type.SECURITY),
+        VULNERABILITY_ID(PolicyViolation.Type.SECURITY);
+
+        private final PolicyViolation.Type violationType;
+
+        Subject(final PolicyViolation.Type violationType) {
+            this.violationType = violationType;
+        }
     }
 
     @PrimaryKey
@@ -104,12 +111,18 @@ public enum Subject {
     private Subject subject;
 
     @Persistent
-    @Column(name = "VALUE", allowsNull = "false")
+    @Column(name = "VALUE", allowsNull = "false", jdbcType = "CLOB")
     @NotBlank
-    @Size(min = 1, max = 255)
+    @Size(min = 1)
     @Pattern(regexp = RegexSequence.Definition.PRINTABLE_CHARS, message = "The value may only contain printable characters")
     private String value;
 
+    @Persistent
+    @Column(name = "VIOLATIONTYPE", allowsNull = "true")
+    @Size(min = 1, max = 255)
+    @Pattern(regexp = RegexSequence.Definition.PRINTABLE_CHARS, message = "The violation type may only contain printable characters")
+    private PolicyViolation.Type violationType;
+
     /**
      * The unique identifier of the object.
      */
@@ -159,6 +172,18 @@ public void setValue(String value) {
         this.value = value;
     }
 
+    public PolicyViolation.Type getViolationType() {
+        if (subject != null && subject.violationType != null) {
+            return subject.violationType;
+        }
+
+        return violationType;
+    }
+
+    public void setViolationType(PolicyViolation.Type violationType) {
+        this.violationType = violationType;
+    }
+
     public UUID getUuid() {
         return uuid;
     }

diff --git a/src/main/java/org/dependencytrack/model/PolicyViolation.java b/src/main/java/org/dependencytrack/model/PolicyViolation.java
@@ -46,6 +46,9 @@
 @PersistenceCapable
 @JsonInclude(JsonInclude.Include.NON_NULL)
 @JsonIgnoreProperties(ignoreUnknown = true)
+// TODO: Add @Unique composite constraint on the fields component, policyCondition, and type.
+//   The legacy PolicyEngine erroneously allows for duplicates on those fields, but CelPolicyEngine
+//   will never produce such duplicates. Until we remove the legacy engine, we can't add this constraint.
 public class PolicyViolation implements Serializable {
 
     public enum Type {

diff --git a/src/main/java/org/dependencytrack/model/VulnerabilityAlias.java b/src/main/java/org/dependencytrack/model/VulnerabilityAlias.java
@@ -200,6 +200,7 @@ private String getBySource(final Vulnerability.Source source) {
             case NVD -> getCveId();
             case OSSINDEX -> getSonatypeId();
             case OSV -> getOsvId();
+            case SNYK -> getSnykId();
             case VULNDB -> getVulnDbId();
             default -> null;
         };

diff --git a/src/main/java/org/dependencytrack/parser/cyclonedx/CycloneDXExporter.java b/src/main/java/org/dependencytrack/parser/cyclonedx/CycloneDXExporter.java
@@ -83,7 +83,7 @@ private Bom create(final List<Component>components, final List<ServiceComponent>
         bom.setComponents(cycloneComponents);
         bom.setServices(cycloneServices);
         bom.setVulnerabilities(cycloneVulnerabilities);
-        if (components != null) {
+        if (cycloneComponents != null) {
             bom.setDependencies(ModelConverter.generateDependencies(project, components));
         }
         return bom;

diff --git a/src/main/java/org/dependencytrack/persistence/PolicyQueryManager.java b/src/main/java/org/dependencytrack/persistence/PolicyQueryManager.java
@@ -185,11 +185,26 @@ public Policy createPolicy(String name, Policy.Operator operator, Policy.Violati
      */
     public PolicyCondition createPolicyCondition(final Policy policy, final PolicyCondition.Subject subject,
                                                  final PolicyCondition.Operator operator, final String value) {
+        return createPolicyCondition(policy, subject, operator, value, null);
+    }
+
+    /**
+     * Creates a policy condition for the specified Project.
+     * @return the created PolicyCondition object
+     */
+    public PolicyCondition createPolicyCondition(final Policy policy, final PolicyCondition.Subject subject,
+                                                 final PolicyCondition.Operator operator, final String value,
+                                                 final PolicyViolation.Type violationType) {
         final PolicyCondition pc = new PolicyCondition();
         pc.setPolicy(policy);
         pc.setSubject(subject);
-        pc.setOperator(operator);
+        if (subject == PolicyCondition.Subject.EXPRESSION) {
+            pc.setOperator(PolicyCondition.Operator.MATCHES);
+        } else {
+            pc.setOperator(operator);
+        }
         pc.setValue(value);
+        pc.setViolationType(violationType);
         return persist(pc);
     }
 
@@ -200,8 +215,13 @@ public PolicyCondition createPolicyCondition(final Policy policy, final PolicyCo
     public PolicyCondition updatePolicyCondition(final PolicyCondition policyCondition) {
         final PolicyCondition pc = getObjectByUuid(PolicyCondition.class, policyCondition.getUuid());
         pc.setSubject(policyCondition.getSubject());
-        pc.setOperator(policyCondition.getOperator());
+        if (policyCondition.getSubject() == PolicyCondition.Subject.EXPRESSION) {
+            pc.setOperator(PolicyCondition.Operator.MATCHES);
+        } else {
+            pc.setOperator(policyCondition.getOperator());
+        }
         pc.setValue(policyCondition.getValue());
+        pc.setViolationType(policyCondition.getViolationType());
         return persist(pc);
     }
 

diff --git a/src/main/java/org/dependencytrack/persistence/QueryManager.java b/src/main/java/org/dependencytrack/persistence/QueryManager.java
@@ -655,6 +655,12 @@ public PolicyCondition createPolicyCondition(final Policy policy, final PolicyCo
         return getPolicyQueryManager().createPolicyCondition(policy, subject, operator, value);
     }
 
+    public PolicyCondition createPolicyCondition(final Policy policy, final PolicyCondition.Subject subject,
+                                                 final PolicyCondition.Operator operator, final String value,
+                                                 final PolicyViolation.Type violationType) {
+        return getPolicyQueryManager().createPolicyCondition(policy, subject, operator, value, violationType);
+    }
+
     public PolicyCondition updatePolicyCondition(final PolicyCondition policyCondition) {
         return getPolicyQueryManager().updatePolicyCondition(policyCondition);
     }

diff --git a/src/main/java/org/dependencytrack/policy/PolicyEngine.java b/src/main/java/org/dependencytrack/policy/PolicyEngine.java
@@ -213,6 +213,9 @@ public PolicyViolation.Type determineViolationType(final PolicyCondition.Subject
             case AGE, COORDINATES, PACKAGE_URL, CPE, SWID_TAGID, COMPONENT_HASH, VERSION ->
                     PolicyViolation.Type.OPERATIONAL;
             case LICENSE, LICENSE_GROUP -> PolicyViolation.Type.LICENSE;
+            // Just here to satisfy the switch exhaustiveness. Conditions with subject EXPRESSION
+            // will never yield any violations, because there's no evaluator supporting it.
+            case EXPRESSION -> null;
         };
     }