Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for CEL policy conditions #316

Merged
merged 47 commits into from
Sep 27, 2023
Merged

Add support for CEL policy conditions #316

merged 47 commits into from
Sep 27, 2023

Conversation

nscuro
Copy link
Member

@nscuro nscuro commented Sep 15, 2023

Description

This PR adds support for policy conditions using the Common Expression Language (CEL), as originally brought up in DependencyTrack/dependency-track#2673.

Corresponding frontend changes are here: https://github.com/nscuro/dependency-track-frontend/tree/cel-policies-hyades

TODOs

  • Implement remaining CelPolicyScriptSourceBuilders for compatibility with legacy conditions:
  • Implement tests for all CelPolicyScriptSourceBuilders
    • Won't be done; We have tests for the actual conditions now
  • @nscuro Reduce code duplication in CelPolicyEngine that was introduced during initial implementation
  • @nscuro Implement thorough test suite for CelPolicyEngine, that exercises:
    • All legacy policy conditions (new implementations should behave exactly like legacy ones)
    • All custom functions defined in CelPolicyLibrary
    • Loading of all fields defined in policy.proto
  • @nscuro Write documentation, including:
    • Overview of CEL
    • What variables, fields, functions, etc. are available
    • Example policies
    • Will be done in a separate PR in the hyades repository

Addressed Issue

N/A

Additional Details

CEL Language Definition: https://github.com/google/cel-spec/blob/master/doc/langdef.md
CEL-Java implementation by Project Nessie: https://github.com/projectnessie/cel-java

Example expressions

// Spring Framework components affected by CVE-2022-22965,
// where the project is packaged as WAR, and embedded Tomcat
// is present in the project.
vulns.exists(vuln, vuln.id == "CVE-2022-22965") 
  && component.name.matches("^spring-.*")
  && project.purl.contains("type=war")
  && project.depends_on(org.dependencytrack.policy.v1.Component{
       group: "org.apache.tomcat.embed",
       name: "tomcat-embed-core"
     })
// Components in public-facing projects, with at least one HIGH or CRITICAL 
// vulnerability, where the CVSSv3 attack vector is "Network".
"public-facing" in project.tags
  && vulns.exists(vuln, 
       vuln.severity in ["HIGH", "CRITICAL"] 
       && vuln.cvssv3_vector.matches(".*/AV:N/.*")
     )
// "Blacklist" of specific, high-profile vulnerabilities in projects acquired from a third party.
"3rd-party" in project.tags
  && vulns.exists(vuln, vuln.id in [
       "CVE-2017-5638",  // struts RCE
       "CVE-2021-44228", // log4shell
       "CVE-2022-22965", // spring4shell
     ])
// Components where the PURL matches a certain pattern,
// and the version matches a given range.
// 
// Supports PURL version range notation with 1-N (potentially ecosystem-specific) constraints:
//   https://github.com/package-url/purl-spec/blob/version-range-spec/VERSION-RANGE-SPEC.rst
component.purl.matches("^pkg:maven/com.acme/acme-lib\\b.*") 
   && component.matches_range("vers:maven/>0|<1|!=0.2.4")
// Components with non-permissive licenses.
component.resolved_license.groups.exists(licenseGroup, licenseGroup.name == "Permissive")

Behavior changes over legacy policy engine

  • Each policy condition can now only yield one violation per component. Previously, evaluators like SeverityPolicyEvaluator and VulnerabilityIdPolicyEvaluator could yield multiple violations; One for each matched vulnerability. Because violations are attached to components, not individual vulnerabilities, this behavior did not make sense. It is thus not carried over to the CEL policy engine.

Checklist

  • I have read and understand the contributing guidelines
  • This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
  • This PR introduces changes to the database model, and I have added corresponding update logic
  • This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

@nscuro nscuro added the enhancement New feature or request label Sep 15, 2023
Signed-off-by: nscuro <[email protected]>
Signed-off-by: nscuro <[email protected]>
DataNucleus on its own loads too much data, and does so using too many queries.

Signed-off-by: nscuro <[email protected]>
nscuro and others added 12 commits September 26, 2023 11:23
Using `escapeJson` doesn't work quite right when special characters / regular expressions are provided. All we need is prevention of "breaking out" of strings, so escaping double quotes alone is sufficient.

Signed-off-by: nscuro <[email protected]>
* added tests for hash policy

Signed-off-by: mehab <[email protected]>

* updated tests

Signed-off-by: mehab <[email protected]>

---------

Signed-off-by: mehab <[email protected]>
* Add version cel policy script builder

Signed-off-by: vithikashukla <[email protected]>

* add version support for coordinates cel policy

Signed-off-by: vithikashukla <[email protected]>

* Added unit test for version policy script builder

Signed-off-by: vithikashukla <[email protected]>

* added coordninates condition test

Signed-off-by: vithikashukla <[email protected]>

* added coordinates condition test

Signed-off-by: vithikashukla <[email protected]>

---------

Signed-off-by: vithikashukla <[email protected]>
Co-authored-by: vithikashukla <[email protected]>
* Add version cel policy script builder

Signed-off-by: vithikashukla <[email protected]>

* add version support for coordinates cel policy

Signed-off-by: vithikashukla <[email protected]>

* Added unit test for version policy script builder

Signed-off-by: vithikashukla <[email protected]>

* added coordninates condition test

Signed-off-by: vithikashukla <[email protected]>

* added coordinates condition test

Signed-off-by: vithikashukla <[email protected]>

* added more conditions to test

Signed-off-by: vithikashukla <[email protected]>

* Added license condition test

Signed-off-by: vithikashukla <[email protected]>

* Update src/main/java/org/dependencytrack/policy/cel/CelPolicyEngine.java

Co-authored-by: Niklas <[email protected]>
Signed-off-by: VithikaS <[email protected]>

* Added license group condition test

Signed-off-by: vithikashukla <[email protected]>

* updated comment

Signed-off-by: vithikashukla <[email protected]>

---------

Signed-off-by: vithikashukla <[email protected]>
Signed-off-by: VithikaS <[email protected]>
Co-authored-by: vithikashukla <[email protected]>
Co-authored-by: Niklas <[email protected]>
@nscuro nscuro marked this pull request as ready for review September 26, 2023 17:08
As this feature will be backported, we need to make sure policies will be compatible once folks start upgrading to Hyades.

Signed-off-by: nscuro <[email protected]>
@nscuro nscuro merged commit 5400bc5 into main Sep 27, 2023
5 of 6 checks passed
@nscuro nscuro deleted the cel-policies branch September 27, 2023 13:09
mehab pushed a commit that referenced this pull request Sep 30, 2023
Signed-off-by: mehab <[email protected]>
Add support for CEL policy conditions (#316)

* Initial commit of CEL policy work

Signed-off-by: nscuro <[email protected]>

* Add a few custom CEL functions

Signed-off-by: nscuro <[email protected]>

* Make policies work with legacy way of reporting violations

Signed-off-by: nscuro <[email protected]>

* Implement `is_dependency_of` CEL function

Signed-off-by: nscuro <[email protected]>

* Support vuln aliases in CEL policies

Signed-off-by: nscuro <[email protected]>

* Few minor adjustments

Signed-off-by: nscuro <[email protected]>

* Return CEL errors in API response

Signed-off-by: nscuro <[email protected]>

* Fix some vulnerability fields not being fetched for policies

Signed-off-by: nscuro <[email protected]>

* Bump `versatile` to `0.3.0`

Signed-off-by: nscuro <[email protected]>

* Use AST visitor to determine which fields are accessed for any given type

Signed-off-by: nscuro <[email protected]>

* Cleanup

Signed-off-by: nscuro <[email protected]>

* Cleanup

Signed-off-by: nscuro <[email protected]>

* WIP: Loading of required fields; Project policy evaluation

Signed-off-by: nscuro <[email protected]>

* Improve violation reconciliation for projects

Signed-off-by: nscuro <[email protected]>

* Add test with bloated BOM to debug performance bottlenecks

Signed-off-by: nscuro <[email protected]>

* Disable DataNucleus L1 cache for policy reconciliation

Signed-off-by: nscuro <[email protected]>

* Add field mapping tests

Signed-off-by: nscuro <[email protected]>

* Handle implicit policy script requirements for custom functions

Signed-off-by: nscuro <[email protected]>

* Minor readability and code documentation improvements

Signed-off-by: nscuro <[email protected]>

* Fetch data for policy violation notifications in a single query

DataNucleus on its own loads too much data, and does so using too many queries.

Signed-off-by: nscuro <[email protected]>

* Perform violation reconciliation using direct JDBC access

Signed-off-by: nscuro <[email protected]>

* Include strings library in CEL policy environment

Signed-off-by: nscuro <[email protected]>

* Cleanup; Support project properties, tags, and vulnerability aliases

Signed-off-by: nscuro <[email protected]>

* Add test to verify that all fields can be loaded

Signed-off-by: nscuro <[email protected]>

* Add remaining fields to `testWithAllFields`

Signed-off-by: nscuro <[email protected]>

* Add test for vuln severity evaluation

Signed-off-by: nscuro <[email protected]>

* Remove un-implemented `depends_on` function; Add proper logging for custom functions

Signed-off-by: nscuro <[email protected]>

* Handle invalid scripts and script runtime failures

Signed-off-by: nscuro <[email protected]>

* Add `escapeQuotes` for CEL script builders

Using `escapeJson` doesn't work quite right when special characters / regular expressions are provided. All we need is prevention of "breaking out" of strings, so escaping double quotes alone is sufficient.

Signed-off-by: nscuro <[email protected]>

* Add tests for some legacy conditions

Signed-off-by: nscuro <[email protected]>

* More tests for `CelPolicyEngine`

Signed-off-by: nscuro <[email protected]>

* Add more tests; Implement script cache bypass for REST API interactions

Signed-off-by: nscuro <[email protected]>

* Add tests for hash policy (#326)

* added tests for hash policy

Signed-off-by: mehab <[email protected]>

* updated tests

Signed-off-by: mehab <[email protected]>

---------

Signed-off-by: mehab <[email protected]>

* Add version cel policy script builder (#324)

* Add version cel policy script builder

Signed-off-by: vithikashukla <[email protected]>

* add version support for coordinates cel policy

Signed-off-by: vithikashukla <[email protected]>

* Added unit test for version policy script builder

Signed-off-by: vithikashukla <[email protected]>

* added coordninates condition test

Signed-off-by: vithikashukla <[email protected]>

* added coordinates condition test

Signed-off-by: vithikashukla <[email protected]>

---------

Signed-off-by: vithikashukla <[email protected]>
Co-authored-by: vithikashukla <[email protected]>

* Fix new UNIQUE constraint breaking existing behavior

Signed-off-by: nscuro <[email protected]>

* Add feature flag for CEL policy engine

Signed-off-by: nscuro <[email protected]>

* Add `UpgradeItem` to update type of `"POLICYCONDITION"."VALUE"` to `TEXT`

Signed-off-by: nscuro <[email protected]>

* Handle policy evaluation for individual components

Signed-off-by: nscuro <[email protected]>

* added unit tests for cwe cel policy

Signed-off-by: mehab <[email protected]>

* Add license condition test (#332)

* Add version cel policy script builder

Signed-off-by: vithikashukla <[email protected]>

* add version support for coordinates cel policy

Signed-off-by: vithikashukla <[email protected]>

* Added unit test for version policy script builder

Signed-off-by: vithikashukla <[email protected]>

* added coordninates condition test

Signed-off-by: vithikashukla <[email protected]>

* added coordinates condition test

Signed-off-by: vithikashukla <[email protected]>

* added more conditions to test

Signed-off-by: vithikashukla <[email protected]>

* Added license condition test

Signed-off-by: vithikashukla <[email protected]>

* Update src/main/java/org/dependencytrack/policy/cel/CelPolicyEngine.java

Co-authored-by: Niklas <[email protected]>
Signed-off-by: VithikaS <[email protected]>

* Added license group condition test

Signed-off-by: vithikashukla <[email protected]>

* updated comment

Signed-off-by: vithikashukla <[email protected]>

---------

Signed-off-by: vithikashukla <[email protected]>
Signed-off-by: VithikaS <[email protected]>
Co-authored-by: vithikashukla <[email protected]>
Co-authored-by: Niklas <[email protected]>

* Fix projection mapping for `Double` / `BigDecimal` fields

Signed-off-by: nscuro <[email protected]>

* support wildcard

Signed-off-by: vithikashukla <[email protected]>

* Add `buf` config and workflow

Signed-off-by: nscuro <[email protected]>

* Change Proto package from `hyades` to `dependencytrack`

As this feature will be backported, we need to make sure policies will be compatible once folks start upgrading to Hyades.

Signed-off-by: nscuro <[email protected]>

* Fix failing tests due to Proto package change

Signed-off-by: nscuro <[email protected]>

* Un-ignore `cyclonedx.proto` from breaking changes check

Signed-off-by: Niklas <[email protected]>

---------

Signed-off-by: nscuro <[email protected]>
Signed-off-by: mehab <[email protected]>
Signed-off-by: vithikashukla <[email protected]>
Signed-off-by: VithikaS <[email protected]>
Signed-off-by: Niklas <[email protected]>
Co-authored-by: meha <[email protected]>
Co-authored-by: VithikaS <[email protected]>
Co-authored-by: vithikashukla <[email protected]>
Co-authored-by: mehab <[email protected]>

changes for sending repo meta analysis events from apiserver

Signed-off-by: mehab <[email protected]>

initial refactoring

Signed-off-by: mehab <[email protected]>

refactored code

Signed-off-by: mehab <[email protected]>

code changes completed for sending

Signed-off-by: mehab <[email protected]>

fixed component resource unit tests

Signed-off-by: mehab <[email protected]>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants