-
-
Notifications
You must be signed in to change notification settings - Fork 11
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* integrity events sent from apiserver
Signed-off-by: mehab <[email protected]> Add support for CEL policy conditions (#316) * Initial commit of CEL policy work Signed-off-by: nscuro <[email protected]> * Add a few custom CEL functions Signed-off-by: nscuro <[email protected]> * Make policies work with legacy way of reporting violations Signed-off-by: nscuro <[email protected]> * Implement `is_dependency_of` CEL function Signed-off-by: nscuro <[email protected]> * Support vuln aliases in CEL policies Signed-off-by: nscuro <[email protected]> * Few minor adjustments Signed-off-by: nscuro <[email protected]> * Return CEL errors in API response Signed-off-by: nscuro <[email protected]> * Fix some vulnerability fields not being fetched for policies Signed-off-by: nscuro <[email protected]> * Bump `versatile` to `0.3.0` Signed-off-by: nscuro <[email protected]> * Use AST visitor to determine which fields are accessed for any given type Signed-off-by: nscuro <[email protected]> * Cleanup Signed-off-by: nscuro <[email protected]> * Cleanup Signed-off-by: nscuro <[email protected]> * WIP: Loading of required fields; Project policy evaluation Signed-off-by: nscuro <[email protected]> * Improve violation reconciliation for projects Signed-off-by: nscuro <[email protected]> * Add test with bloated BOM to debug performance bottlenecks Signed-off-by: nscuro <[email protected]> * Disable DataNucleus L1 cache for policy reconciliation Signed-off-by: nscuro <[email protected]> * Add field mapping tests Signed-off-by: nscuro <[email protected]> * Handle implicit policy script requirements for custom functions Signed-off-by: nscuro <[email protected]> * Minor readability and code documentation improvements Signed-off-by: nscuro <[email protected]> * Fetch data for policy violation notifications in a single query DataNucleus on its own loads too much data, and does so using too many queries. Signed-off-by: nscuro <[email protected]> * Perform violation reconciliation using direct JDBC access Signed-off-by: nscuro <[email protected]> * Include strings library in CEL policy environment Signed-off-by: nscuro <[email protected]> * Cleanup; Support project properties, tags, and vulnerability aliases Signed-off-by: nscuro <[email protected]> * Add test to verify that all fields can be loaded Signed-off-by: nscuro <[email protected]> * Add remaining fields to `testWithAllFields` Signed-off-by: nscuro <[email protected]> * Add test for vuln severity evaluation Signed-off-by: nscuro <[email protected]> * Remove un-implemented `depends_on` function; Add proper logging for custom functions Signed-off-by: nscuro <[email protected]> * Handle invalid scripts and script runtime failures Signed-off-by: nscuro <[email protected]> * Add `escapeQuotes` for CEL script builders Using `escapeJson` doesn't work quite right when special characters / regular expressions are provided. All we need is prevention of "breaking out" of strings, so escaping double quotes alone is sufficient. Signed-off-by: nscuro <[email protected]> * Add tests for some legacy conditions Signed-off-by: nscuro <[email protected]> * More tests for `CelPolicyEngine` Signed-off-by: nscuro <[email protected]> * Add more tests; Implement script cache bypass for REST API interactions Signed-off-by: nscuro <[email protected]> * Add tests for hash policy (#326) * added tests for hash policy Signed-off-by: mehab <[email protected]> * updated tests Signed-off-by: mehab <[email protected]> --------- Signed-off-by: mehab <[email protected]> * Add version cel policy script builder (#324) * Add version cel policy script builder Signed-off-by: vithikashukla <[email protected]> * add version support for coordinates cel policy Signed-off-by: vithikashukla <[email protected]> * Added unit test for version policy script builder Signed-off-by: vithikashukla <[email protected]> * added coordninates condition test Signed-off-by: vithikashukla <[email protected]> * added coordinates condition test Signed-off-by: vithikashukla <[email protected]> --------- Signed-off-by: vithikashukla <[email protected]> Co-authored-by: vithikashukla <[email protected]> * Fix new UNIQUE constraint breaking existing behavior Signed-off-by: nscuro <[email protected]> * Add feature flag for CEL policy engine Signed-off-by: nscuro <[email protected]> * Add `UpgradeItem` to update type of `"POLICYCONDITION"."VALUE"` to `TEXT` Signed-off-by: nscuro <[email protected]> * Handle policy evaluation for individual components Signed-off-by: nscuro <[email protected]> * added unit tests for cwe cel policy Signed-off-by: mehab <[email protected]> * Add license condition test (#332) * Add version cel policy script builder Signed-off-by: vithikashukla <[email protected]> * add version support for coordinates cel policy Signed-off-by: vithikashukla <[email protected]> * Added unit test for version policy script builder Signed-off-by: vithikashukla <[email protected]> * added coordninates condition test Signed-off-by: vithikashukla <[email protected]> * added coordinates condition test Signed-off-by: vithikashukla <[email protected]> * added more conditions to test Signed-off-by: vithikashukla <[email protected]> * Added license condition test Signed-off-by: vithikashukla <[email protected]> * Update src/main/java/org/dependencytrack/policy/cel/CelPolicyEngine.java Co-authored-by: Niklas <[email protected]> Signed-off-by: VithikaS <[email protected]> * Added license group condition test Signed-off-by: vithikashukla <[email protected]> * updated comment Signed-off-by: vithikashukla <[email protected]> --------- Signed-off-by: vithikashukla <[email protected]> Signed-off-by: VithikaS <[email protected]> Co-authored-by: vithikashukla <[email protected]> Co-authored-by: Niklas <[email protected]> * Fix projection mapping for `Double` / `BigDecimal` fields Signed-off-by: nscuro <[email protected]> * support wildcard Signed-off-by: vithikashukla <[email protected]> * Add `buf` config and workflow Signed-off-by: nscuro <[email protected]> * Change Proto package from `hyades` to `dependencytrack` As this feature will be backported, we need to make sure policies will be compatible once folks start upgrading to Hyades. Signed-off-by: nscuro <[email protected]> * Fix failing tests due to Proto package change Signed-off-by: nscuro <[email protected]> * Un-ignore `cyclonedx.proto` from breaking changes check Signed-off-by: Niklas <[email protected]> --------- Signed-off-by: nscuro <[email protected]> Signed-off-by: mehab <[email protected]> Signed-off-by: vithikashukla <[email protected]> Signed-off-by: VithikaS <[email protected]> Signed-off-by: Niklas <[email protected]> Co-authored-by: meha <[email protected]> Co-authored-by: VithikaS <[email protected]> Co-authored-by: vithikashukla <[email protected]> Co-authored-by: mehab <[email protected]> changes for sending repo meta analysis events from apiserver Signed-off-by: mehab <[email protected]> initial refactoring Signed-off-by: mehab <[email protected]> refactored code Signed-off-by: mehab <[email protected]> code changes completed for sending Signed-off-by: mehab <[email protected]> fixed component resource unit tests Signed-off-by: mehab <[email protected]>
- Loading branch information
Showing
82 changed files
with
5,702 additions
and
72 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
name: Buf | ||
|
||
on: | ||
pull_request: | ||
branches: [ "main" ] | ||
|
||
permissions: { } | ||
|
||
jobs: | ||
buf: | ||
name: Buf | ||
runs-on: ubuntu-latest | ||
timeout-minutes: 5 | ||
steps: | ||
- name: Checkout Repository | ||
uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # tag=v4.1.0 | ||
- name: Setup buf | ||
uses: bufbuild/buf-setup-action@eb60cd0de4f14f1f57cf346916b8cd69a9e7ed0b # tag=v1.26.1 | ||
with: | ||
github_token: ${{ github.token }} | ||
- name: Lint Protobuf | ||
uses: bufbuild/buf-lint-action@bd48f53224baaaf0fc55de9a913e7680ca6dbea4 # tag=v1.0.3 | ||
with: | ||
input: src/main/proto | ||
- name: Detect Breaking Changes | ||
uses: bufbuild/buf-breaking-action@a074e988ee34efcd4927079e79c611f428354c01 # tag=v1.1.3 | ||
with: | ||
input: src/main/proto | ||
against: https://github.com/${{ github.repository }}.git#branch=main,subdir=src/main/proto |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
17 changes: 7 additions & 10 deletions
17
src/main/java/org/dependencytrack/event/ComponentRepositoryMetaAnalysisEvent.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,21 +1,18 @@ | ||
package org.dependencytrack.event; | ||
|
||
import alpine.event.framework.Event; | ||
import com.github.packageurl.PackageURL; | ||
import org.dependencytrack.model.Component; | ||
|
||
import java.util.Optional; | ||
|
||
/** | ||
* Defines an {@link Event} triggered when requesting a component to be analyzed for meta information. | ||
* | ||
* @param purlCoordinates The package URL coordinates of the {@link Component} to analyze | ||
* @param internal Whether the {@link Component} is internal | ||
* @param purlCoordinates The package URL coordinates of the {@link Component} to analyze | ||
* @param internal Whether the {@link Component} is internal | ||
* @param fetchIntegrityData Whether component hash information needs to be fetched from external api | ||
* @param fetchLatestVersion Whether to fetch latest version meta information for a component. | ||
*/ | ||
public record ComponentRepositoryMetaAnalysisEvent(String purlCoordinates, Boolean internal) implements Event { | ||
|
||
public ComponentRepositoryMetaAnalysisEvent(final Component component) { | ||
this(Optional.ofNullable(component.getPurlCoordinates()).map(PackageURL::canonicalize).orElse(null), component.isInternal()); | ||
} | ||
public record ComponentRepositoryMetaAnalysisEvent(String purlCoordinates, Boolean internal, | ||
boolean fetchIntegrityData, | ||
boolean fetchLatestVersion) implements Event { | ||
|
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
src/main/java/org/dependencytrack/event/kafka/componentmeta/AbstractMetaHandler.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
package org.dependencytrack.event.kafka.componentmeta; | ||
|
||
import org.dependencytrack.event.kafka.KafkaEventDispatcher; | ||
import org.dependencytrack.model.FetchStatus; | ||
import org.dependencytrack.model.IntegrityMetaComponent; | ||
import org.dependencytrack.persistence.QueryManager; | ||
|
||
import java.time.Instant; | ||
import java.util.Date; | ||
|
||
public abstract class AbstractMetaHandler implements Handler { | ||
|
||
ComponentProjection componentProjection; | ||
QueryManager queryManager; | ||
KafkaEventDispatcher kafkaEventDispatcher; | ||
boolean fetchLatestVersion; | ||
|
||
|
||
public static IntegrityMetaComponent createIntegrityMetaComponent(String purl) { | ||
IntegrityMetaComponent integrityMetaComponent1 = new IntegrityMetaComponent(); | ||
integrityMetaComponent1.setStatus(FetchStatus.IN_PROGRESS); | ||
integrityMetaComponent1.setPurl(purl); | ||
integrityMetaComponent1.setLastFetch(Date.from(Instant.now())); | ||
return integrityMetaComponent1; | ||
} | ||
|
||
} |
4 changes: 4 additions & 0 deletions
4
src/main/java/org/dependencytrack/event/kafka/componentmeta/ComponentProjection.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
package org.dependencytrack.event.kafka.componentmeta; | ||
|
||
public record ComponentProjection(String purlCoordinates, Boolean internal, String purl) { | ||
} |
5 changes: 5 additions & 0 deletions
5
src/main/java/org/dependencytrack/event/kafka/componentmeta/Handler.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
package org.dependencytrack.event.kafka.componentmeta; | ||
|
||
public interface Handler { | ||
void handle(); | ||
} |
19 changes: 19 additions & 0 deletions
19
src/main/java/org/dependencytrack/event/kafka/componentmeta/HandlerFactory.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
package org.dependencytrack.event.kafka.componentmeta; | ||
|
||
import com.github.packageurl.MalformedPackageURLException; | ||
import com.github.packageurl.PackageURL; | ||
import org.dependencytrack.event.kafka.KafkaEventDispatcher; | ||
import org.dependencytrack.persistence.QueryManager; | ||
|
||
public class HandlerFactory { | ||
|
||
public static Handler createHandler(ComponentProjection componentProjection, QueryManager queryManager, KafkaEventDispatcher kafkaEventDispatcher, boolean fetchLatestVersion) throws MalformedPackageURLException { | ||
PackageURL packageURL = new PackageURL(componentProjection.purl()); | ||
boolean result = RepoMetaConstants.SUPPORTED_PACKAGE_URLS_FOR_INTEGRITY_CHECK.contains(packageURL.getType()); | ||
if (result) { | ||
return new SupportedMetaHandler(componentProjection, queryManager, kafkaEventDispatcher, fetchLatestVersion); | ||
} else { | ||
return new UnSupportedMetaHandler(componentProjection, queryManager, kafkaEventDispatcher, fetchLatestVersion); | ||
} | ||
} | ||
} |
9 changes: 9 additions & 0 deletions
9
src/main/java/org/dependencytrack/event/kafka/componentmeta/RepoMetaConstants.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
package org.dependencytrack.event.kafka.componentmeta; | ||
|
||
import java.util.List; | ||
|
||
public class RepoMetaConstants { | ||
|
||
public static final long TIME_SPAN = 60 * 60 * 1000L; | ||
public static final List<String> SUPPORTED_PACKAGE_URLS_FOR_INTEGRITY_CHECK =List.of("maven", "npm", "pypi"); | ||
} |
43 changes: 43 additions & 0 deletions
43
src/main/java/org/dependencytrack/event/kafka/componentmeta/SupportedMetaHandler.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
package org.dependencytrack.event.kafka.componentmeta; | ||
|
||
import org.dependencytrack.event.ComponentRepositoryMetaAnalysisEvent; | ||
import org.dependencytrack.event.kafka.KafkaEventDispatcher; | ||
import org.dependencytrack.model.FetchStatus; | ||
import org.dependencytrack.model.IntegrityMetaComponent; | ||
import org.dependencytrack.persistence.QueryManager; | ||
|
||
import java.time.Instant; | ||
import java.util.Date; | ||
|
||
import static org.dependencytrack.event.kafka.componentmeta.RepoMetaConstants.TIME_SPAN; | ||
|
||
public class SupportedMetaHandler extends AbstractMetaHandler { | ||
|
||
public SupportedMetaHandler(ComponentProjection componentProjection, QueryManager queryManager, KafkaEventDispatcher kafkaEventDispatcher, boolean fetchLatestVersion) { | ||
this.componentProjection = componentProjection; | ||
this.kafkaEventDispatcher = kafkaEventDispatcher; | ||
this.queryManager = queryManager; | ||
this.fetchLatestVersion = fetchLatestVersion; | ||
} | ||
|
||
@Override | ||
public void handle() { | ||
KafkaEventDispatcher kafkaEventDispatcher = new KafkaEventDispatcher(); | ||
try (QueryManager queryManager = new QueryManager()) { | ||
IntegrityMetaComponent integrityMetaComponent = queryManager.getIntegrityMetaComponent(componentProjection.purl()); | ||
if (integrityMetaComponent != null) { | ||
if (integrityMetaComponent.getStatus() == null || (integrityMetaComponent.getStatus() == FetchStatus.IN_PROGRESS && Date.from(Instant.now()).getTime() - integrityMetaComponent.getLastFetch().getTime() > TIME_SPAN)) { | ||
integrityMetaComponent.setLastFetch(Date.from(Instant.now())); | ||
queryManager.updateIntegrityMetaComponent(integrityMetaComponent); | ||
kafkaEventDispatcher.dispatchAsync(new ComponentRepositoryMetaAnalysisEvent(componentProjection.purlCoordinates(), componentProjection.internal(), true, fetchLatestVersion)); | ||
} else { | ||
kafkaEventDispatcher.dispatchAsync(new ComponentRepositoryMetaAnalysisEvent(componentProjection.purlCoordinates(), componentProjection.internal(), false, fetchLatestVersion)); | ||
} | ||
} else { | ||
queryManager.createIntegrityMetaComponent(createIntegrityMetaComponent(componentProjection.purl())); | ||
kafkaEventDispatcher.dispatchAsync(new ComponentRepositoryMetaAnalysisEvent(componentProjection.purlCoordinates(), componentProjection.internal(), true, fetchLatestVersion)); | ||
} | ||
} | ||
} | ||
|
||
} |
21 changes: 21 additions & 0 deletions
21
src/main/java/org/dependencytrack/event/kafka/componentmeta/UnSupportedMetaHandler.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
package org.dependencytrack.event.kafka.componentmeta; | ||
|
||
import org.dependencytrack.event.ComponentRepositoryMetaAnalysisEvent; | ||
import org.dependencytrack.event.kafka.KafkaEventDispatcher; | ||
import org.dependencytrack.persistence.QueryManager; | ||
|
||
public class UnSupportedMetaHandler extends AbstractMetaHandler { | ||
|
||
public UnSupportedMetaHandler(ComponentProjection componentProjection, QueryManager queryManager, KafkaEventDispatcher kafkaEventDispatcher,boolean fetchLatestVersion) { | ||
this.componentProjection = componentProjection; | ||
this.kafkaEventDispatcher = kafkaEventDispatcher; | ||
this.queryManager = queryManager; | ||
this.fetchLatestVersion = fetchLatestVersion; | ||
} | ||
|
||
@Override | ||
public void handle() { | ||
KafkaEventDispatcher kafkaEventDispatcher = new KafkaEventDispatcher(); | ||
kafkaEventDispatcher.dispatchAsync(new ComponentRepositoryMetaAnalysisEvent(componentProjection.purlCoordinates(), componentProjection.internal(), false, fetchLatestVersion)); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,13 @@ | ||
package org.dependencytrack.model; | ||
|
||
public enum FetchStatus { | ||
//request processed successfully | ||
PROCESSED, | ||
TIMED_OUT | ||
|
||
//fetching information for this component is in progress | ||
IN_PROGRESS, | ||
|
||
//to be used when information is not available in source of truth so we don't go fetching this repo information again | ||
//after first attempt | ||
NOT_AVAILABLE | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.