Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(scanner): Investigate bugs on large scale scans #247

Closed
3 tasks done
lolaapenna opened this issue Sep 24, 2024 · 0 comments · Fixed by #252, #251 or #250
Closed
3 tasks done

fix(scanner): Investigate bugs on large scale scans #247

lolaapenna opened this issue Sep 24, 2024 · 0 comments · Fixed by #252, #251 or #250
Assignees

Comments

@lolaapenna
Copy link
Collaborator

lolaapenna commented Sep 24, 2024

Task Description
The first iteration of scanners for scanning for vulnerabilities on Kubernetes, Assets, and CVEs from NVD has been implemented. However, during scanning on a larger scale, we encountered a lot of errors.

Therefore we need to investigate those errors and fix them.

Acceptance Criteria:

  • Scanners run on large-scale environments without errors
  • Data from Scans is at least partially checked for correctness and completeness
  • Address Performance issues during scanning

Expected Test:

  • Where it does make sense, add regression tests. Use common sense
@drochow drochow mentioned this issue Sep 25, 2024
26 tasks
MR2011 added a commit that referenced this issue Sep 25, 2024
* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* First implementation

* Wip

* Fix tests

* Fixing tests part 2

* Implement tests for BuildIssueVariantMap

* Fix tests

* Wip

* chore(deps): bump github.com/prometheus/client_golang (#210)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.2 to 1.20.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.20.3/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.2...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang from 1.23.0 to 1.23.1 (#211)

Bumps golang from 1.23.0 to 1.23.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add tests for handling issue repositories with different priorities

* chore: removed changes unrelated to this feature

* fix: fixed imports

* fix: fixed imports

* Improve tests

* resolve issues

* feat(scanner/nvd): Adding possibility to fetch NVD from 2001

* feat: made time window configurable

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Victor Dorneanu <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Reimsbach <[email protected]>
dustindemmerle pushed a commit that referenced this issue Sep 26, 2024
* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* First implementation

* Wip

* Fix tests

* Fixing tests part 2

* Implement tests for BuildIssueVariantMap

* Fix tests

* Wip

* chore(deps): bump github.com/prometheus/client_golang (#210)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.2 to 1.20.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.20.3/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.2...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang from 1.23.0 to 1.23.1 (#211)

Bumps golang from 1.23.0 to 1.23.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add tests for handling issue repositories with different priorities

* chore: removed changes unrelated to this feature

* fix: fixed imports

* fix: fixed imports

* Improve tests

* resolve issues

* feat(scanner/nvd): Adding possibility to fetch NVD from 2001

* feat: made time window configurable

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Victor Dorneanu <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Reimsbach <[email protected]>
dustindemmerle pushed a commit that referenced this issue Oct 8, 2024
* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* First implementation

* Wip

* Fix tests

* Fixing tests part 2

* Implement tests for BuildIssueVariantMap

* Fix tests

* Wip

* chore(deps): bump github.com/prometheus/client_golang (#210)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.2 to 1.20.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.20.3/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.2...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang from 1.23.0 to 1.23.1 (#211)

Bumps golang from 1.23.0 to 1.23.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add tests for handling issue repositories with different priorities

* chore: removed changes unrelated to this feature

* fix: fixed imports

* fix: fixed imports

* Improve tests

* resolve issues

* feat(scanner/nvd): Adding possibility to fetch NVD from 2001

* feat: made time window configurable

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Victor Dorneanu <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Reimsbach <[email protected]>
MR2011 added a commit that referenced this issue Oct 8, 2024
)

* feat(comp.backend): Implement filter values for component instance

* feat(comp.backend): Implement filter values for component instance

* chore(contirbuting): Change guidelines for branch naming (#231)

* chore(deps): bump github.com/99designs/gqlgen from 0.17.51 to 0.17.54 (#233)

Bumps [github.com/99designs/gqlgen](https://github.com/99designs/gqlgen) from 0.17.51 to 0.17.54.
- [Release notes](https://github.com/99designs/gqlgen/releases)
- [Changelog](https://github.com/99designs/gqlgen/blob/master/CHANGELOG.md)
- [Commits](99designs/gqlgen@v0.17.51...v0.17.54)

---
updated-dependencies:
- dependency-name: github.com/99designs/gqlgen
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/prometheus/client_golang (#234)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.3 to 1.20.4.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.3...v1.20.4)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/docker/docker (#235)

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 27.2.1+incompatible to 27.3.1+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v27.2.1...v27.3.1)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(matches): Implement Service Issue-Repository Matches (#209)

* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* First implementation

* Wip

* Fix tests

* Fixing tests part 2

* chore: removed changes unrelated to this feature

* fix: fixed imports

* Solves #209 (comment)

* Make default issue priority configurable

* Change mocks

* Fix tests

* Automatic application of license header

* Get rid of whole config in the db

* Automatic application of license header

* resolve issues

* resolve issues

---------

Co-authored-by: David Rochow <[email protected]>
Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>

* feat(matches): Implement Issue Matching ON creation of new component instances (#168)

* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* Implement tests for BuildIssueVariantMap

* Fix tests

* Wip

* chore(deps): bump github.com/prometheus/client_golang (#210)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.2 to 1.20.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.20.3/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.2...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang from 1.23.0 to 1.23.1 (#211)

Bumps golang from 1.23.0 to 1.23.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add tests for handling issue repositories with different priorities

* fix: fixed imports

* Improve tests

* resolve issues

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: David Rochow <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Reimsbach <[email protected]>

* feat(nvd): allow fetching of issues from 2021 to now #247 (#250)

* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* First implementation

* Wip

* Fix tests

* Fixing tests part 2

* Implement tests for BuildIssueVariantMap

* Fix tests

* Wip

* chore(deps): bump github.com/prometheus/client_golang (#210)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.2 to 1.20.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.20.3/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.2...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang from 1.23.0 to 1.23.1 (#211)

Bumps golang from 1.23.0 to 1.23.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add tests for handling issue repositories with different priorities

* chore: removed changes unrelated to this feature

* fix: fixed imports

* fix: fixed imports

* Improve tests

* resolve issues

* feat(scanner/nvd): Adding possibility to fetch NVD from 2001

* feat: made time window configurable

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Victor Dorneanu <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Reimsbach <[email protected]>

* fix: adding fqdn and account to component name (#252)

* fix: adding fqdn and account to component name

* fix: removing import

* fix(k8s): fixed k8s scanner (#251)

* feat(scanner/k8s): fix context timeout config

* fix(scanner/k8s): fix logical error during ImageID parsing

* fix(scanner/k8s): fix processing

* Automatic application of license header

* fix(scanner/k8s): added componentName ot filter

* refactor(mariadb/issue): reverted debugging change

* fix: addressed comments

* feat: adding error generalization

* Automatic application of license header

* fix: added import

* fix: re-generated gql

---------

Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>

* fix(k8s): fixed k8s scanner (#251)

* feat(scanner/k8s): fix context timeout config

* fix(scanner/k8s): fix logical error during ImageID parsing

* fix(scanner/k8s): fix processing

* Automatic application of license header

* fix(scanner/k8s): added componentName ot filter

* refactor(mariadb/issue): reverted debugging change

* fix: addressed comments

* feat: adding error generalization

* Automatic application of license header

* fix: added import

* fix: re-generated gql

---------

Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>

* feat(comp.backend): Implement filter values for component instance

* fix(k8s): fixed k8s scanner (#251)

* feat(scanner/k8s): fix context timeout config

* fix(scanner/k8s): fix logical error during ImageID parsing

* fix(scanner/k8s): fix processing

* Automatic application of license header

* fix(scanner/k8s): added componentName ot filter

* refactor(mariadb/issue): reverted debugging change

* fix: addressed comments

* feat: adding error generalization

* Automatic application of license header

* fix: added import

* fix: re-generated gql

---------

Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>

* feat(comp.backend): Implement filter values for component instance

* fix(k8s): fixed k8s scanner (#251)

* feat(scanner/k8s): fix context timeout config

* fix(scanner/k8s): fix logical error during ImageID parsing

* fix(scanner/k8s): fix processing

* Automatic application of license header

* fix(scanner/k8s): added componentName ot filter

* refactor(mariadb/issue): reverted debugging change

* fix: addressed comments

* feat: adding error generalization

* Automatic application of license header

* fix: added import

* fix: re-generated gql

---------

Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>

* Automatic application of license header

* regenerate

* server.go update

* add E2E Component Instance Filter Test

* service_test.go fixing

* feat(matches): Implement Service Issue-Repository Matches (#209)

* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* First implementation

* Wip

* Fix tests

* Fixing tests part 2

* chore: removed changes unrelated to this feature

* fix: fixed imports

* Solves #209 (comment)

* Make default issue priority configurable

* Change mocks

* Fix tests

* Automatic application of license header

* Get rid of whole config in the db

* Automatic application of license header

* resolve issues

* resolve issues

---------

Co-authored-by: David Rochow <[email protected]>
Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>

* feat(matches): Implement Issue Matching ON creation of new component instances (#168)

* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* Implement tests for BuildIssueVariantMap

* Fix tests

* Wip

* chore(deps): bump github.com/prometheus/client_golang (#210)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.2 to 1.20.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.20.3/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.2...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang from 1.23.0 to 1.23.1 (#211)

Bumps golang from 1.23.0 to 1.23.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add tests for handling issue repositories with different priorities

* fix: fixed imports

* Improve tests

* resolve issues

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: David Rochow <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Reimsbach <[email protected]>

* feat(nvd): allow fetching of issues from 2021 to now #247 (#250)

* chore:  WIP for basic matching

* chore: updated eventHandler initialization in tests

* Change EventHandler signature and logic

* Fix tests

* Add more tests regarding concurrent processing

* chore: refactored function for lower coginitive complexity & fixed logic

* First implementation

* Wip

* Fix tests

* Fixing tests part 2

* Implement tests for BuildIssueVariantMap

* Fix tests

* Wip

* chore(deps): bump github.com/prometheus/client_golang (#210)

Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.20.2 to 1.20.3.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.20.3/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.20.2...v1.20.3)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump golang from 1.23.0 to 1.23.1 (#211)

Bumps golang from 1.23.0 to 1.23.1.

---
updated-dependencies:
- dependency-name: golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add tests for handling issue repositories with different priorities

* chore: removed changes unrelated to this feature

* fix: fixed imports

* fix: fixed imports

* Improve tests

* resolve issues

* feat(scanner/nvd): Adding possibility to fetch NVD from 2001

* feat: made time window configurable

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Victor Dorneanu <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Michael Reimsbach <[email protected]>

* fix: adding fqdn and account to component name (#252)

* fix: adding fqdn and account to component name

* fix: removing import

* fix(k8s): fixed k8s scanner (#251)

* feat(scanner/k8s): fix context timeout config

* fix(scanner/k8s): fix logical error during ImageID parsing

* fix(scanner/k8s): fix processing

* Automatic application of license header

* fix(scanner/k8s): added componentName ot filter

* refactor(mariadb/issue): reverted debugging change

* fix: addressed comments

* feat: adding error generalization

* Automatic application of license header

* fix: added import

* fix: re-generated gql

---------

Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>

* updated generated.go

* event_registry.go bug fixing

* change variable names

* Automatic application of license header

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Victor Dorneanu <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Rochow <[email protected]>
Co-authored-by: License Bot <[email protected]>
Co-authored-by: Michael Reimsbach <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment