refactor(nvd): migrate to API 2.0 #374

DmitriyLewen · 2023-12-05T09:47:24Z

Description

Migrate to NVD API 2.0

Related Issues

Migrate from NVD Data Feeds to API 2.0 trivy#5658

Related PRs

refactor(nvd): use API instead of JSON feeds vuln-list-update#258

DmitriyLewen · 2023-12-07T03:35:38Z

pkg/vulnsrc/nvd/nvd.go

+func getCvssV3(metricsV31, metricsV30 []CvssMetricV3) (score float64, vector string, severity types.Severity) {
+	// order: v3.1 metrics => v3.0 metrics
+	// save the first `primary` metric
+	// if the `Primary` metric does not exist => save the first `Secondary` metric (v3.1 => v3.0)
+	for _, metricV3 := range append(metricsV31, metricsV30...) {
+		// save first metric or the `Primary` metric if `Secondary` metric was saved previously
+		if score == 0 || metricV3.Type == primaryType {
+			score = metricV3.CvssData.BaseScore
+			vector = metricV3.CvssData.VectorString
+			severity, _ = types.NewSeverity(metricV3.CvssData.BaseSeverity)
+			if metricV3.Type == primaryType {
+				return
+			}
+		}
+	}
+	return
+}


I'm not sure about this case.
what to choose: “Secondary metric V31” or “Primary metric v30”.

As I commented above, we should take metrics only from NVD here. So, the answer is:

cvssMetricV31 with source: [email protected]

cvssMetricV30 with source: [email protected]

cvssMetricV31 from other vendors (ignore)

cvssMetricV30 from other vendors (ignore)

We may want to use metrics from other vendors in the future, but it should be done in another PR.

Thanks for your help!
Changed in 96b8f7d

namandf · 2023-12-13T11:52:30Z

Hi @DmitriyLewen , @knqyf263 , Hope you are doing great.

Just came across this PR.

Do we intend to bring in this change before 15th?
Will this fix require a trivy upgrade? I am assuming no, because its just a change in the way we gather data right?
In case , we don't get this change by 15th which is the deadline for deprecation and 18th December NVD will stop supporting older feeds if i am not wrong, then what will be impact on trivy scans? Will we continue seeing older vulnerabilities while the new ones or updates will be missing?

Thank you

DmitriyLewen · 2023-12-14T04:05:41Z

Hello @namandf

Do we intend to bring in this change before 15th?

We are trying to finish these changes before the 15th.

Will this fix require a trivy upgrade? I am assuming no, because its just a change in the way we gather data right?

right. This fix only for trivy-db. You will need only doewnload new DB.

In case , we don't get this change by 15th which is the deadline for deprecation and 18th December NVD will stop supporting older feeds if i am not wrong, then what will be impact on trivy scans? Will we continue seeing older vulnerabilities while the new ones or updates will be missing?

We only receive advisory information from nvd (severity, descriptions, etc.).
In this case, you will get all old and new CVEs, but there may be some typos/omissions in the vulnerability information.

namandf · 2023-12-14T04:30:15Z

Thank you for the update @DmitriyLewen .

We only receive advisory information from nvd (severity, descriptions, etc.). In this case, you will get all old and new CVEs, but there may be some typos/omissions in the vulnerability information.

Out of curiosity, do we rely on MITRE/cve.org for the CVE list? or are you suggesting that other databases bridge that gap?
MITRE also seems to have gone through a similar change. https://www.cve.org/Media/News/item/blog/2023/07/25/Legacy-Downloads-being-Phased-Out

DmitriyLewen · 2023-12-14T04:38:01Z

Out of curiosity, do we rely on MITRE/cve.org for the CVE list?

No, we don't use MITRE/cve.org.

We use the following databases to get CVE list:
https://aquasecurity.github.io/trivy/v0.48/docs/scanner/vulnerability/#data-sources
https://aquasecurity.github.io/trivy/v0.48/docs/scanner/vulnerability/#data-sources_1

namandf · 2023-12-15T02:34:17Z

Out of curiosity, do we rely on MITRE/cve.org for the CVE list?

No, we don't use MITRE/cve.org.

We use the following databases to get CVE list: https://aquasecurity.github.io/trivy/v0.48/docs/scanner/vulnerability/#data-sources https://aquasecurity.github.io/trivy/v0.48/docs/scanner/vulnerability/#data-sources_1

Got it. Thank you.

You might already be aware but looks like there is again a change in deadline.

DmitriyLewen · 2023-12-15T03:00:19Z

yes, thanks!

FYI - We have aquasecurity/trivy#5658

knqyf263 · 2023-12-17T20:24:49Z

pkg/vulnsrc/nvd/nvd.go

+func getCvssV2(metricsV2 []CvssMetricV2) (score float64, vector string, severity types.Severity) {
+	for _, metricV2 := range metricsV2 {
+		// save first metric or the `Primary` metric if `Secondary` metric was saved previously
+		if score == 0 || metricV2.Type == primaryType {


If I understand correctly, the type doesn't matter. We need metrics from NVD, so I think we should take metrics with source: [email protected].

source identifies the organization that provided the metrics information and type identifies whether the organization is a primary or secondary source. Primary sources include the NVD and CNA who have reached the provider level in CVMAP. 10% of provider level submissions are audited by the NVD. If a submission has been audited the NVD will appear as the primary source and the provider level CNA will appear as the secondary source.

"cvssMetricV30": [ { "source": "[email protected]", "type": "Secondary", "cvssData": { "version": "3.0", "vectorString": "CVSS:3.0/AV:P/AC:H/PR:N/UI:N/S:U/C:N/I:L/A:H", "attackVector": "PHYSICAL", "attackComplexity": "HIGH", "privilegesRequired": "NONE", "userInteraction": "NONE", "scope": "UNCHANGED", "confidentialityImpact": "NONE", "integrityImpact": "LOW", "availabilityImpact": "HIGH", "baseScore": 4.8, "baseSeverity": "MEDIUM" }, "exploitabilityScore": 0.5, "impactScore": 4.2 } ],

Thanks! fixed in 96b8f7d

knqyf263 · 2023-12-17T20:26:59Z

pkg/vulnsrc/nvd/nvd.go

+func getCvssV3(metricsV31, metricsV30 []CvssMetricV3) (score float64, vector string, severity types.Severity) {
+	// order: v3.1 metrics => v3.0 metrics
+	// save the first `primary` metric
+	// if the `Primary` metric does not exist => save the first `Secondary` metric (v3.1 => v3.0)
+	for _, metricV3 := range append(metricsV31, metricsV30...) {
+		// save first metric or the `Primary` metric if `Secondary` metric was saved previously
+		if score == 0 || metricV3.Type == primaryType {
+			score = metricV3.CvssData.BaseScore
+			vector = metricV3.CvssData.VectorString
+			severity, _ = types.NewSeverity(metricV3.CvssData.BaseSeverity)
+			if metricV3.Type == primaryType {
+				return
+			}
+		}
+	}
+	return
+}


As I commented above, we should take metrics only from NVD here. So, the answer is:

cvssMetricV31 with source: [email protected]

cvssMetricV30 with source: [email protected]

cvssMetricV31 from other vendors (ignore)

cvssMetricV30 from other vendors (ignore)

We may want to use metrics from other vendors in the future, but it should be done in another PR.

knqyf263

LGTM. I left a small comment.

knqyf263 · 2023-12-18T18:25:37Z

pkg/vulnsrc/nvd/types.go

+	ID                string          `json:"id"`
+	SourceIdentifier  string          `json:"sourceIdentifier,omitempty"`
+	Published         string          `json:"published"`
+	LastModified      string          `json:"lastModified"`
+	VulnStatus        string          `json:"vulnStatus,omitempty"`
+	EvaluatorComment  string          `json:"evaluatorComment,omitempty"`
+	EvaluatorSolution string          `json:"evaluatorSolution,omitempty"`
+	EvaluatorImpact   string          `json:"evaluatorImpact,omitempty"`
+	CisaExploitAdd    string          `json:"cisaExploitAdd,omitempty"`
+	CisaActionDue     string          `json:"cisaActionDue,omitempty"`
+	Descriptions      []LangString    `json:"descriptions"`
+	Metrics           Metrics         `json:"metrics,omitempty"`
+	Weaknesses        []Weakness      `json:"weaknesses,omitempty"`
+	Configurations    []Configuration `json:"configurations,omitempty"`
+	References        []Reference     `json:"references"`
+	VendorComments    []VendorComment `json:"vendorComments,omitempty"`


Can we remove unneeded fields? It helps us understand what fields we are using. Other structs as well.

Removed. Please let me know if I did something wrong.

Good point!
I checked your changes and don't see problem with that.

Signed-off-by: knqyf263 <[email protected]>

DmitriyLewen added 2 commits December 5, 2023 11:56

refactor: use api dir

b7e0472

refactor: update cve struct

181c2fc

DmitriyLewen self-assigned this Dec 5, 2023

fix linter errors

02490e9

DmitriyLewen commented Dec 7, 2023

View reviewed changes

knqyf263 reviewed Dec 17, 2023

View reviewed changes

refactor: use only NVD cvss metrics

96b8f7d

knqyf263 marked this pull request as ready for review December 18, 2023 18:24

knqyf263 reviewed Dec 18, 2023

View reviewed changes

refactor: remove unused fields

a4adfde

Signed-off-by: knqyf263 <[email protected]>

knqyf263 approved these changes Dec 18, 2023

View reviewed changes

knqyf263 merged commit 51be8d9 into aquasecurity:main Dec 18, 2023
2 checks passed

DmitriyLewen deleted the nvd-api branch December 19, 2023 04:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(nvd): migrate to API 2.0 #374

refactor(nvd): migrate to API 2.0 #374

DmitriyLewen commented Dec 5, 2023

DmitriyLewen Dec 7, 2023

knqyf263 Dec 17, 2023

DmitriyLewen Dec 18, 2023

namandf commented Dec 13, 2023 •

edited

Loading

DmitriyLewen commented Dec 14, 2023

namandf commented Dec 14, 2023 •

edited

Loading

DmitriyLewen commented Dec 14, 2023

namandf commented Dec 15, 2023

DmitriyLewen commented Dec 15, 2023

knqyf263 Dec 17, 2023

DmitriyLewen Dec 18, 2023

knqyf263 Dec 17, 2023

knqyf263 left a comment

knqyf263 Dec 18, 2023 •

edited

Loading

knqyf263 Dec 18, 2023

DmitriyLewen Dec 19, 2023

refactor(nvd): migrate to API 2.0 #374

refactor(nvd): migrate to API 2.0 #374

Conversation

DmitriyLewen commented Dec 5, 2023

Description

Related Issues

Related PRs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

namandf commented Dec 13, 2023 • edited Loading

DmitriyLewen commented Dec 14, 2023

namandf commented Dec 14, 2023 • edited Loading

DmitriyLewen commented Dec 14, 2023

namandf commented Dec 15, 2023

DmitriyLewen commented Dec 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 left a comment

Choose a reason for hiding this comment

knqyf263 Dec 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

namandf commented Dec 13, 2023 •

edited

Loading

namandf commented Dec 14, 2023 •

edited

Loading

knqyf263 Dec 18, 2023 •

edited

Loading