Migrate from NVD Data Feeds to API 2.0

[knqyf263]

Description

On December 15th, 2023, the NVD plans to retire all legacy data feeds while guiding any remaining data feed users to updated application-programming interfaces (APIs). APIs have many benefits over data feeds and have been the proven and preferred approach to web-based automation for over a decade. For additional information on the NVD API, please visit the developers pages. Click here for more information on the NVD timeline.

https://nvd.nist.gov/vuln/data-feeds

Here are the details of the API.
https://nvd.nist.gov/developers/vulnerabilities

Discussed in #5652

[knqyf263]

I am considering the following tasks:

1. Generate Go structs from JSON schema:
   NVD publishes the JSON schema. We should generate Go structs from the schema. I've not seen any actively maintained tool written in Go for generating Go structs from a JSON schema. Although not written in Go, quicktype might be a good option. I haven’t tried it yet, so we need to see what kind of structs it generates.

2. Store the last updated date:
   This can either be saved in a file as done here, or it could be retrieved using the GitHub API. The last update date should be taken from timestamp included in the response.

timestamp identifies when the response was generated.

3. Retrieve all data since the last updated date:
   It seems appropriate to use the lastModStartDate parameter for this purpose as seen here. However, it is noted that the maximum allowable range for any date range parameters is 120 consecutive days.

The maximum allowable range when using any date range parameters is 120 consecutive days.

Therefore, if the range exceeds 120 days, it will be necessary to split the dates. This will always be the case initially, and also if there's a need to retrieve all dates due to some bug, so implementing date splitting is necessary.

4. Follow All pagination:
   Use resultsPerPage, startIndex, and totalResults to retrieve all results.

5. Retry:
   Multiple retries might be necessary due to the potential instability of the API.

6. Parallelization (Optional):
   As data retrieval after the initial request will likely involve smaller data sets, parallelization might not be necessary. However, if the API is exceedingly slow, it might be worth considering.

[oliverlockwood]

Regarding (5), it's worth noting that the experience in the Dependecy-Check project (see e.g. jeremylong/DependencyCheck#6107, jeremylong/DependencyCheck#6144) is that:

• HTTP 503 (Service Unavailable) is quite common
• HTTP 502 (Bad Gateway) and HTTP 504 (Gateway Timeout) also occur intermittently, but with sufficient frequency to cause problems.

There might be some merit in allowing timeouts (for example) to the NVD API to be configurable, as well.

[DmitriyLewen]

Hello @oliverlockwood

You are right.
Yesterday I saw a lot of 503 errors.

[jgamblin]

So that you know, they have decided to keep the JSON feeds for now.
https://groups.google.com/a/list.nist.gov/g/nvd-news/c/aofnAd3HP2g

[knqyf263]

We've moved to NVD API 2.0.