Delta.json and deltaLog.json metadata files to support 3rd party integrations #13

hkong-mitre · 2023-09-13T03:09:19Z

Update the old recent_activities.json file, which was used for debugging, to 2 more useful "delta" files:

delta.json specifies the changes to the repository in the current commit
deltaLog.json specifies the changes to the repository with a history that goes back 30 days (changeable in github-actions/update.yml).

Example delta.json:

{
  "fetchTime": "2023-08-23T18:47:51.227Z",
  "numberOfChanges": 2,
  "new": [
    {
      "cveId": "CVE-1970-0001",
      "cveOrgLink": "https://www.cve.org/CVERecord?id=CVE-1970-0001",
      "githubLink": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/1970/0xxx/CVE-1970-0001.json"
    }
  ],
  "updated": [
    {
      "cveId": "CVE-1970-0002",
      "cveOrgLink": "https://www.cve.org/CVERecord?id=CVE-1970-0002",
      "githubLink": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/1970/0xxx/CVE-1970-0002.json",
      "dateUpdated": "1970-01-01T01:02:00.000Z"
    }
  ],
  "unknown": []
}

Example of deltaLog.json:

[
  {
    "fetchTime": "2023-07-24T17:34:04.394Z",
    "durationInMsecs": 3044,
    "numberOfChanges": 2,
    "new": [],
    "updated": [
      {
        "cveId": "CVE-2023-3319",
        "cveOrgLink": "https://www.cve.org/CVERecord?id=CVE-2023-3319",
        "githubLink": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/2023/3xxx/CVE-2023-3319.json",
        "dateUpdated": "2023-07-24T17:31:04.394Z"
      },
      {
        "cveId": "CVE-2023-3321",
        "cveOrgLink": "https://www.cve.org/CVERecord?id=CVE-2023-3321",
        "githubLink": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/2023/3xxx/CVE-2023-3321.json",
        "dateUpdated": "2023-07-24T17:32:04.394Z"
      }
    ],
    "unknown": []
  },
  {
    "fetchTime": "2023-07-24T17:14:04.394Z",
    "durationInMsecs": 3244,
    "numberOfChanges": 2,
    "new": [
      {
        "cveId": "CVE-2023-3320",
        "cveOrgLink": "https://www.cve.org/CVERecord?id=CVE-2023-3320",
        "githubLink": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/2023/3xxx/CVE-2023-3320.json",
        "dateUpdated": "2023-07-24T17:08:04.394Z"
      }
    ],
    "updated": [
      {
        "cveId": "CVE-2023-3321",
        "cveOrgLink": "https://www.cve.org/CVERecord?id=CVE-2023-3321",
        "githubLink": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/2023/3xxx/CVE-2023-3321.json",
        "dateUpdated": "2023-07-24T17:12:04.394Z"
      }
    ],
    "unknown": []
  },
  {
    "fetchTime": "2023-07-24T17:02:00.394Z",
    "durationInMsecs": 2244,
    "numberOfChanges": 1,
    "new": [
      {
        "cveId": "CVE-2023-3321",
        "cveOrgLink": "https://www.cve.org/CVERecord?id=CVE-2023-3321",
        "githubLink": "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves/2023/3xxx/CVE-2023-3321.json",
        "dateUpdated": "2023-07-24T17:06:31.093Z"
      }
    ],
    "updated": [],
    "unknown": []
  }
]

Develop->main

… code repository

Delta.json and deltaLog.json metadata files to support 3rd party integrations

test/pretend_github_repository/1970/0xxx/CVE-1970-0002.json

test/pretend_github_repository/1970/0xxx/CVE-1970-0998.json

…pendabot/4

mprpic

Just some high-level comments/questions. LGTM!

src/core/Delta.ts

M-nj

Adding PR comments, some are for HK to review and update, some I will commit changes for myself.

src/common/IsoDateString.test.ts

src/core/CveCore.test.ts

src/core/CveCore.ts

src/core/CveCorePlus.ts

src/core/CveCorePlus.test.ts

src/core/DeltaLog.ts

src/core/git.test.ts

github-actions/cves.sh

chickpoint · 2023-09-15T19:45:57Z

Looks like a workable result to continue.
It is still in the same line as the recent_activities.json but simplified. Think that makes the file easier to handle and more readable for others that want to use it.

I really like the idea of having the log as well, as my implementation is based on Docker. So, it would be nice to have a small history to prepopulate the list.

The only preference I would have would be a relative path to the Json, to handle the lookups locally instead of pulling them from GitHub again. But that might just be a personal preference and not useful for everyone.

Anyhow I like the changes and thanks for the great work.

make CodeQL run on all branches

hkong-mitre · 2023-09-18T22:21:12Z

@chickpoint, that's an interesting point. I wanted to make the delta file available to apps (e.g., browser web apps) that don't have a local repository, but you can make use of the githubLink property by doing a string split at https://raw.githubusercontent.com/CVEProject/cvelistV5/main/ to get the local (git cloned)repository file path.

…ich runs codeQL.yml on all branches when pushed

- genericcommands.tests-> made tests more deterministic. - isodates -> define leap year behavior and added tests. - cverecord -> modify variable name.

…-bulk-download into feature/delta

M-nj

Planned future changes were linked in their respective issues. LGTM

M-nj · 2023-09-26T21:18:50Z

src/core/CveDate.test.ts

-    const yesterdate = parse(yesterday, 'yyyy-MM-dd', new Date());
-    expect(differenceInCalendarDays(now, yesterdate)).toBe(1);
-  });
+  // it(`getYesterday() should output yesterday's date as a string`, async () => {


Default Yesterday to ISO #16

M-nj · 2023-09-26T21:31:57Z

src/core/git.test.ts

+    // !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
+    // all tests above this line do not modify git status
+    // all tests below this line MUST undo modifications to git status so each test can assume a common starting point
+    // !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


For #16, can we move this batch into a new describe block and use beforeEach/afterEach?

src/core/git.test.ts

M-nj · 2023-09-26T21:36:12Z

cves.bat

@@ -0,0 +1,2 @@
+@echo off
+node ./dist/index.js "%1"


See tree/feature/cve-bat

athu-tran and others added 8 commits June 20, 2023 15:45

Merge pull request #8 from CVEProject/develop

7a417d4

Develop->main

add new test fixtures

fded704

clean up

f8a0357

add delta test fixture

c0d7612

move github actions from .github/workflow so they don't run in source…

28ccfaf

… code repository

version 1.0.2+delta_2023-09-12

2304564

Merge pull request #12 from hkong-mitre/feature/delta

2ead584

Delta.json and deltaLog.json metadata files to support 3rd party integrations

Update .env-EXAMPLE

140d3db

hkong-mitre commented Sep 13, 2023

View reviewed changes

test/pretend_github_repository/1970/0xxx/CVE-1970-0002.json Outdated Show resolved Hide resolved

hkong-mitre commented Sep 13, 2023

View reviewed changes

test/pretend_github_repository/1970/0xxx/CVE-1970-0998.json Outdated Show resolved Hide resolved

fixes https://github.com/CVEProject/cvelist-bulk-download/security/de…

bdd33ca

…pendabot/4

mprpic reviewed Sep 13, 2023

View reviewed changes

src/core/Delta.ts Outdated Show resolved Hide resolved

src/core/Delta.ts Show resolved Hide resolved

src/core/Delta.ts Outdated Show resolved Hide resolved

hkong-mitre mentioned this pull request Sep 13, 2023

recent_activities.json not being updated CVEProject/cvelistV5#23

Closed

hkong-mitre added 3 commits September 13, 2023 21:13

fix gitignore and minor test tweaks for deltalog and git

48c4071

remove durationInMsecs per #13 (comment)

ff818d1

make CodeQL run on all branches

35d3ff2

M-nj reviewed Sep 15, 2023

View reviewed changes

hkong-mitre added 2 commits September 18, 2023 16:41

Merge pull request #14 from CVEProject/hkong-mitre-patch-1

c560c2a

make CodeQL run on all branches

update simple-git library to latest version

3f7d310

hkong-mitre and others added 9 commits September 18, 2023 18:22

Merge branch 'main' into feature/delta to get the newer codeQL.yml wh…

6262788

…ich runs codeQL.yml on all branches when pushed

changes to PR as promised.

6dd1853

- genericcommands.tests-> made tests more deterministic. - isodates -> define leap year behavior and added tests. - cverecord -> modify variable name.

Merge branch 'feature/delta' of https://github.com/CVEProject/cvelist…

98c454e

…-bulk-download into feature/delta

update default constructor test to be more deterministic.

0f67232

add test for valid leap year date

5f67d9d

updated test for CveCore.fromRepositoryFilePath()

ef7e905

clean up whitespace

5a22064

change the delta.unknown list to delta.error

35bf539

update CveCore* with PR requests

160e9c4

hkong-mitre added 2 commits September 20, 2023 14:55

clean up obsolete code and add additional comments to clarify usage

665cd27

remove dotenv.config() everywhere except main.ts and CveId.ts

f740e15

hkong-mitre mentioned this pull request Sep 20, 2023

simplify the CveCore and CveRecord classes #15

Open

update version in prep for end-to-end testing

be001f2

M-nj approved these changes Sep 26, 2023

View reviewed changes

hkong-mitre merged commit 8f75b1b into develop Sep 26, 2023

This was referenced Sep 26, 2023

Update version number #17

Merged

Merge delta features into main branch #18

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delta.json and deltaLog.json metadata files to support 3rd party integrations #13

Delta.json and deltaLog.json metadata files to support 3rd party integrations #13

hkong-mitre commented Sep 13, 2023

mprpic left a comment

M-nj left a comment

chickpoint commented Sep 15, 2023

hkong-mitre commented Sep 18, 2023

M-nj left a comment

M-nj Sep 26, 2023 •

edited

Loading

M-nj Sep 26, 2023

M-nj Sep 26, 2023

Delta.json and deltaLog.json metadata files to support 3rd party integrations #13

Delta.json and deltaLog.json metadata files to support 3rd party integrations #13

Conversation

hkong-mitre commented Sep 13, 2023

mprpic left a comment

Choose a reason for hiding this comment

M-nj left a comment

Choose a reason for hiding this comment

chickpoint commented Sep 15, 2023

hkong-mitre commented Sep 18, 2023

M-nj left a comment

Choose a reason for hiding this comment

M-nj Sep 26, 2023 • edited Loading

Choose a reason for hiding this comment

M-nj Sep 26, 2023

Choose a reason for hiding this comment

M-nj Sep 26, 2023

Choose a reason for hiding this comment

M-nj Sep 26, 2023 •

edited

Loading