Skip to content

Commit

Permalink
Use a concrete struct while parsing JSON to reduce memory consumption. (
Browse files Browse the repository at this point in the history
#153)

* Use a concrete struct while parsing json to reduce memory consuption.

The npm package json can be very large. Using the
`map[string]interface{}` results in the entire json data structure being
deserialized. This costs a lot of memory and cpu time.

The feed parser only needs the "time" portion of the json data, so by
restricting the struct to that data we avoid a lot of overhead.

Testing shows memory peaks close to 70Mb with this fix, rather than the
512Mb+ previously.

* Values in the time map can be either string, or a struct, so use the
empty interface.
  • Loading branch information
calebbrown authored Nov 8, 2021
1 parent 4eb515f commit 2ce6d6d
Showing 1 changed file with 10 additions and 9 deletions.
19 changes: 10 additions & 9 deletions pkg/feeds/npm/npm.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,18 +90,19 @@ func fetchPackage(baseURL, pkgTitle string) ([]*Package, error) {
if err != nil {
return nil, err
}
var jsonMap map[string]interface{}
err = json.Unmarshal(body, &jsonMap)

// We only care about the `time` field as it contains all the versions in
// date order, from oldest to newest.
// Using a struct for parsing also avoids the cost of deserializing data
// that is ultimately unused.
var packageDetails struct {
Time map[string]interface{} `json:"time"`
}
err = json.Unmarshal(body, &packageDetails)
if err != nil {
return nil, fmt.Errorf("%w : %v for package %s", errJSON, err, pkgTitle)
}

// The json string `time` contains versions in date order, oldest to newest.
versions, ok := jsonMap["time"].(map[string]interface{})
if !ok {
return nil, fmt.Errorf("%w : 'time' not found for package %s ",
errJSON, pkgTitle)
}
versions := packageDetails.Time

// If `unpublished` exists in the version map then at a given point in time
// the package was 'entirely' removed, the packageEvent(s) received are for package
Expand Down

0 comments on commit 2ce6d6d

Please sign in to comment.