Job log encryption at rest

[arivictor]

Great application, we're currently trialling it for our team and looking to allow other internal teams to access it as a self-service for HTTP API scheduling.

One of our requirements before we can proceed is to protect the job logs, specifically the output that you can see once a job has completed (HTTP response for example). Our use case for this is that end-user events may return secrets in their response, and at the very least we'd like to ensure that they are not left exposed on our end when stored.

I am just wondering where the best point of entry would be to A) Encrypt the job response, an B) decrypt the job response?

I'm currently investigating lib/job.js specifically the functions that start and finish a job. Just before it gets written to data/jobs/*.json I want to encrypt it. Then whenever it gets called again (User view its via the UI) I'd like to decrypt it. If someone has done something similar I'm open to ideas. Just interested in any "gotchas" or things to take into consideration.

Thanks!

[jhuckaby]

The place you want to do this is in the storage layer, which is handled by pixl-server-storage. Cronicle uses this module for all its storage needs, including all job metadata and job logs. This is the perfect place to introduce an encryption layer. One of the available storage engines is Amazon S3, which has an encryption at rest option.

If you want to use the local filesystem for storage, then you will need to write your own storage Plugin. I would just copy the Filesystem.js file, add your encryption and decryption operations in the put(), get(), putStream() and getStream() methods, then configure Cronicle to point to your custom storage engine Plugin.

pixl-server-storage (and thus Cronicle) can be configured to use any storage Plugin anywhere on the filesystem. In fact, it is best to keep your custom storage Plugin script somewhere outside of the NPM node_modules tree, so it doesn't get clobbered on upgrades.

Example: /opt/myencryption/MyStoragePlugin.js

Then configure your Cronicle storage options like this:

"Storage": {
	"engine": "Filesystem",
	"engine_path": "/opt/myencryption/MyStoragePlugin.js",
	"list_page_size": 50,
	"concurrency": 4,
	"log_event_types": { "get": 1, "put": 1, "head": 1, "delete": 1, "expire_set": 1 },
	"Filesystem": {
		"base_dir": "data",
		"key_namespaces": 1
	}
},

Please don't change the __name property in your engine class, as that is used to locate the configuration.

Note that if you do this, you will have to start completely fresh with a new, untouched storage area on disk. Once you have your engine Plugin ready, go through the Cronicle first setup instructions to initialize the storage system, which writes a number of initial records.

I have never attempted anything like this before, so there may be unforeseen hurdles.

Good luck!

[arivictor]

Brilliant! Much appreciated. Regarding AWS, we actually opted to use EFS as a detached file storage which works perfect with the out of the box local file system storage configuration (+ Docker volumes). It has encryption at rest as well, but I believe the security team is asking we lock it down further. You've pointed us in the right direction, thanks again!