Composite data masking utility
Implement instruments, describe practices, contracts to solve sensitive data masking problem in JS/TS. For secure logging, for public data output, for internal mimt-proxies (kuber sensitive-data-policy) and so on.
🚧 Work in progress / MVP#0 is available for testing
- Implement masking composer/processor
- Introduce (declarative?) masking directives: schema
- Describe masking strategies and add masking utils
- Support logging tools integration
- Both sync and async API
- Declarative configuration
- Deep customization
- TS and Flow typings
With npm:
npm install --save @qiwi/maskeror yarn:
yarn add @qiwi/maskerimport {masker} from '@qiwi/masker'
// Suitable for most std cases: strings, objects, json strings, which may contain any standard secret keys/values or card PANs.
masker('411111111111111') // Promise<4111 **** **** 1111>
masker.sync('4111111111111111') // 4111 **** **** 1111import {masker, registry} from '@qiwi/masker'
masker.sync({
secret: 'foo',
nested: {
pans: [4111111111111111]
},
foo: 'str with printed password=foo and smth else',
json: 'str with json inside {"secret":"bar"} {"4111111111111111":"bar"}',
}, {
registry, // plugin storage
pipeline: [
'split', // to recursively process object's children. The origin `pipeline` will be applied to internal keys and values
'pan', // to mask card PANs
'secret-key', // to conceal sensitive fields like `secret` or `token` (pattern is configurable)
'secret-value', // to replace sensitive parts of strings like `token=foobar` (pattern is configurable)
'json', // to find jsons in strings
]
})
// result:
{
secret: '***', // secret-key
nested: { // split
pans: [ // split
'4111 **** **** 1111' // pan
],
},
foo: 'str with printed *** and smth else', // secret-value
// json
// chunk#1: split, secret-key
// chunk#2: split, pan (applied to key!)
json: 'str with json inside {"secret":"***"} {"4111 **** **** 1111":"bar"}'
}Declare masker directives over json-schema. See @qiwi/masker-schema for details.
import {masker} from '@qiwi/masker';
masker.sync({
fo: 'fo',
foo: 'bar',
foofoo: 'barbar',
baz: 'qux',
arr: [4111111111111111, 1234123412341234]
}, {
pipeline: ['schema'],
schema: {
type: 'object',
properties: {
fo: {
type: 'string',
maskKey: ['plain']
},
foo: {
type: 'string',
maskKey: ['plain']
},
foofoo: {
type: 'string',
maskKey: ['strike'],
maskValue: ['plain']
},
arr: {
type: 'array',
items: {
type: 'number',
maskValue: ['pan']
}
}
}
}
})
// result:
{
baz: 'qux',
arr: [ '4111 **** **** 1111', '1234123412341234' ],
'***': 'fo',
'***(2)': 'bar',
'******': '***',
}npx masquer "4111 1111 1111 1111"
# returns 4111 **** **** 1111codesandbox.io/s/qiwi-masker-sandbox-ngrnu
Override global console methods to print sensitive data free output to stderr/stdout:
import {masker} from '@qiwi/masker'
['log', 'info', 'error'].forEach(method => {
const _method = console[method]
console[method] = (...args: any[]) => _method(...args.map(masker))
})Create a custom masker formatter, then attach it to your reporter / transport:
const winston = require('winston')
const {masker} = require('@qiwi/masker')
const logger = winston.createLogger({
levels: winston.config.syslog.levels,
transports: [
new winston.transports.Console({
format: winston.format.combine(
winston.format((info) => Object.assign(info, masker.sync(info)))(),
winston.format.json(),
),
})
]
})
logger.log({
level: 'info',
message: {foo: 'bar', secret: 'foobar', pan: [4111111111111111, 1234123412341234]},
})
// stdout
{"level":"info","message":{"foo":"bar","secret":"***","pan":["4111 **** **** 1111","1234123412341234"]}}stackoverflow.com/how-to-make-a-custom-json-formatter-for-winston3-logger
The masker bases on the middleware pattern: it takes a piece of data and pushes it forward the pipeline.
The output of each pipe is the input for the next one. Each pipe is a dual interface data processor:
export interface IMaskerPipe {
name: IMaskerPipeName
exec: IMaskerPipeAsync | IMaskerPipeDual
execSync: IMaskerPipeSync | IMaskerPipeDual,
opts?: IMaskerPipeOpts
}During the execution, every pipe handler takes full control of the context. It can override next steps, change the executor impl (replace, append hook, etc),
create internal masker threads, parallelize invocation queues and sync them back together, and so on.
Each pipe is fed with a normalized context which consists of:
export interface IMaskerPipeInput {
value: any // value to process
_value?: any // pipe result
id: IContextId // ctx unique key
context: IMaskerPipeInput // ctx self ref
parentId?: IContextId // parent ctx id
registry: IMaskerRegistry // pipe registry attached to ctx
execute: IExecutor // executor
sync: boolean // sync / async switch
mode: IExecutionMode // lagacy sync switch
opts: IMaskerOpts // current pipe options
pipe?: IMaskerPipeNormalized // current pipe ref
pipeline: IMaskerPipelineNormalized // actual pipeline
originPipeline: IMaskerPipelineNormalized // origin pipeline
[key: string]: any
}Both. In different situations, each approach has pros and cons. For this reason, the masker provides a choice:
masker(data) // async
masker.sync(data) // sync
masker(data, {sync: true}) // syncThere is also a bunch of plugins, that extend the available masking scenarios. Please follow their internal docs.
| Package | Description | Version |
|---|---|---|
| @qiwi/masker | Composite data masking utility with common pipeline preset |  |
| masquer | CLI for @qiwi/masker |  |
| @qiwi/masker-common | Masker common components: interfaces, executor, utils |  |
| @qiwi/masker-debug | Debug plugin to observe pipe effects |  |
| @qiwi/masker-infra | Infra package: build configs, tools, etc |  |
| @qiwi/masker-json | Plugin to search and parse JSONs chunks in strings |  |
| @qiwi/masker-limiter | Plugin to limit masking steps count and duration |  |
| @qiwi/masker-pan | Plugin to search and conceal PANs |  |
| @qiwi/masker-plain | Plugin to substitute any kind of data with *** |
 |
| @qiwi/masker-schema | Masker schema builder and executor |  |
| @qiwi/masker-secret-key | Plugin to hide sensitive data by key/path pattern match |  |
| @qiwi/masker-secret-value | Plugin to conceal substrings by pattern match |  |
| @qiwi/masker-split | Executor hook to recursively process any object inners |  |
| @qiwi/masker-strike | Plugin to |
 |
| @qiwi/masker-trycatch | Executor hook to capture and handle exceptions |  |
Feel free to open any issues: for bugs, feature requests or questions. You're always welcome to suggest a PR. Just fork this repo, write some code, add some tests and push your changes. Any feedback is appreciated.