Enriching messages with local static data

[lawrencejob]

Hi all, please forgive me if the answer is obvious, but I have spent three days bashing my head into the docs trying to understand the correct way to do this (and read all the discussions) and have run out of ideas.

I'm trying to enrich messages with data stored in a JSON file (key->object reference data, basically). I can see that it's possible, if the JSON file were hosted somewhere, to use the http processor to get the data, but I'd rather not create an HTTP server just to host a file that doesn't change. I've also tried using broker but it seems way too complicated for what I'm trying to do.

Given that I have a medium-sized JSON file locally, what is the best/simplest way to enrich messages with data from that file?

--
(PS I am working on a POC to get my organisation to switch to Benthos for ETL projects because I think we could potentially decommission about 10 micro services and replace them with this amazing amazing tool. I am so impressed and excited by this tool.)

[lawrencejob]

I got it "working" in what feels like a horrible way:

map toc_color {
    let toc_colors = {
        "VT": "#004354",
        "CC": "#b7007c",
        "CS": "#1d2e35",
        "CH": "#00bfff",
        # ...
        "TP": "#09a4ec",
        "GM": "#fedc43",
        "AW": "#d02f27",
        "WM": "#ff8300",
        "LM": "#e07609"
    }
    root = $toc_colors.get(this)
}

...but I am surely missing something about the idiomatic way to do this. If needs be I'll just pretend this file doesn't exist and not look in it...

[mihaitodor]

Hey @lawrencejob 👋 Assuming that the "medium-sized JSON file" you have easily fits in memory in its entirety, then you can use a mapping processor where you have something like let toc_colors = file(path:"path_to_the_json_file", no_cache:false).parse_json(). If the file can change in time, you can set no_cache:true, but that will be slow since it will read the file each time this processor is invoked. There are workarounds if you need to cache it for a certain period of time, like using an in-memory cache with some ttl. There's also file_rel() if you wish to keep this mapping along with the json file in some separate folder.

[lawrencejob]

This is fantastic, thank you so much! I can't believe that in probably 15 hours of reading documentation, I didn't see the file() function - I thought I'd read every single line by that point. I feel very silly now. Thank you again.

In hindsight I am laughing at my past self for writing 'medium sized JSON.' It was 3am for me. The file is 34MB so comfortably fits inside memory.

If my company does adopt Benthos, I hope that we can contribute some plugins to the ecosystem, particularly around Salesforce's CDC system and Azure infrastructure.

[mihaitodor]

Glad I could help! Please feel free to reach out via the community channels if you need any help with the adoption. I'm in the process of adding CosmosDB support (see #2358 - if you have any feedback, please let me know). Any other PRs are welcome!