Opencanary

[colin-stubbs]

• Bug
• Enhancement

Proposed commit message

Resolves issues #12911, #13024, #13025. Relevant to resolution of #2518.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Resolves multiple issues:

• Only set event.kind == alert if event is clearly not a generic application log message
• Migrate logfile to filestream based filebeat configuration
• Add http_endpoint for webhook based ingest
• Add system tests for both filestream and http_endpoint as none currently exist.
• Add pipeline test for webhook'ed events
• Retain password fields if desired, e.g. only remove password field if redaction requested. The current behaviour always removes passwords.
• Provide option to remove or retain ECS mapped fields, currently this option does not exist and ECS mapped fields are always removed.
• Add dashboard and dashboard screenshot. Currently not included in integration.
• Fix confused tftp/vnc field names. This is a typeo/bug due to lack of sufficient sample logs for pipeline or system testing.
• Add more complete/wider variety of sample logs for testing, e.g. TFTP & VNC events, NTP events, SNMP events etc
• Define appropriate fields based on more complete/wider variety of sample logs
• Fix as yet unknown logtype handling, e.g. current ingest pipeline script allows the logtype integer value to be left in a field defined as keyword leading to type conflicts and incomplete search results.
• Update known logtype map based on latest opencanary repo code. Current list is not up to date with opencanary code.
• Improves error handling for some error-like events that can be produced by OpenCanary, e.g. when LLMNR module fails
• Adds basic GeoIP enrichment using source.ip and destination.ip

Testing:

• elastic-package lint && check && build
• elastic-package test system --generate
• elastic-package test pipeline --generate
• elastic-package test
• Manual deploy on local elastic-package managed stack and ingest of logfile
• Manual deploy on remote Elastic Cloud stack and ingest of webhooks from real opencanary honeypots

How to test this PR locally

Install and operate OpenCanary, ideally via docker.
Use Elastic Agent to ingest OpenCanary log file OR webhooks.
Scan OpenCanary with nmap with scripting to trigger events, e.g. nmap -sC 127.0.0.1
Review

Related issues

• Closes [opencanary]: Sets all log message event.kind == alert leading to unwanted alert noise in Elastic Security #12911
• Closes [opencanary]: Add http_endpoint to support OpenCanary webhook output #13024
• Closes [opencanary]: Various bug fixes / enhancements #13025
• Relates Migrate uses of the logfile input to filestream #2518

Screenshots

New basic summary dashboard added,

[colin-stubbs]

@navnit-elastic - new PR opened.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[navnit-elastic]

This is a copy of PR #13026, which was reviewed but eventually closed due to inactivity.

[kcreddy]

/test

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 1d0c6a2

Failed CI Steps

• Check integrations opencanary

History

[efd6]

The failure reported by CI Error: can't install the package: could not zip-install package; API status code = 422; response body = {"statusCode":422,"error":"Unprocessable Entity","message":"Document \"opencanary-96cfb6c4-bc46-4fd4-9476-e8e9550442d9\" belongs to a more recent version of Kibana [10.2.0] when the last known version is [8.9.0]."} is repeatable locally when running on the stack version specified in the manifest (v8.13.0).

I have tried running on v8.15.0 and v8.18.0. Both of these succeed, so either the Event Summary dashboard needs to be made to conform to v8.13's expectation, or the kibana version needs to be bumped in the manifest.

[navnit-elastic]

Thanks @efd6 for reporting, @colin-stubbs since there are major changes into the integration, I think it would be better to bump Kibana version to ^v8.18.0 in the manifest. Could you please do that? Thank you!

[colin-stubbs]

8.17.0 makes more sense.

@efd6 - out of curiosity what's the situation that has you running tests on an 8.13.x stack? Did you explicitly look at the constraint and fire up an 8.13.x version to test with because it had 8.13.0?

UPDATE: I realised you mentioned the CI pipeline... so I suppose it's smart enough to look at the constraint and fire up tests on the version/s listed there?

I'm typically testing with the latest elastic-package release whatever container image versions it pulls, at the moment that's 8.17.3, as well as an EC stack running 8.18.x, hence I've missed any issue with the constraint. I've not come across this kind of issue before either so I hadn't even thought to update the constraint version.

It strikes me that perhaps elastic-package should be a little more version aware and, if not have something added to it to check kibana asset versioning against any kibana constraint, at least present a warning to people that testing is occurring with a version greater than the lowest advertised compatible version, and hence version drift is potentially significant and that they should double check the constraints and assets will still work with the advertised constraint version?

I've updated constraints now to 8.17.0 after explicitly re-testing using an 8.17.0 stack.

user@box opencanary % elastic-package test      
Run asset tests for the package
2025/05/27 22:04:50  INFO License text found in "/SRC/GitHub/routedlogic/integrations/LICENSE.txt" will be included in package
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                                           │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ opencanary │             │ asset     │ dashboard opencanary-96cfb6c4-bc46-4fd4-9476-e8e9550442d9 is loaded │ PASS   │        792ns │
│ opencanary │ events      │ asset     │ index_template logs-opencanary.events is loaded                     │ PASS   │        209ns │
│ opencanary │ events      │ asset     │ ingest_pipeline logs-opencanary.events-0.5.0 is loaded              │ PASS   │        292ns │
╰────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: opencanary - END   ---
Done
Run pipeline tests for the package
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬───────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                                     │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼───────────────────────────────────────────────┼────────┼──────────────┤
│ opencanary │ events      │ pipeline  │ (ingest pipeline warnings test-events.log)    │ PASS   │ 325.332333ms │
│ opencanary │ events      │ pipeline  │ (ingest pipeline warnings test-webhooks.json) │ PASS   │ 284.813959ms │
│ opencanary │ events      │ pipeline  │ test-events.log                               │ PASS   │  1.16716425s │
│ opencanary │ events      │ pipeline  │ test-webhooks.json                            │ PASS   │  45.164667ms │
╰────────────┴─────────────┴───────────┴───────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: opencanary - END   ---
Done
Run policy tests for the package
--- Test results for package: opencanary - START ---
No test results
--- Test results for package: opencanary - END   ---
Done
Run static tests for the package
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ opencanary │ events      │ static    │ Verify sample_event.json │ PASS   │  66.262833ms │
╰────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: opencanary - END   ---
Done
Run system tests for the package
2025/05/27 22:04:56  INFO License text found in "/SRC/GitHub/routedlogic/integrations/LICENSE.txt" will be included in package
2025/05/27 22:06:05  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/test-http_endpoint-1748347565803275000.log
2025/05/27 22:06:09  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/elastic-agent-1748347569388539000.log
2025/05/27 22:06:56  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/test-filestream-1748347616644031000.log
2025/05/27 22:06:59  INFO Write container logs to file: /SRC/GitHub/routedlogic/integrations/build/container-logs/elastic-agent-1748347619465036000.log
--- Test results for package: opencanary - START ---
╭────────────┬─────────────┬───────────┬───────────────┬────────┬───────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME     │ RESULT │  TIME ELAPSED │
├────────────┼─────────────┼───────────┼───────────────┼────────┼───────────────┤
│ opencanary │ events      │ system    │ filestream    │ PASS   │ 39.461239209s │
│ opencanary │ events      │ system    │ http_endpoint │ PASS   │ 56.612648666s │
╰────────────┴─────────────┴───────────┴───────────────┴────────┴───────────────╯
--- Test results for package: opencanary - END   ---
Done
user@box opencanary % docker ps | grep elastic
67dc3cc87104   docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.17.0   "/usr/bin/tini -- /u…"   4 minutes ago   Up 3 minutes (healthy)   127.0.0.1:1514->1514/udp, 127.0.0.1:8082->80/tcp                 elastic-package-stack-elastic-agent-1
076a599d54ce   docker.elastic.co/elastic-agent/elastic-agent-wolfi:8.17.0   "/usr/bin/tini -- /u…"   4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:8220->8220/tcp                                         elastic-package-stack-fleet-server-1
d45bb16814cf   docker.elastic.co/kibana/kibana:8.17.0                       "/bin/tini -- /usr/l…"   4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:5601->5601/tcp                                         elastic-package-stack-kibana-1
ae8fbe84011f   docker.elastic.co/elasticsearch/elasticsearch:8.17.0         "/bin/tini -- /usr/l…"   4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:9200->9200/tcp, 9300/tcp                               elastic-package-stack-elasticsearch-1
a08274c944ee   elastic-package-stack-package-registry                       "./package-registry"     4 minutes ago   Up 4 minutes (healthy)   127.0.0.1:8080->8080/tcp, 127.0.0.1:9000->9000/tcp               elastic-package-stack-package-registry-1
user@box opencanary % 

[efd6]

out of curiosity what's the situation that has you running tests on an 8.13.x stack? Did you explicitly look at the constraint and fire up an 8.13.x version to test with because it had 8.13.0?

CI runs the tests on the lowest claimed stack version specified in the kibana.version field in the manifest. So, yes, that's what I did.

Since it works for 8.15, I'd be happy for it to be marked at that version. This drops the least number of users.