Attribute label changes may break imported searches

[pburkholder]

The migration from ELK/Logsearch to OpenSearch included a change in the backend log aggregator, and some field labels have changed. For example,logmessage.message_type is now @level. This has broken some imported searches and visualizations, as show below.

We will be updating this issue with identified differences. Many of these will not be fixed between the two systems, as they're inherent to the updated technology.

[markdboyd]

I don't think this is a case of a label change.

I think logmessage.message_type and logmessage.event_type are fields that existed in Kibana when logs were ingested from the Loggregator Firehose, but no longer exist when ingesting logs into Kibana from an aggregate drain.

The @level field exists in both systems, though interestingly, seems to have been set incorrectly for error logs in Kibana, while in OpenSearch @level: ERROR for logs with errors.

If anything, we may just want to note that the logmessage.* fields no longer exist

[markdboyd]

Yeah, a raw log from Kibana looks like:

<6> 2024-12-10T03:17:01Z c0ddec80-0c93-4566-9e6c-2e61bf65f5b6 doppler[29]: {"cf_app_id":"abc123","cf_app_name":"fake-app","cf_org_id":"abc123","cf_org_name":"org-
1","cf_origin":"firehose","cf_space_id":"abc123","cf_space_name":"space-1","deployment":"cf-
production","event_type":"LogMessage","ip":"<redacted>","job":"diego-
cell","job_index":"abc123","level":"info","message_type":"ERR","msg":"ERROR - {\"app_metadata\":{\"b3\":\"abc123-
abc123\",\"x_b3_traceid\":\"abc123\",\"x_b3_spanid\":\"abc123\"},\"app_message\":\"fake app 
error\"}","origin":"rep","source_instance":"3","source_type":"APP/PROC/WEB","time":"2024-12-
10T03:17:01Z","timestamp":1733800620813807013}

Notice the event_type and message_type fields.

And a raw log from OpenSearch looks like:

957 <11>1 2024-12-10T03:17:00.813807+00:00 org-1.fake-app.fake-app abc123 [APP/PROC/WEB/3] - [tags@47450 
app_id="abc123" app_name="fake-app" deployment="cf-production" index="abc123" instance_id="3" ip="<redacted>" 
job="diego-cell" organization_id="abc123" organization_name="org-1" origin="rep" process_id="abc123" 
process_instance_id="abc123" process_type="web" source_id="abc123" source_type="APP/PROC/WEB" 
space_id="abc123" space_name="space-1"] ERROR - {"app_metadata":{"b3":"abc123-
abc123","x_b3_traceid":"abc123","x_b3_spanid":"abc123"},"app_message":"fake app error"}

Notice that there is no event_type or message_type fields on the raw message.

Thus, the logmessage.event_type and logmessage.message_type fields simply do not exist in OpenSearch because the source data no longer exists on the raw messages that OpenSearch is ingesting.