Skip to content

[Apache_Tomcat] Ingest pipeline pattern enhancement #13896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
May 29, 2025
Merged

[Apache_Tomcat] Ingest pipeline pattern enhancement #13896

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f9e9f51
Add new field in ingest pipeline
Linu-Elias May 13, 2025
1ee9360
change.log
Linu-Elias May 14, 2025
66265ac
changelog
Linu-Elias May 19, 2025
2f86289
lint fix
Linu-Elias May 20, 2025
120a98a
build fix
Linu-Elias May 21, 2025
c17933e
build fix
Linu-Elias May 21, 2025
9724021
added units
Linu-Elias May 21, 2025
74369a1
build fix
Linu-Elias May 21, 2025
03f5794
doc fix
Linu-Elias May 23, 2025
a1ce6d5
resolved comments
Linu-Elias May 27, 2025
f9c9988
Merge branch 'main' into apache_add_pattern
Linu-Elias May 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions packages/apache_tomcat/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,14 +89,14 @@ Here are the steps to configure Log format in Apache Tomcat instance:
```
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern='%h %l %u %t "%r" %s %b %A %X %T "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i"' />
pattern='%h %l %u %t "%r" %s %b %A %X %F "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i"' />
```

3. The supported log formats are:
```
Common Log Format :- '%h %l %u %t "%r" %s %b'
Combined Log Format :- '%h %l %u %t "%r" %s %b "%{Referrer}i" "%{User-Agent}i"'
Combined Log Format + X-Forwarded-For header :- '%h %l %u %t "%r" %s %b %A %X %T "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i"'
Common Log Format :- '%h %l %u %t "%r" %s %b ms:%D'
Combined Log Format :- '%h %l %u %t "%r" %s %b ms:%D "%{Referrer}i" "%{User-Agent}i"'
Combined Log Format + X-Forwarded-For header :- '%h %l %u %t "%r" %s %b ms:%D %A %X %F "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i"'
```

4. Run the following commands to restart Apache Tomcat instance: -
Expand Down
5 changes: 5 additions & 0 deletions packages/apache_tomcat/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.10.0"
changes:
- description: Added support for %D attribute in access logs
type: enhancement
link: https://github.com/elastic/integrations/pull/13896
- version: "1.9.1"
changes:
- description: Added description to ssl nodes including links to documentation.
Expand Down
7 changes: 6 additions & 1 deletion packages/apache_tomcat/data_stream/access/_dev/test/pipeline/test-access.log
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,9 @@
81.2.69.144 - admin [02/Mar/2023:19:01:17 +0530] "GET /manager/status HTTP/1.1" 200 4654 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36
81.2.69.144 - admin [02/Mar/2023:19:02:25 +0530] "GET / HTTP/1.1" 200 11235
81.2.69.144 - - [24/Oct/2024:14:18:49 +1100] "-" 400 - 81.2.69.145 + 0.000 "-" "-" X-Forwarded-For="-"
10.10.10.10 - - [28/May/2024:17:20:05 +0200] "GET / " 200 17
10.10.10.10 - - [28/May/2024:17:20:05 +0200] "GET / " 200 17
81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] "POST /host-manager/images/asf-logo.svg HTTP/1.1" 200 20486 ms:71198 "http://localhost:8080/host-manager/html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" X-Forwarded-For=""
81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] "POST /host-manager/images/asf-logo.svg HTTP/1.1" 200 20486 ms:54321 X 400 "http://localhost:8080/host-manager/html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" X-Forwarded-For="127.0.0.1"
81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] "POST /host-manager/images/asf-logo.svg HTTP/1.1" 200 20486 ms:3214 81.2.69.145 40 "http://localhost:8080/host-manager/html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" X-Forwarded-For="127.0.0.1, 127.0.0.3"
81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] "POST /host-manager/images/asf-logo.svg HTTP/1.1" 200 20486 ms:98765 50 "http://localhost:8080/host-manager/html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" X-Forwarded-For=""
10.10.10.10 - - [28/May/2024:17:20:05 +0200] "GET / " 200 17 ms:1234
344 changes: 344 additions & 0 deletions packages/apache_tomcat/data_stream/access/_dev/test/pipeline/test-access.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -746,6 +746,350 @@
"tags": [
"preserve_original_event"
]
},
{
"@timestamp": "2023-03-02T13:28:17.000Z",
"apache_tomcat": {
"access": {
"http": {
"ident": "-",
"useragent": "admin"
},
"request_process_time": 71198.0
}
},
"destination": {
"bytes": 20486
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"web"
],
"kind": "event",
"module": "apache_tomcat",
"original": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 ms:71198 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"",
"outcome": "success",
"type": [
"access"
]
},
"http": {
"request": {
"method": "POST",
"referrer": "http://localhost:8080/host-manager/html"
},
"response": {
"status_code": 200
},
"version": "1.1"
},
"related": {
"ip": [
"81.2.69.144"
]
},
"source": {
"ip": "81.2.69.144"
},
"tags": [
"preserve_original_event"
],
"url": {
"extension": "svg",
"original": "/host-manager/images/asf-logo.svg",
"path": "/host-manager/images/asf-logo.svg"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Chrome",
"original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
"os": {
"full": "Windows 10",
"name": "Windows",
"version": "10"
},
"version": "109.0.0.0"
}
},
{
"@timestamp": "2023-03-02T13:28:17.000Z",
"apache_tomcat": {
"access": {
"connection_status": "X",
"http": {
"ident": "-",
"useragent": "admin"
},
"request_process_time": 54321.0,
"response_time": 400.0
}
},
"client": {
"ip": [
"127.0.0.1"
]
},
"destination": {
"bytes": 20486
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"web"
],
"kind": "event",
"module": "apache_tomcat",
"original": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 ms:54321 X 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1\"",
"outcome": "success",
"type": [
"access"
]
},
"http": {
"request": {
"method": "POST",
"referrer": "http://localhost:8080/host-manager/html"
},
"response": {
"status_code": 200
},
"version": "1.1"
},
"related": {
"ip": [
"81.2.69.144",
"127.0.0.1"
]
},
"source": {
"ip": "81.2.69.144"
},
"tags": [
"preserve_original_event"
],
"url": {
"extension": "svg",
"original": "/host-manager/images/asf-logo.svg",
"path": "/host-manager/images/asf-logo.svg"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Chrome",
"original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
"os": {
"full": "Windows 10",
"name": "Windows",
"version": "10"
},
"version": "109.0.0.0"
}
},
{
"@timestamp": "2023-03-02T13:28:17.000Z",
"apache_tomcat": {
"access": {
"http": {
"ident": "-",
"useragent": "admin"
},
"ip": {
"local": "81.2.69.145"
},
"request_process_time": 3214.0,
"response_time": 40.0
}
},
"client": {
"ip": [
"127.0.0.1",
"127.0.0.3"
]
},
"destination": {
"bytes": 20486
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"web"
],
"kind": "event",
"module": "apache_tomcat",
"original": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 ms:3214 81.2.69.145 40 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.3\"",
"outcome": "success",
"type": [
"access"
]
},
"http": {
"request": {
"method": "POST",
"referrer": "http://localhost:8080/host-manager/html"
},
"response": {
"status_code": 200
},
"version": "1.1"
},
"related": {
"ip": [
"81.2.69.144",
"81.2.69.145",
"127.0.0.1",
"127.0.0.3"
]
},
"source": {
"ip": "81.2.69.144"
},
"tags": [
"preserve_original_event"
],
"url": {
"extension": "svg",
"original": "/host-manager/images/asf-logo.svg",
"path": "/host-manager/images/asf-logo.svg"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Chrome",
"original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
"os": {
"full": "Windows 10",
"name": "Windows",
"version": "10"
},
"version": "109.0.0.0"
}
},
{
"@timestamp": "2023-03-02T13:28:17.000Z",
"apache_tomcat": {
"access": {
"http": {
"ident": "-",
"useragent": "admin"
},
"request_process_time": 98765.0,
"response_time": 50.0
}
},
"destination": {
"bytes": 20486
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"web"
],
"kind": "event",
"module": "apache_tomcat",
"original": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 ms:98765 50 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\"",
"outcome": "success",
"type": [
"access"
]
},
"http": {
"request": {
"method": "POST",
"referrer": "http://localhost:8080/host-manager/html"
},
"response": {
"status_code": 200
},
"version": "1.1"
},
"related": {
"ip": [
"81.2.69.144"
]
},
"source": {
"ip": "81.2.69.144"
},
"tags": [
"preserve_original_event"
],
"url": {
"extension": "svg",
"original": "/host-manager/images/asf-logo.svg",
"path": "/host-manager/images/asf-logo.svg"
},
"user_agent": {
"device": {
"name": "Other"
},
"name": "Chrome",
"original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
"os": {
"full": "Windows 10",
"name": "Windows",
"version": "10"
},
"version": "109.0.0.0"
}
},
{
"@timestamp": "2024-05-28T15:20:05.000Z",
"apache_tomcat": {
"access": {
"http": {
"ident": "-",
"useragent": "-"
},
"request_process_time": 1234.0
}
},
"destination": {
"bytes": 17
},
"ecs": {
"version": "8.11.0"
},
"event": {
"category": [
"web"
],
"kind": "event",
"module": "apache_tomcat",
"original": "10.10.10.10 - - [28/May/2024:17:20:05 +0200] \"GET / \" 200 17 ms:1234",
"outcome": "success",
"type": [
"access"
]
},
"http": {
"response": {
"status_code": 200
}
},
"related": {
"ip": [
"10.10.10.10"
]
},
"source": {
"ip": "10.10.10.10"
},
"tags": [
"preserve_original_event"
]
}
]
}
Loading