Build back‐end server monitoring and logging environments

This Document is translate from 백엔드 서버 모니터링 및 로깅

Backend Server Environment

The backend server structure routes requests through Nginx, using ports 80/443, to a Spring Boot application running on port 8080 within an EC2 instance. To monitor logs and various metrics in this environment, we decided to set up a monitoring and logging system.

Monitoring & Logging System Requirements

1. Monitor the following items:
   • JVM memory usage
   • CPU usage
   • Connection and thread pool exhaustion
   • Error-level log spikes, as a sudden increase suggests potential issues
   • API call frequency per unit time to understand popular functionalities
   • API response time to determine which APIs would benefit most from optimization
2. Log data should be viewable on a visual dashboard.
3. The monitoring server must be separated from the Spring server.
4. Monitoring and logging information should not be accessible from outside.

Steps to Meet the Requirements

Monitoring Metrics

All the metrics we want to monitor originate from Spring Boot, so the application must provide this information. To achieve this, we used Spring Boot Actuator. The following dependencies were added to Gradle, referencing the official documentation.

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'io.micrometer:micrometer-registry-prometheus'
}

Additionally, the following settings were added to application.yml to enable the endpoints provided by Spring Boot Actuator.

```yaml
management:
  server:
    port: 8082
  endpoints:
    web:
      exposure:
        include: "prometheus,health"

To satisfy the requirement that monitoring information must be inaccessible from the outside, we set the metrics endpoint to port 8082.

Separation of Monitoring Server

To ensure reliable monitoring, we decided to separate the Spring Boot server from the monitoring server. This setup is shown in the diagram below.

Both servers are within the same subnet, with security group settings allowing unrestricted communication within the subnet. This enables Spring Boot Actuator to provide information on port 8082.

To periodically collect metrics provided by Spring Boot Actuator, we chose Prometheus due to its abundant resources and reliable integration with Spring Boot. For visualization, we used Grafana, which is well-supported by Prometheus and recommended in the Prometheus official documentation. The diagram below illustrates this setup.

To facilitate easy installation and management of Prometheus and Grafana, we used Docker and Docker Compose, allowing us to sequentially restart or expand them to other servers. We referenced the awesome-compose repository for Docker Compose configuration examples. Specific configurations are not disclosed due to security considerations.

Log Storage

At the time of building the monitoring and logging system, the logging policy was still incomplete due to ongoing feature development. Thus, the initial goal was to set up the logging system temporarily, allowing Spring Boot logs to be viewed in Grafana. To achieve this, we used Loki, a database specialized for log storage. By configuring Spring Boot’s Logback to log according to a specific configuration, we set up logs to be stored in Loki. Here is the configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/base.xml"/>
    <springProperty scope="context" name="appName" source="spring.application.name"/>

    <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <charset>UTF-8</charset>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss.SSS} | %t | %highlight(%-5p) | %cyan(%logger{36}) | %m%n
            </pattern>
        </encoder>
    </appender>

    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
        <http>
            <url>http://10.0.0.35:3100/loki/api/v1/push</url>
        </http>
        <format>
            <label>
                <pattern>app=${appName},host=${HOSTNAME},traceID=%X{traceId:-NONE},level=%level</pattern>
            </label>
            <message>
                <pattern>${FILE_LOG_PATTERN}</pattern>
            </message>
            <sortByTime>true</sortByTime>
        </format>
    </appender>

    <springProfile name="local">
        <root level="INFO">
            <appender-ref ref="Console"/>
        </root>

        <logger name="com.sb" level="DEBUG"/>
        <logger name="org.springframework.orm.jpa" level="DEBUG"/>
        <logger name="org.springframework.orm.transaction" level="DEBUG"/>
        <logger name="org.hibernate.orm.jdbc.bind" level="TRACE"/>
    </springProfile>

    <springProfile name="dev">
        <root level="INFO">
            <appender-ref ref="LOKI"/>
        </root>
    </springProfile>

</configuration>

After integrating Loki, the updated architecture is shown below.