Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

무중단 배포 스크립크 작성 (issue #456) #480

Open
wants to merge 43 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
d66d971
feat: HealthController 추가
robinjoon Sep 12, 2024
92d4886
feat: compose.yml 무중단 배포 환경 변화에 따라 수정
robinjoon Sep 12, 2024
ba943ac
feat: Rolling 방식 무중단 배포 스크립트 작성
robinjoon Sep 12, 2024
179aa1e
fix: 슬랙 메세지 임시로 비활성화
robinjoon Sep 12, 2024
fa7405f
fix: 워크플로우 이름 변경
robinjoon Sep 12, 2024
c686913
fix: 클래스 이름 변경
robinjoon Sep 12, 2024
fd2bbfb
fix: compose 파일 변경
robinjoon Sep 12, 2024
0eaf465
fix: healthcheck api 응답 변경
robinjoon Sep 12, 2024
34bae07
fix: healthcheck 로직 스크립트로 분리
robinjoon Sep 12, 2024
a03832a
fix: 작업 경로 지정
robinjoon Sep 12, 2024
a808943
fix: healthcheck 주기 변경
robinjoon Sep 12, 2024
1ff5a6b
chore: 헬스 체크가 실패할 경우 이후 배포가 진행되지 않는지 확인
robinjoon Sep 19, 2024
44e6cb0
feat: A 배포 실패 시 롤백하는지 확인
robinjoon Sep 19, 2024
1a25e5d
chore: 트리거를 위한 공백 추가
robinjoon Sep 19, 2024
ab9ccb4
fix: ci cd 스크립트 오류 수정
robinjoon Sep 19, 2024
0f9cafd
공백 제거
robinjoon Sep 19, 2024
85af4c6
스크립트에 컨티뉴 온 에러 제거
robinjoon Sep 19, 2024
f5d360d
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon Sep 19, 2024
bdfe119
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon Sep 19, 2024
a5b32b5
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon Sep 19, 2024
d04f149
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon Sep 19, 2024
63c37f6
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon Sep 19, 2024
2e7807d
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon Sep 19, 2024
005c840
fix: 롤백 시 이미지의 이전 버전을 제대로 불러오지 못하는 오류 수정
robinjoon Sep 19, 2024
dd474b9
chore: A 배포 후 헬스 체크 실패시 롤백 되는지 확인
robinjoon Sep 19, 2024
4434851
fix: 스크립트 정리
robinjoon Sep 19, 2024
0085c7c
정상적인 상황 배포 테스트
robinjoon Sep 19, 2024
75966b1
스크립트 오타수정
robinjoon Sep 19, 2024
4a34742
feat: 알림 보내는 스크립트 작성
robinjoon Sep 19, 2024
a9499b3
배포 성공 테스트
robinjoon Sep 19, 2024
650dda8
A 롤백 성공 테스트
robinjoon Sep 19, 2024
4fbb827
슬랙 알림 보내는 조건 수정
robinjoon Sep 19, 2024
9817eb9
A 롤백 실패 테스트
robinjoon Sep 19, 2024
513950d
A 롤백 실패 테스트
robinjoon Sep 19, 2024
a7e05a0
A 롤백 실패 테스트
robinjoon Sep 19, 2024
bd2f300
continue on error 제거
robinjoon Sep 19, 2024
9c289a4
테스트
robinjoon Sep 19, 2024
6d9e972
롤백 실패시 알림 테스트
robinjoon Sep 19, 2024
834b1e0
배포 성공 시 알림 테스트
robinjoon Sep 19, 2024
22ccc1d
롤백 성공 시 알림 테스트
robinjoon Sep 19, 2024
e6a2645
롤백 성공 시 알림 테스트 2
robinjoon Sep 19, 2024
6f5babb
fix: 헬스 체크 api 정상 동작하도록 수정
robinjoon Sep 19, 2024
146a299
fix: 타겟 브랜치 main으로 변경
robinjoon Sep 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 149 additions & 23 deletions .github/workflows/backend_cd.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Backend CD
name: Backend None Stop CD

on:
workflow_dispatch:
Expand All @@ -9,6 +9,18 @@ on:
- backend/**

jobs:
findPreviousImageVersion:
name: 🔎Find Previous Docker Image Version
runs-on: [ self-hosted, devel-up-prod-a ]
outputs:
previousImageVersion: ${{steps.find_version.outputs.name}}
steps:
- id: find_version
run: |
PREVIOUS_IMAGE_NAME=$(docker ps --format "{{.Image}}")
echo "name=$PREVIOUS_IMAGE_NAME" >> $GITHUB_OUTPUT
echo $PREVIOUS_IMAGE_NAME

Comment on lines +12 to +22
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Commnet]

build job이 해당 작업 안기다려줘도 되는건지 궁금하네요. 👀
만약, build job -> deployToA job까지 수행 했어요.
그리고 healthCheckA 수행 이전에 컨테이너가 잠깐 실행되었고 실패했다고 했을때, docks ps를 찍으면 실패한 최신 도커 이미지 버전이 롤백 대상 버전이 될 가능성은 없나요??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image
그런 시나리오가 원천적으로 불가능한 건 아니라고 생각해요. 그런데, 현실적으로 빌드 작업과 롤백 버전 찾는 작업이 동시에 트리거되면 그 작업의 부하 차이가 워낙 커서 롤백 버전 찾는 작업이 먼저 수행되게 되요. 위에 사진 보시면 속도 차이가 엄청 나요

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

굳굳 좋아요. 그럴 가능성이 적긴하네요. 👍
cd 과정에 평균 find previus docker image version 잡 수행 시간이 추가되는 것보다는 자동 롤백 실패 비용이 더 작겠네요.
graceful shutdown 논의는 슬랙으로 이어나가시고, 헬스 체크 api 선택적으로 반영해주시고 다시 요청 주시면 approve 처리할게요.

build:
name: 🏗️ Build Jar and Upload Docker Image
runs-on: ubuntu-latest
Expand Down Expand Up @@ -48,10 +60,10 @@ jobs:
tags: ${{ secrets.DOCKER_REPOSITORY_NAME }}:${{ github.sha }}
platforms: linux/arm64

deploy:
name: 🚀 Server Deployment
deployToA:
name: 🚀 Server A Deployment
needs: build
runs-on: [ self-hosted, develup ]
runs-on: [ self-hosted, devel-up-prod-a ]
defaults:
run:
working-directory: backend
Expand All @@ -71,20 +83,100 @@ jobs:
- name: 🐳 Docker Compose up
run: docker compose -f compose.yml up -d

- name: 🐳 Clean Unused Image
run: docker image prune -af
healthCheckA:
name: 🙏 Server A Health Check
needs: deployToA
defaults:
run:
working-directory: backend
runs-on: [ self-hosted, devel-up-prod-a ]
steps:
- name: ♻️ Send Helth Check Request
run: chmod u+x ./scripts/healthcheck.sh && ./scripts/healthcheck.sh

rollBackA:
name: 🚀 Server A RollBack
needs: [healthCheckA, findPreviousImageVersion]
if: failure()
runs-on: [ self-hosted, devel-up-prod-a ]
defaults:
run:
working-directory: backend

env:
BACKEND_APP_IMAGE_NAME: ${{ needs.findPreviousImageVersion.outputs.previousImageVersion }}

steps:
- uses: actions/checkout@v4

slack-notify_success:
- name: 🐳 Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: 🐳 Docker Compose up
run: docker compose -f compose.yml up -d

healthCheckAAfterRollBack:
name: 🙏 Server A Health Check After RollBack
needs: rollBackA
if : ${{always() && needs.rollBackA.result != 'skipped'}}
defaults:
run:
working-directory: backend
runs-on: [ self-hosted, devel-up-prod-a ]
steps:
- name: ♻️ Send Helth Check Request
run: chmod u+x ./scripts/healthcheck.sh && ./scripts/healthcheck.sh

deployToB:
name: 🚀 Server B Deployment
needs: healthCheckA
if: ${{needs.healthCheckA.result == 'success'}}
runs-on: [ self-hosted, devel-up-prod-b ]
defaults:
run:
working-directory: backend

env:
BACKEND_APP_IMAGE_NAME: ${{ secrets.DOCKER_REPOSITORY_NAME }}:${{ github.sha }}

steps:
- uses: actions/checkout@v4

- name: 🐳 Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: 🐳 Docker Compose up
run: docker compose -f compose.yml up -d

healthCheckB:
name: 🙏 Server B Health Check
needs: deployToB
defaults:
run:
working-directory: backend
runs-on: [ self-hosted, devel-up-prod-b ]
steps:
- name: ♻️ Send Helth Check Request
run: chmod u+x ./scripts/healthcheck.sh && ./scripts/healthcheck.sh

deploySuccessNotifiy:
name: 📢Send Deploy Success Notification
runs-on: ubuntu-latest
needs:
- build
- deploy
if: success()
- healthCheckA
- healthCheckB
if: ${{needs.healthCheckA.result == 'success' && needs.healthCheckB.result == 'success'}}
steps:
- name: Extract Commit Title
run: |
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1)
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV

- name: Build and Deploy Success
uses: slackapi/[email protected]
Expand All @@ -106,18 +198,19 @@ jobs:
env:
SLACK_BOT_TOKEN: ${{ secrets.BOT_TOKEN }}

slack-notify_build-fail:
deployToBFailNotifiy:
name: 📢Send Deploy To Server B Fail Notification
runs-on: ubuntu-latest
needs:
- build
if: failure()
- healthCheckB
if: ${{failure() && needs.healthCheckB.result == 'failure'}}
steps:
- name: Extract Commit Title
run: |
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1)
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV

- name: Build Fail
- name: Build and Deploy Success
uses: slackapi/[email protected]
with:
channel-id: ${{ secrets.ISSUE_CHANNEL }}
Expand All @@ -129,26 +222,59 @@ jobs:
"type": "section",
"text": {
"type": "mrkdwn",
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🔴 Build Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>"
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀 Build Success \n\t • 🔴 Server B Deploy Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>"
}
}
]
}
env:
SLACK_BOT_TOKEN: ${{ secrets.BOT_TOKEN }}

slack-notify_deploy-fail:
rollBackSuccessNotifiy:
name: 📢Send Server A RollBack Success Notification
runs-on: ubuntu-latest
needs:
- deploy
if: failure()
- healthCheckAAfterRollBack
if: ${{failure() && needs.healthCheckAAfterRollBack.result == 'success'}}
steps:
- name: Extract Commit Title
run: |
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1)
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV

- name: Build and Deploy Success
uses: slackapi/[email protected]
with:
channel-id: ${{ secrets.ISSUE_CHANNEL }}
payload: |
{
"text": "Build and Deploy Status",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀 Build Success \n\t • 🟠 Server A RollBack Success \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>"
}
}
]
}
env:
SLACK_BOT_TOKEN: ${{ secrets.BOT_TOKEN }}

rollBackFailNotifiy:
name: 📢Send Server A RollBack Fail Notification
runs-on: ubuntu-latest
needs:
- healthCheckAAfterRollBack
if: ${{failure() && needs.healthCheckAAfterRollBack.result == 'failure'}}
steps:
- name: Extract Commit Title
run: |
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1)
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV

- name: Deploy Fail
- name: Build and Deploy Success
uses: slackapi/[email protected]
with:
channel-id: ${{ secrets.ISSUE_CHANNEL }}
Expand All @@ -160,7 +286,7 @@ jobs:
"type": "section",
"text": {
"type": "mrkdwn",
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀Build Success \n\t • 🔴Deploy Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>"
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀 Build Success \n\t • 🔴Server A RollBack Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>"
}
}
]
Expand Down
23 changes: 2 additions & 21 deletions backend/compose.yml
Original file line number Diff line number Diff line change
@@ -1,30 +1,11 @@
services:
nginx:
image: nginx
depends_on:
- application
networks:
- nginx-app-net
ports:
- "80:80"
- "443:443"
volumes:
- /home/ubuntu/custom.conf:/etc/nginx/conf.d/default.conf
- /etc/letsencrypt/live/api.devel-up.co.kr/fullchain.pem:/etc/letsencrypt/live/api.devel-up.co.kr/fullchain.pem
- /etc/letsencrypt/live/api.devel-up.co.kr/privkey.pem:/etc/letsencrypt/live/api.devel-up.co.kr/privkey.pem

application:
image: ${BACKEND_APP_IMAGE_NAME}
networks:
- nginx-app-net
ports:
- "8080:8080"
- "80:8080"
- "8082:8082"
environment:
TZ: "Asia/Seoul"
SPRING_PROFILE: prod
SPRING_PROFILE: local
restart: always
container_name: develup-app

networks:
nginx-app-net:
12 changes: 12 additions & 0 deletions backend/scripts/healthcheck.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
for i in {1..5}; do
response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/health)
echo "Attempt $i: Response Code $response"

if [ "$i" -eq 5 ] && [ "$response" -ne 200 ]; then
echo "Health check failed after 10 attempts."
exit 1
fi

sleep 5
done
echo "Health check passed."
15 changes: 15 additions & 0 deletions backend/src/main/java/develup/api/HealthApi.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package develup.api;

import develup.api.common.ApiResponse;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class HealthApi {

@GetMapping("/health")
public ResponseEntity<ApiResponse<String>> health() {
return ResponseEntity.status(200).body(new ApiResponse<>("up"));
}
Comment on lines +12 to +14
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Approve]

ResponseEntity.ok 레츠고?

}
19 changes: 19 additions & 0 deletions backend/src/test/java/develup/api/HealthApiTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package develup.api;

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultHandlers.print;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;

class HealthApiTest extends ApiTestSupport {

@Test
@DisplayName("성공 응답을 반환한다")
void health() throws Exception {
mockMvc.perform(get("/health"))
.andDo(print())
.andExpect(status().isOk());
}
}
Loading