-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
무중단 배포 스크립크 작성 (issue #456) #480
Open
robinjoon
wants to merge
43
commits into
dev
Choose a base branch
from
feat/#456
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
d66d971
feat: HealthController 추가
robinjoon 92d4886
feat: compose.yml 무중단 배포 환경 변화에 따라 수정
robinjoon ba943ac
feat: Rolling 방식 무중단 배포 스크립트 작성
robinjoon 179aa1e
fix: 슬랙 메세지 임시로 비활성화
robinjoon fa7405f
fix: 워크플로우 이름 변경
robinjoon c686913
fix: 클래스 이름 변경
robinjoon fd2bbfb
fix: compose 파일 변경
robinjoon 0eaf465
fix: healthcheck api 응답 변경
robinjoon 34bae07
fix: healthcheck 로직 스크립트로 분리
robinjoon a03832a
fix: 작업 경로 지정
robinjoon a808943
fix: healthcheck 주기 변경
robinjoon 1ff5a6b
chore: 헬스 체크가 실패할 경우 이후 배포가 진행되지 않는지 확인
robinjoon 44e6cb0
feat: A 배포 실패 시 롤백하는지 확인
robinjoon 1a25e5d
chore: 트리거를 위한 공백 추가
robinjoon ab9ccb4
fix: ci cd 스크립트 오류 수정
robinjoon 0f9cafd
공백 제거
robinjoon 85af4c6
스크립트에 컨티뉴 온 에러 제거
robinjoon f5d360d
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon bdfe119
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon a5b32b5
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon d04f149
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon 63c37f6
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon 2e7807d
fix: 롤백 시 이미지 이름 출력하도록 수정
robinjoon 005c840
fix: 롤백 시 이미지의 이전 버전을 제대로 불러오지 못하는 오류 수정
robinjoon dd474b9
chore: A 배포 후 헬스 체크 실패시 롤백 되는지 확인
robinjoon 4434851
fix: 스크립트 정리
robinjoon 0085c7c
정상적인 상황 배포 테스트
robinjoon 75966b1
스크립트 오타수정
robinjoon 4a34742
feat: 알림 보내는 스크립트 작성
robinjoon a9499b3
배포 성공 테스트
robinjoon 650dda8
A 롤백 성공 테스트
robinjoon 4fbb827
슬랙 알림 보내는 조건 수정
robinjoon 9817eb9
A 롤백 실패 테스트
robinjoon 513950d
A 롤백 실패 테스트
robinjoon a7e05a0
A 롤백 실패 테스트
robinjoon bd2f300
continue on error 제거
robinjoon 9c289a4
테스트
robinjoon 6d9e972
롤백 실패시 알림 테스트
robinjoon 834b1e0
배포 성공 시 알림 테스트
robinjoon 22ccc1d
롤백 성공 시 알림 테스트
robinjoon e6a2645
롤백 성공 시 알림 테스트 2
robinjoon 6f5babb
fix: 헬스 체크 api 정상 동작하도록 수정
robinjoon 146a299
fix: 타겟 브랜치 main으로 변경
robinjoon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
name: Backend CD | ||
name: Backend None Stop CD | ||
|
||
on: | ||
workflow_dispatch: | ||
|
@@ -9,6 +9,18 @@ on: | |
- backend/** | ||
|
||
jobs: | ||
findPreviousImageVersion: | ||
name: 🔎Find Previous Docker Image Version | ||
runs-on: [ self-hosted, devel-up-prod-a ] | ||
outputs: | ||
previousImageVersion: ${{steps.find_version.outputs.name}} | ||
steps: | ||
- id: find_version | ||
run: | | ||
PREVIOUS_IMAGE_NAME=$(docker ps --format "{{.Image}}") | ||
echo "name=$PREVIOUS_IMAGE_NAME" >> $GITHUB_OUTPUT | ||
echo $PREVIOUS_IMAGE_NAME | ||
|
||
build: | ||
name: 🏗️ Build Jar and Upload Docker Image | ||
runs-on: ubuntu-latest | ||
|
@@ -48,10 +60,10 @@ jobs: | |
tags: ${{ secrets.DOCKER_REPOSITORY_NAME }}:${{ github.sha }} | ||
platforms: linux/arm64 | ||
|
||
deploy: | ||
name: 🚀 Server Deployment | ||
deployToA: | ||
name: 🚀 Server A Deployment | ||
needs: build | ||
runs-on: [ self-hosted, develup ] | ||
runs-on: [ self-hosted, devel-up-prod-a ] | ||
defaults: | ||
run: | ||
working-directory: backend | ||
|
@@ -71,20 +83,100 @@ jobs: | |
- name: 🐳 Docker Compose up | ||
run: docker compose -f compose.yml up -d | ||
|
||
- name: 🐳 Clean Unused Image | ||
run: docker image prune -af | ||
healthCheckA: | ||
name: 🙏 Server A Health Check | ||
needs: deployToA | ||
defaults: | ||
run: | ||
working-directory: backend | ||
runs-on: [ self-hosted, devel-up-prod-a ] | ||
steps: | ||
- name: ♻️ Send Helth Check Request | ||
run: chmod u+x ./scripts/healthcheck.sh && ./scripts/healthcheck.sh | ||
|
||
rollBackA: | ||
name: 🚀 Server A RollBack | ||
needs: [healthCheckA, findPreviousImageVersion] | ||
if: failure() | ||
runs-on: [ self-hosted, devel-up-prod-a ] | ||
defaults: | ||
run: | ||
working-directory: backend | ||
|
||
env: | ||
BACKEND_APP_IMAGE_NAME: ${{ needs.findPreviousImageVersion.outputs.previousImageVersion }} | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
|
||
slack-notify_success: | ||
- name: 🐳 Login to Docker Hub | ||
uses: docker/login-action@v3 | ||
with: | ||
username: ${{ secrets.DOCKERHUB_USERNAME }} | ||
password: ${{ secrets.DOCKERHUB_TOKEN }} | ||
|
||
- name: 🐳 Docker Compose up | ||
run: docker compose -f compose.yml up -d | ||
|
||
healthCheckAAfterRollBack: | ||
name: 🙏 Server A Health Check After RollBack | ||
needs: rollBackA | ||
if : ${{always() && needs.rollBackA.result != 'skipped'}} | ||
defaults: | ||
run: | ||
working-directory: backend | ||
runs-on: [ self-hosted, devel-up-prod-a ] | ||
steps: | ||
- name: ♻️ Send Helth Check Request | ||
run: chmod u+x ./scripts/healthcheck.sh && ./scripts/healthcheck.sh | ||
|
||
deployToB: | ||
name: 🚀 Server B Deployment | ||
needs: healthCheckA | ||
if: ${{needs.healthCheckA.result == 'success'}} | ||
runs-on: [ self-hosted, devel-up-prod-b ] | ||
defaults: | ||
run: | ||
working-directory: backend | ||
|
||
env: | ||
BACKEND_APP_IMAGE_NAME: ${{ secrets.DOCKER_REPOSITORY_NAME }}:${{ github.sha }} | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
|
||
- name: 🐳 Login to Docker Hub | ||
uses: docker/login-action@v3 | ||
with: | ||
username: ${{ secrets.DOCKERHUB_USERNAME }} | ||
password: ${{ secrets.DOCKERHUB_TOKEN }} | ||
|
||
- name: 🐳 Docker Compose up | ||
run: docker compose -f compose.yml up -d | ||
|
||
healthCheckB: | ||
name: 🙏 Server B Health Check | ||
needs: deployToB | ||
defaults: | ||
run: | ||
working-directory: backend | ||
runs-on: [ self-hosted, devel-up-prod-b ] | ||
steps: | ||
- name: ♻️ Send Helth Check Request | ||
run: chmod u+x ./scripts/healthcheck.sh && ./scripts/healthcheck.sh | ||
|
||
deploySuccessNotifiy: | ||
name: 📢Send Deploy Success Notification | ||
runs-on: ubuntu-latest | ||
needs: | ||
- build | ||
- deploy | ||
if: success() | ||
- healthCheckA | ||
- healthCheckB | ||
if: ${{needs.healthCheckA.result == 'success' && needs.healthCheckB.result == 'success'}} | ||
steps: | ||
- name: Extract Commit Title | ||
run: | | ||
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1) | ||
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV | ||
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV | ||
|
||
- name: Build and Deploy Success | ||
uses: slackapi/[email protected] | ||
|
@@ -106,18 +198,19 @@ jobs: | |
env: | ||
SLACK_BOT_TOKEN: ${{ secrets.BOT_TOKEN }} | ||
|
||
slack-notify_build-fail: | ||
deployToBFailNotifiy: | ||
name: 📢Send Deploy To Server B Fail Notification | ||
runs-on: ubuntu-latest | ||
needs: | ||
- build | ||
if: failure() | ||
- healthCheckB | ||
if: ${{failure() && needs.healthCheckB.result == 'failure'}} | ||
steps: | ||
- name: Extract Commit Title | ||
run: | | ||
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1) | ||
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV | ||
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV | ||
|
||
- name: Build Fail | ||
- name: Build and Deploy Success | ||
uses: slackapi/[email protected] | ||
with: | ||
channel-id: ${{ secrets.ISSUE_CHANNEL }} | ||
|
@@ -129,26 +222,59 @@ jobs: | |
"type": "section", | ||
"text": { | ||
"type": "mrkdwn", | ||
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🔴 Build Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>" | ||
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀 Build Success \n\t • 🔴 Server B Deploy Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>" | ||
} | ||
} | ||
] | ||
} | ||
env: | ||
SLACK_BOT_TOKEN: ${{ secrets.BOT_TOKEN }} | ||
|
||
slack-notify_deploy-fail: | ||
rollBackSuccessNotifiy: | ||
name: 📢Send Server A RollBack Success Notification | ||
runs-on: ubuntu-latest | ||
needs: | ||
- deploy | ||
if: failure() | ||
- healthCheckAAfterRollBack | ||
if: ${{failure() && needs.healthCheckAAfterRollBack.result == 'success'}} | ||
steps: | ||
- name: Extract Commit Title | ||
run: | | ||
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1) | ||
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV | ||
|
||
- name: Build and Deploy Success | ||
uses: slackapi/[email protected] | ||
with: | ||
channel-id: ${{ secrets.ISSUE_CHANNEL }} | ||
payload: | | ||
{ | ||
"text": "Build and Deploy Status", | ||
"blocks": [ | ||
{ | ||
"type": "section", | ||
"text": { | ||
"type": "mrkdwn", | ||
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀 Build Success \n\t • 🟠 Server A RollBack Success \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>" | ||
} | ||
} | ||
] | ||
} | ||
env: | ||
SLACK_BOT_TOKEN: ${{ secrets.BOT_TOKEN }} | ||
|
||
rollBackFailNotifiy: | ||
name: 📢Send Server A RollBack Fail Notification | ||
runs-on: ubuntu-latest | ||
needs: | ||
- healthCheckAAfterRollBack | ||
if: ${{failure() && needs.healthCheckAAfterRollBack.result == 'failure'}} | ||
steps: | ||
- name: Extract Commit Title | ||
run: | | ||
COMMIT_TITLE=$(echo "${{ github.event.head_commit.message }}" | head -n 1) | ||
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV | ||
echo "COMMIT_TITLE=$COMMIT_TITLE" >> $GITHUB_ENV | ||
|
||
- name: Deploy Fail | ||
- name: Build and Deploy Success | ||
uses: slackapi/[email protected] | ||
with: | ||
channel-id: ${{ secrets.ISSUE_CHANNEL }} | ||
|
@@ -160,7 +286,7 @@ jobs: | |
"type": "section", | ||
"text": { | ||
"type": "mrkdwn", | ||
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀Build Success \n\t • 🔴Deploy Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>" | ||
"text": "<!channel> \n 📣 Server Build & Deploy 결과를 안내 드립니다. 📣 \n\t • 🚀 Build Success \n\t • 🔴Server A RollBack Fail \n\t • 🏷️ 관련 Commit: <${{ github.event.head_commit.url }}|${{ env.COMMIT_TITLE }}>" | ||
} | ||
} | ||
] | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,30 +1,11 @@ | ||
services: | ||
nginx: | ||
image: nginx | ||
depends_on: | ||
- application | ||
networks: | ||
- nginx-app-net | ||
ports: | ||
- "80:80" | ||
- "443:443" | ||
volumes: | ||
- /home/ubuntu/custom.conf:/etc/nginx/conf.d/default.conf | ||
- /etc/letsencrypt/live/api.devel-up.co.kr/fullchain.pem:/etc/letsencrypt/live/api.devel-up.co.kr/fullchain.pem | ||
- /etc/letsencrypt/live/api.devel-up.co.kr/privkey.pem:/etc/letsencrypt/live/api.devel-up.co.kr/privkey.pem | ||
|
||
application: | ||
image: ${BACKEND_APP_IMAGE_NAME} | ||
networks: | ||
- nginx-app-net | ||
ports: | ||
- "8080:8080" | ||
- "80:8080" | ||
- "8082:8082" | ||
environment: | ||
TZ: "Asia/Seoul" | ||
SPRING_PROFILE: prod | ||
SPRING_PROFILE: local | ||
restart: always | ||
container_name: develup-app | ||
|
||
networks: | ||
nginx-app-net: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
for i in {1..5}; do | ||
response=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/health) | ||
echo "Attempt $i: Response Code $response" | ||
|
||
if [ "$i" -eq 5 ] && [ "$response" -ne 200 ]; then | ||
echo "Health check failed after 10 attempts." | ||
exit 1 | ||
fi | ||
|
||
sleep 5 | ||
done | ||
echo "Health check passed." |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
package develup.api; | ||
|
||
import develup.api.common.ApiResponse; | ||
import org.springframework.http.ResponseEntity; | ||
import org.springframework.web.bind.annotation.GetMapping; | ||
import org.springframework.web.bind.annotation.RestController; | ||
|
||
@RestController | ||
public class HealthApi { | ||
|
||
@GetMapping("/health") | ||
public ResponseEntity<ApiResponse<String>> health() { | ||
return ResponseEntity.status(200).body(new ApiResponse<>("up")); | ||
} | ||
Comment on lines
+12
to
+14
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Approve] ResponseEntity.ok 레츠고? |
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
package develup.api; | ||
|
||
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; | ||
import static org.springframework.test.web.servlet.result.MockMvcResultHandlers.print; | ||
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; | ||
|
||
import org.junit.jupiter.api.DisplayName; | ||
import org.junit.jupiter.api.Test; | ||
|
||
class HealthApiTest extends ApiTestSupport { | ||
|
||
@Test | ||
@DisplayName("성공 응답을 반환한다") | ||
void health() throws Exception { | ||
mockMvc.perform(get("/health")) | ||
.andDo(print()) | ||
.andExpect(status().isOk()); | ||
} | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Commnet]
build job이 해당 작업 안기다려줘도 되는건지 궁금하네요. 👀
만약, build job -> deployToA job까지 수행 했어요.
그리고 healthCheckA 수행 이전에 컨테이너가 잠깐 실행되었고 실패했다고 했을때, docks ps를 찍으면 실패한 최신 도커 이미지 버전이 롤백 대상 버전이 될 가능성은 없나요??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
그런 시나리오가 원천적으로 불가능한 건 아니라고 생각해요. 그런데, 현실적으로 빌드 작업과 롤백 버전 찾는 작업이 동시에 트리거되면 그 작업의 부하 차이가 워낙 커서 롤백 버전 찾는 작업이 먼저 수행되게 되요. 위에 사진 보시면 속도 차이가 엄청 나요
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
굳굳 좋아요. 그럴 가능성이 적긴하네요. 👍
cd 과정에 평균 find previus docker image version 잡 수행 시간이 추가되는 것보다는 자동 롤백 실패 비용이 더 작겠네요.
graceful shutdown 논의는 슬랙으로 이어나가시고, 헬스 체크 api 선택적으로 반영해주시고 다시 요청 주시면 approve 처리할게요.