[공도한] W2 미션 (W1은 closed 해버려서 이것 기준으로 확인해주시면 감사하겠습니다..)

.githooks/pre-commit

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# 스테이징된 Python 파일들만 가져오기
+files=$(git diff --cached --name-only --diff-filter=d | grep '\.py$')
+
+if [ -z "$files" ]; then
+    echo "No Python files to check"
+    exit 0
+fi
+
+echo "Starting code checks..."
+
+# 각 명령어의 실행 결과를 체크
+if ! poetry run isort $files; then
+    echo "isort failed"
+    exit 1
+fi
+
+if ! poetry run black $files; then
+    echo "black failed"
+    exit 1
+fi
+
+# 수정된 파일들 다시 스테이징
+git add $files
+echo "Code checks completed successfully"
+
+exit 0

.gitignore

@@ -1,160 +1,11 @@
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
-cover/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-.pybuilder/
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-#   For a library or package, you might want to ignore these files since the code is
-#   intended to run in multiple environments; otherwise, check them in:
-# .python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# poetry
-#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
-#   This is especially recommended for binary packages to ensure reproducibility, and is more
-#   commonly ignored for libraries.
-#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
-#poetry.lock
-
-# pdm
-#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
-#pdm.lock
-#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
-#   in version control.
-#   https://pdm.fming.dev/#use-with-ide
-.pdm.toml
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
-__pypackages__/
-
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env
+.git
 .venv
-env/
-venv/
-ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
-
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
-
-# Pyre type checker
-.pyre/
-
-# pytype static type analyzer
-.pytype/
-
-# Cython debug symbols
-cython_debug/
+__pycache__
+.ipynb_checkpoints
+
+.DS_Store
 
-# PyCharm
-#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
-#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
-#  and can be added to the global gitignore or merged into this file.  For a more nuclear
-#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
+temp
+private
+data
+log

.python-version

@@ -0,0 +1 @@
+3.12

README.md

@@ -0,0 +1,90 @@
+# Softeer Data Engineering WIKI
+
+## 개요
+
+Softeer Data Engineering 코스를 수강하며 수행한 과제와
+학습 내용을 정리한 저장소입니다.
+
+## 디렉터리 구조
+
+```
+softeer
+├─ .python-version
+├─ README.md
+├─ missions
+│  └─ W1
+│     ├─ M1
+│     │  ├─ column_info.md
+│     │  ├─ data_exploration.ipynb
+│     │  └─ mtcars.csv
+│     ├─ M2
+│     │  ├─ learning_sql.ipynb
+│     │  └─ test.db
+│     └─ M3
+│        ├─ data
+│        │  ├─ Countries_by_GDP.json
+│        │  ├─ Countries_by_GDP_etl_processed.json
+│        │  ├─ World_Economies.db
+│        │  ├─ api
+│        │  │  └─ extracted
+│        │  │     ├─ gdp_2019.parquet
+│        │  │     ├─ gdp_2020.parquet
+│        │  │     ├─ gdp_2021.parquet
+│        │  │     ├─ gdp_2022.parquet
+│        │  │     ├─ gdp_2023.parquet
+│        │  │     └─ gdp_2024.parquet
+│        │  └─ country_regions.json
+│        ├─ etl_project_gdp.py
+│        ├─ etl_project_gdp_using_api.py
+│        ├─ etl_project_gdp_visualization.ipynb
+│        ├─ etl_project_gdp_with_sql.py
+│        ├─ etl_project_gdp_with_sql_visualization.ipynb
+│        ├─ log
+│        │  ├─ etl_project_log.txt
+│        │  ├─ etl_project_log_etl_processed.txt
+│        │  └─ etl_project_log_using_api.txt
+│        ├─ processor
+│        │  ├─ __init__.py
+│        │  ├─ api
+│        │  │  ├─ __init__.py
+│        │  │  ├─ extractor.py
+│        │  │  └─ transformer.py
+│        │  ├─ extractor.py
+│        │  ├─ io_handler.py
+│        │  ├─ json_loader.py
+│        │  ├─ sqlite_loader.py
+│        │  └─ transformer.py
+│        └─ utils
+│           ├─ __init__.py
+│           └─ logging.py
+├─ poetry.lock
+├─ pyproject.toml
+├─ retrospect
+│  ├─ 2025-01-02.md
+│  ├─ 2025-01-03.md
+│  ├─ 2025-01-06.md
+│  ├─ 2025-01-08.md
+│  ├─ 2025-01-09.md
+│  └─ 2525-01-07.md
+├─ self_study
+│  ├─ README.md
+│  ├─ learning_pandas
+│  │  ├─ data.csv
+│  │  └─ pandas.ipynb
+│  └─ parallellism
+│     ├─ files
+│     │  ├─ pool_test.py
+│     │  └─ queue_test.py
+│     ├─ testing_does_pandas_use_multicore.ipynb
+│     └─ testing_multiprocessing.ipynb
+└─ slides
+   ├─ W1 Introduction to Data Engineering.pdf
+   ├─ W2 Introduction to Big Data.pdf
+   ├─ W3 Introduction to Apache Hadoop.pdf
+   ├─ W4 Introduction to Apache Spark.pdf
+   ├─ W5 How Spark Works Internally - RDD and DAG.pdf
+   ├─ W6 Optimizing Spark Job.pdf
+   ├─ W7 Monitoring and Optimizing Spark Job.pdf
+   └─ W8 Adaptive Query Execution in Spark.pdf
+
+```

missions/W1/Dockerfile

@@ -0,0 +1,16 @@
+FROM quay.io/jupyter/base-notebook:latest
+
+# 작업 디렉터리 설정
+WORKDIR /app
+
+# Python 패키지 목록 복사
+COPY requirements.txt .
+
+# Python 패키지 설치
+RUN pip install -r requirements.txt
+
+# 프로젝트 파일 복사
+COPY . .
+
+# juptyer notebook 실행
+CMD ["jupyter", "notebook", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root"]

missions/W1/M1/column_info.md

@@ -0,0 +1,19 @@
+# 데이터셋 이해
+
+* **mpg (연비)**: 1갤런의 연료로 주행할 수 있는 거리를 마일 단위로 나타낸 값. 높을수록 연비가 좋음.
+* **cyl (실린더 수)**: 엔진의 실린더 개수. 보통 4기통, 6기통, 8기통 등이 있음.
+  * 4 ~ 8 (4, 6, 8)
+* **disp (배기량)**: 모든 실린더의 총 부피를 입방 인치(cubic inch) 단위로 나타낸 값.
+* **hp (마력)**: 엔진이 낼 수 있는 최대 출력.
+* **drat (후륜 차축비)**: 엔진 동력을 바퀴로 전달할 때의 기어비.
+* **wt (중량)**: 차량의 무게를 1000파운드 단위로 나타낸 값.
+* **qsec (1/4마일 주행시간)**: 1/4마일(약 400m)을 주행하는데 걸리는 시간으로, 자동차의 가속 성능을 나타냄.
+* **vs (엔진 형태)**: V형 엔진인지 직렬 엔진인지를 나타냄.
+  * 0 or 1
+* **am (변속기 종류)**: 자동변속기(0)인지 수동변속기(1)인지를 나타냄.
+  * Transmission
+  * 0 or 1
+* **gear (전진 기어 수)**: 전진 기어의 단수.
+  * 3 ~ 5
+* **carb (기화기 수)**: 엔진에 장착된 기화기의 개수. 기화기는 연료와 공기를 혼합하는 장치.
+  * 1 ~ 8