TerraformPipProject

Initiate the Terraform project and collecting provider resources

terraform init

Deploy the Terraform Environemnt with respective variables

terraform apply -var-file=secrets.tfvars

SSH the respective key:

ssh -i "tutorial_kp.pem" ubuntu@$(terraform output -raw web_public_dns)

Required

mkdir /home/ubuntu/etc
mkdir /home/ubuntu/etc/scripts
mkdir /home/ubuntu/etc/scripts/result

There is a bootstrap script to install all required packages:

./configuration/bootstrap.sh

Most important is to pip install all requirements

pip install -r /home/ubuntu/scipts/requirements.txt

Run data pipeline

cd /home/ubuntu/scipts/
python3 extract_nasa.api.py
python3 transform_nasa_data.py
python3 load_nasa_data.py

To install required packages to mysql your RDS

sudo apt-get update -y && sudo apt install mysql-client -y

mysql -h <dbhost> -P <dbport> -u <dbuser> -p

To end you project:

Destroy your Terraform Environment

terraform destroy

.env file

API_key
db_user
db_password
db_host
db_port
db_name

Project Day 1-2:

Project Day 3: