Local testing

High level steps to test the framework locally before using it AWS Lambda

If you are running on AMazon Linux 2023 you may run it bug, please use Amazon Linux 2

Browse to the Docker folder with all the required files. Build the Docker image locally by executing the Dockerfile locally.

Browse to the local folder and run the docker build along with the desired FRAMEWORK, HUDI, DELTA and ICEBERG

docker build --build-arg FRAMEWORK=DELTA -t sparkonlambda .

Once the image is build successfully, use the below docker run to provide the input variables. Note : Input_path and Target_path should be handled in the spark_script only the path to the script should be provided in the docker run

docker run -e SCRIPT_BUCKET=<s3-bucket> -e SPARK_SCRIPT=<s3-folder>/sample-spark-script-csv-to-<FRAMEWORK>-table.py -e AWS_REGION=us-east-1 -e AWS_ACCESS_KEY_ID=$(aws configure get default.aws_access_key_id) -e AWS_SECRET_ACCESS_KEY=$(aws configure get default.aws_secret_access_key) -e AWS_SESSION_TOKEN=$(aws configure get default.aws_session_token) -p 9000:8080 sparkonlambda

Now the Docker container is up running, trigger the Lambda_handler just like the AWS Lambda trigger

curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'