Skip to content

Upload files to S3

Jump to bottom
E. Lynette Rayle edited this page Nov 18, 2021 · 10 revisions

back to Overview

Upload files to S3

Between executing aws-resources.yaml and aws-deploy.yaml, there are several files that need to be copied to the newly created S3 storage area.

Upload .env in S3

Copy example .env.example

Make a copy of the application's example .env file (LD4P/qa_server_container/.env.example) and rename it with a name ending in .env (e.g. .env, staging.env, or production.env).
info_24 Note that .env.example does not live in the deploy-templates directory. You will need to copy it from the LD4P/qa_server_container repo which defines the application Docker container.

Update environment variables

application variables

Edit the .env file to update the environment variables to match your setup. Many of the example values can be used as is. Values you will want to update include:

  • RAILS_ENV - set to the environment that the current deploy is creating (e.g. development, staging, production)
  • SECRET_KEY - update the secret key to any UUID that is a random base64 string. Rails provides tools to do this. If you have a Rails app, you can use rake secret from the command line at the root of any Rails app. If not, there are online generators for UUIDs. Search for uuid generator.
  • RELOAD_TOKEN - This allows you to start long running tasks from the web browser by including this token as part of the command. Update it to a unique alpha-numeric string.
authorities path variable

Make sure the AUTHORITIES_PATH variable has the following value. This is required for the application to find the authorities when deployed on AWS.

AUTHORITIES_PATH=authorities
database variables

You will also need to set database information. Docker will create the database users with the passwords assigned in the environment variables when it creates the database the first time.

  • MYSQL_DATABASE
  • MYSQL_USER
  • MYSQL_PASSWORD
  • MYSQL_ROOT_PASSWORD
additional changes

Follow other notes in env.example to determine if there are other changes you would like to make.

Put the .env file in S3 bucket

Copy the application .env file that you just created and edited in to the root level of the S3 bucket that was created by the aws-resources.yaml template. The name of the S3 bucket was assigned in the resource parameters file.

See How to copy files to S3 for more information on the copy process.

Upload authority files in S3

Authorities are select-in by placing the authority config files and associated files for the authorities you want to support under the /authorities/ directory in S3. Specific locations under /authorities/ are dependent on the type of authority.

By placing authority files under the /authorities/ directory in S3, the files will be copied to EFS through datasync.
warning_24 You must copy in at least one authority for the application to function.

See How to copy files to S3 for more information on the copy process.

linked data authorities

Reference: Using the Linked Data module to access authorities

The linked data authority module of questioning authority supports access to authorities that have a REST style API that returns search results as linked data.

Select the authorities you would like to include from LD4P/linked_data_authorities and copy them to S3.

Copy each AUTHORITY_NAME.json config file (e.g. agrovoc_direct.json) to:

/authorities/linked_data/

Copy each associated scenario file AUTHORITY_NAME_validation.yml (e.g. agrovoc_direct_validation.yml) to:

/authorities/linked_data/scenarios/

local authorities

Reference: Creating a custom controlled vocabulary via YAML

Local authorities are simple controlled vocabularies with terms defined in a .yml file. To add a local authority, copy the .yml file to:

/authorities/

updating supported authorities

To no longer support an authority, remove its associated files from the /authorities/ folder in the S3 bucket.

To add support for new authorities, add their associated files to the /authorities/ folder in the S3 bucket.

Create directory for database files

There needs to be a directory off the root named db-mysql-data. This will be used by the MariaDB database for the database files. To create the folder in your S3 bucket:

  • Open S3 Console
  • Select the appropriate bucket from the list of Buckets
  • Click Create folder to create a folder and name it db-mysql-data)

About EFS and DataSync

The final step is for the contents of the S3 bucket to get copied to the EFS drive. This task is handled by an AWS service called DataSync, which was configured for you when you ran the resource allocation template. The DataSync task will automatically upload the contents of the authorities folder into the EFS drive, where the application can read the authority files it contains, and it will keep uploading the contents of the S3 bucket once an hour.

If you want to add more authority files to the application, DataSync will copy them over once per hour; conversely, if you want to remove authority files from the application, simply remove them from the S3 bucket and DataSync will remove them from the application the next time it runs.

If you don't want to wait an hour, you can manually run the DataSync task from the web console. To do that, open up your AWS account's DataSync web console and you will see a list of Task IDs. One of those Task IDs should match the S3ToEfsTask output from the resource allocation template; click on it to open up the task. Next you should see an Overview pane showing the last time the task was run. Below that are several tabs; the Location tab should show a Source location pane specifying your S3 bucket, and a Destination location pane with the EFS drive. The "File share" listed for the EFS drive should match the VolumeId output from the resource allocation template. At the top of this page you should see a Start menu. Click on it and then select "Start with defaults" to run the task. This will immediately begin copying the contents of the S3 bucket to the EFS drive.

How to copy files to S3

You can copy files to S3 from the command line or through the S3 console. Instructions that work at the time of writing are described below for your convenience. Refer to the references for each option for up to date information.

UPLOAD OPTION: Command Line

Reference: S3 CLI Commands

Prerequisite: Install AWS CLI

The following is the basic form of the command for uploading a file to S3 from the command line. In a terminal window, execute a command like:

aws s3 cp FILE_TO_COPY s3://YOUR_BUCKET_NAME/DIRECTORY_PATH/

where,

  • FILE_TO_COPY should be replaced with the actual name of the .env file on you local file system with the name you assigned (e.g. staging.env, production.env)
  • YOUR_BUCKET_NAME should be replaced with the name you assigned for the S3 bucket in the resources parameters file (See Set resources parameters).
  • DIRECTORY_PATH is the path where the file should live in S3. Do not add a DIRECTORY_PATH if the file should live at root.
warning_24 The path needs to end with a / or the last part will be interpreted as the filename to use on S3.

Examples: Copy .env to root

aws s3 cp .env s3://lookup-bucket/

Verify file was copied

aws s3 ls s3://lookup-bucket/

Copy local authority to authorities

aws s3 cp property_type.yml s3://lookup-bucket/authorities/

Verify file was copied

aws s3 ls s3://lookup-bucket/authorities/

Copy linked data authority to authorities

aws s3 cp agrovoc_direct.json s3://lookup-bucket/authorities/linked_data/
aws s3 cp agrovoc_direct_validation.yml s3://lookup-bucket/authorities/linked_data/scenarios/

Verify files were copies

aws s3 ls s3://lookup-bucket/authorities/linked_data/
aws s3 ls s3://lookup-bucket/authorities/linked_data/scenarios/
info_24 When using the command line, S3 will create the path for you if it doesn't already exist.

UPLOAD OPTION: S3 Console

Reference: Uploading objects to S3

The following are the basic steps, at the time of writing, for uploading files through the S3 Console. Refer to the reference for up to date information.

  • Open S3 Console
  • Select the appropriate bucket from the list of Buckets
  • If needed, click Create folder to create a folder (e.g. /authorities)
  • If needed, click the folder name to enter the folder. Repeat creating folders until the required path is created. (e.g. /authorities/linked_data/scenarios)
  • Click the Upload button
  • Click the Add files button
  • Select your local copy
  • Click the Upload button to upload the file to your bucket

Previous | Next

Clone this wiki locally