Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Production deploy: Fetch and display sign database from S3 location #1528

Merged
merged 20 commits into from
Jan 17, 2024
Merged
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
6a0dddd
Add dependency on aws-sdk-s3 to download the dictionary file from S3
joshmcarthur Jan 10, 2024
d203ee8
Add code to update dictionary from S3 from nzsl-share
joshmcarthur Jan 10, 2024
8a6a115
Copy env-example during CI run to load default config
joshmcarthur Jan 10, 2024
139a9c2
Use correct model name for Dictionary
joshmcarthur Jan 10, 2024
a8d4a45
Update the sign database each time Rails starts
joshmcarthur Jan 10, 2024
4e87a2b
Provide AWS_REGION for CI configuration
joshmcarthur Jan 10, 2024
2d629a3
Add aws-sdk-s3 to allow presigned URLs to be generated
joshmcarthur Jan 11, 2024
bae5dac
Import Signbank::AssetURL model to handle presigning sign assets wher…
joshmcarthur Jan 11, 2024
2bc14f8
Use Signbank::AssetURL for examples and sign video
joshmcarthur Jan 11, 2024
2a5910a
NZSL-164: Show the database version in the footer
joshmcarthur Jan 11, 2024
bfb223d
Update footer to add alt text to license badge
joshmcarthur Jan 14, 2024
acfcdb1
Merge pull request #1523 from ODNZSL/task/nzsl-159-update-nzsl-dictio…
joshmcarthur Jan 15, 2024
64b8f31
Apply patch to handle empty string column values for sign media
joshmcarthur Jan 15, 2024
b5f9c74
Merge pull request #1525 from ODNZSL/task/nzsl-164-show-database-version
joshmcarthur Jan 15, 2024
e3e1422
Correct string formatting violations with rubocop
joshmcarthur Jan 15, 2024
228d5f1
Merge pull request #1524 from ODNZSL/task/nzsl-167-presign-sign-asset…
joshmcarthur Jan 15, 2024
72a194e
Add some conditionals to handle prerelease signs missing videos
joshmcarthur Jan 15, 2024
a0c33ff
Merge pull request #1526 from ODNZSL/fix/handle-missing-video
joshmcarthur Jan 15, 2024
140685a
Revert "NZSL-167: Presign asset requests"
joshmcarthur Jan 16, 2024
186374d
Merge pull request #1527 from ODNZSL/revert-1524-task/nzsl-167-presig…
joshmcarthur Jan 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add code to update dictionary from S3 from nzsl-share
joshmcarthur committed Jan 10, 2024
commit d203ee82450b71e86c716b905a24b7956e7469a8
27 changes: 20 additions & 7 deletions config/initializers/sign_database.rb
Original file line number Diff line number Diff line change
@@ -1,8 +1,21 @@
# Update the dictionary file if it is older than 1 month
# We update this file in both dictionary modes because our tests
# expect the database to test across both modes
path = Rails.root.join('db', 'dictionary.sqlite3')
Rails.application.load_tasks
deployed = !Rails.env.development? && !Rails.env.test?
Rails.application.reloader.to_prepare do
# Update the dictionary file on boot
Rails.application.load_tasks
deployed = !Rails.env.development? && !Rails.env.test?

Rake::Task['dictionary:update'].execute if deployed || (!path.exist? || path.mtime <= 1.month.ago)
begin
Rake::Task["dictionary:update"].execute if deployed
rescue StandardError => e
warn e
end

##
# All other tables make heavy use of a 'word' column. Add an alias for it here so that
# we can use common queries and ordering.
# There's no ADD COLUMN IF NOT EXISTS, so we just handle the error
begin
DictionarySign.connection.execute("ALTER TABLE words ADD COLUMN word text AS (gloss)")
rescue ActiveRecord::StatementInvalid => e
raise e unless e.message == "SQLite3::SQLException: duplicate column name: word"
end
end
5 changes: 4 additions & 1 deletion env-example
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
S3_BUCKET_URL: "example.s3.url/"
NZSL_ONLINE_SECRET_KEY_BASE: 62da7bed624d0cbbe3d186166fdd88db5bb3989075a2154cebe3e5ee20a4f2a2d540865309958346b7b43799d461be2b37c6e27d1fd6ca03b1f59622c5ccc402
APP_DOMAIN_NAME: "localhost:3000"
APP_PROTOCOL: "http"
APP_PROTOCOL: "http"

# The latest public release
DICTIONARY_DATABASE_S3_LOCATION="s3://nzsl-signbank-media-production/dictionary-exports/nzsl.db"
54 changes: 36 additions & 18 deletions lib/tasks/dictionary.rake
Original file line number Diff line number Diff line change
@@ -1,29 +1,47 @@
namespace :dictionary do
namespace :dictionary do # rubocop:disable Metrics/BlockLength
desc 'Updates the NZSL dictionary packaged with the application to the latest release from Signbank'
task :update do # rubocop:disable Rails/RakeEnvironment - we need to place this file before the app can start
repo = 'odnzsl/nzsl-dictionary-scripts'
filename = 'nzsl.db'
content_type = 'application/vnd.sqlite3'
release_uri = URI::HTTPS.build(host: 'api.github.com', path: "/repos/#{repo}/releases/latest")
release = JSON.parse(release_uri.open.read)
database_asset = release['assets'].find do |asset|
asset['name'] == filename && asset['content_type'] == content_type
end

database_url = database_asset.fetch('browser_download_url')
database_s3_location = URI.parse(ENV.fetch('DICTIONARY_DATABASE_S3_LOCATION') || '')
raise 'DICTIONARY_DATABASE_S3_LOCATION must be an S3 URI' unless database_s3_location.scheme == 's3'

File.open('db/new-dictionary.sqlite3', 'wb') do |f|
f.write URI.parse(database_url).open.read
rescue OpenURI::HTTPError
sleep 5 # Wait a few seconds before retrying
retry
end
download_s3_uri(database_s3_location, 'db/new-dictionary.sqlite3')

database = SQLite3::Database.open('db/new-dictionary.sqlite3')
raise 'Database does not pass integrity check' unless database.integrity_check == [['ok']]

version = database.get_int_pragma('user_version')

FileUtils.mv('db/new-dictionary.sqlite3', 'db/dictionary.sqlite3')

puts "Updated db/dictionary.sqlite3 to #{release['name']}"
puts "Updated db/dictionary.sqlite3 to #{version}"
end

def s3_client
@s3_client ||= Aws::S3::Client.new({
region: ENV.fetch('DICTIONARY_AWS_REGION', ENV.fetch('AWS_REGION', nil)),
access_key_id: ENV.fetch('DICTIONARY_AWS_ACCESS_KEY_ID', nil),
secret_access_key: ENV.fetch('DICTIONARY_AWS_SECRET_ACCESS_KEY', nil)
}.compact)
end

def download_s3_uri(s3_uri, target)
bucket = s3_uri.host
key = s3_uri.path[1..]

begin
s3_client.get_object({ bucket:, key: }, target:)
rescue Aws::Errors::MissingCredentialsError,
Aws::Sigv4::Errors::MissingCredentialsError,
Aws::S3::Errors::ServiceError

# Fallback to public-URL download over HTTP if credentials are not provided or invalid.
# TODO use aws-sdk to leverage aws-client optimizations once unsigned requests are supported:
# https://github.com/aws/aws-sdk-ruby/issues/1149
public_url = URI.parse(Aws::S3::Bucket.new(bucket, credentials: 0).object(key).public_url)
Net::HTTP.start(public_url.host, public_url.port, use_ssl: true) do |http|
response = http.get(public_url.request_uri).tap(&:value)
File.binwrite(target, response.body)
end
end
end
end