Migrate from cloud.gov CDN service to new cloud.gov custom domain service

[lbeaufort]

Summary

What we're after:
Migrate from cloud.gov CDN service to new cloud.gov custom domain service

From the announcement:
https://cloud.gov/2021/08/16/external-domain-migration-announcement/

The custom-domain and cdn-route services leverage Let’s Encrypt to provision certificates on our users’ behalf. Several months ago, Let’s Encrypt announced that they’re deprecating their v1 API. To work with their new API, we’ve written a replacement to the cdn-route and custom-domain services, the external-domain service. We now need to migrate your service instances to the new service to ensure their certificates can continue renewing without issue. To accomplish this, we’ve written internal tooling to migrate instances automatically without service interruption, but it does require some action on your part to initiate the migration.

Completion criteria

• cdn-service has been replaced with external-domain service in all spaces

Tech steps or considerations

• Tech steps: https://cloud.gov/2021/08/16/external-domain-migration-announcement/#what-you-need-to-do
• Might need to make changes to fec-dns repo: https://github.com/fecgov/fec-dns/blob/master/terraform/www.gov.tf
• fec-dns repo has terraform files which generate DNS entries to give to Century Link/Lumen
• cloud.gov announced the deprecation in the newsletter on 9/8/21, which puts our deadline around February 5, 2022
• test in non-production environments first

Future work/related ticket

fecgov/fec-dns#1: Add README info or wiki with instructions on how to update DNS info

[lbeaufort]

Emailed cloud.gov to set up supplemental support hours. cc'd @fec-jli and @pkfec

[lbeaufort]

Notes on work so far: 🔒 https://docs.google.com/document/d/1oZIqrb6Ouy97n8Dct3IU179GVZG8BEOfdZVJp26lAEE/edit 🔒

Next steps:

• Reach out to AWS about modifying A record in Route 53 for apex domain (dev.fec.gov) - currently can't route to a non-AWS resource for the apex domain (dev.fec.gov)
• Cloud.gov is looking into whether we can route to an AWS resource (ARN) instead of the cloud.gov external domain (dev.fec.gov.external-domains-production.cloud.gov) cc: @ccostino

[lbeaufort]

@ccostino was able to confirm that cloud.gov does need to have the domain names match, so we need to have that external-domains-production.cloud.gov piece on the DNS record. They will need to take a closer look in order to figure out a path forward and solution. Pushing our internal deadline back a month, as cloud.gov hasn't set a hard deadline on moving over.

I'm meeting with AWS support today to see if we can work around the Route 53 A-record restriction.

[ccostino]

Thanks, @lbeaufort!

[lbeaufort]

After meeting with AWS, the way our DNS is set up makes it very difficult to make these changes properly. If we move the DNS records into our landing zone environment and change the way we structured the DNS records, we should be able to make these changes. It's my understanding dev.fec.gov shouldn't be an apex domain.

Next steps are to schedule a meeting with the operations team and AWS about next steps and whether we can move DNS migration into the project.

[lbeaufort]

From cloud.gov:

May 4, 2022, 13:45 PDT
Hi Laura,

We still need to update our documentation to match, but this is a change we're going to make - going forward, we'll support CNAME or ALIAS records pointing to the cloudfront distributions underlying domain-with-cdn service instances. We will still require the _acme-challenge.<domain> address to be a CNAME to _acme-challenge.<domain>.external-domains-production.cloud.gov, as that's required for us to provision certificates for you. The intermediate domain will also continue to be required when using the domain plan.

[lbeaufort]

From cloud.gov:

If so, then the next steps for you are (using dev.fec.gov as an example):

· Make sure you have created the CNAME record from _acme-challenge.dev.fec.gov to _acme-challenge.dev.fec.gov.external-domains-production.cloud.gov in accordance with the migration guide

· Use an alias record to point your record for dev.fec.gov at the domain for the Cloudfront distribution, which in that case is <see Laura's email>

After those steps are complete, we have an automated tool that should run to convert your cdn-broker cdn-route service to an external-domain-broker domain-with-cdn service.

Let's make sure that we test this approach on dev.fec.gov before moving on to any more mission-critical domains. So let me know when you have created the necessary DNS records and we can go from there.

[cnlucas]

Closing as we have successfully migrated all our spaces to the new service.