Apply OpenSearch granular access

[mdragon]

Acceptance Criteria

• Implement full IAM/OpenSearch integration so that Ingest and Query are exercising least privilege.

Context

We've hit a blocker in the OpenSearch rollout in AWS. The way user authorization works is tying OpenSearch accounts and AWS IAM accounts together and we need more time to work though that configuration that can only be done in the AWS environment. To get OpenSearch fully working to allow us to push data in and serve search queries, we have disabled OpenSearch’s internal access controls and we're connecting as the admin user.

We provide defense in depth for all of our existing production data stores. Specifically we provide: locked down access to our AWS account, restricting access to the databases to only our VPC, and restricting access to only the IAM role that is supposed to be providing access. At present, the OpenSearch cluster does not allow us to configure the 3rd layer, eg. restricting access to only the IAM role. That means that if someone breaks into either our AWS account or our VPC, then they would be able to easily acquire admin access to our OpenSearch cluster. This would allow someone to do all manner of things, like destroying all the data, exporting all of the data, or installing malicious software.
On the other hand, Simpler Grants.gov only stores already public data today. Beyond that, the API already allows access to everything in OpenSearch. So the risk here is primarily a reputation one, eg. HHS losing reputation because of not properly securing a database.

As such, we recommend that we proceed to Prod using the admin user to ingest and serve queries with the commitment that we will fully enact the more fine-grained access controls over the following 2 Sprints. This gives us the best balance of moving quickly and unblocking user testing and research while also being responsible stewards of the data and ensuring we continue to meet all proper security posture and controls.

[lucasmbrown-usds]

As discussed in Slack, please proceed with the recommended plan. I believe this ticket is ticket to resolve the issue, so let’s keep tracking this as a high-visibility ticket and flag its current status in meetings like Team of Teams (or in the updates on the Agile Deliverable statuses) until it’s resolved.