Remove duplicate logging

[matafc]

We have noticed duplicate logging to datadog for containers in ECS. Container logs are currently sent 2 ways.

1. The container instance datadog agent directly ingests the logs from docker on the ECS instance.
2. The ECS tasks are configured to send the logs to syslog on the instance. Rsyslog then sends the logs to datadog.

Non-rsylog logs
Rsyslog logs

Historically, it seems we sent to rsyslog so that datadog and logentries would both be able to process the logs. We are no longer using logentries, and the datadog direct ingested logs have a lot more metadata than the rsyslog forwarded one.

In this PR we:

• Configure the docker containers to log with the json-file driver, and limit the size of each log file to 1MB.
• We did not want to use the aws-logs driver, as we will be charged to store them in Cloudwatch, and we don't plan to use them there.
• We could not configure it with the none driver, as that stopped all logging, even from the datadog agent ingesting directly.
• Disable the functionality of the logentries plugin

Datadog configuration rollout

After this PR is merged, we do not need to update the datadog plugins districts to update the log functionality. The tasks will use the new configuration when they are deployed, as we are only changing how the plugins alter the task definition.

Logentries removal rollout

Logentries is already deactivated, but we still have the plugin installed on applications. This PR will remove the logentries configuration when an app is deployed. But we will still want to remove the plugin from all the districts using bcn plugin delete. Once it is removed from all districts, we can make a new PR to remove the code completely.

Note:

Due to the rsyslog parsing, using the service: tag facet was different for each container. But the datadog direct ingest maps service to the short image name, which containers share. For queries that need to filter on the specific container, we recommend using the container_name facet now. Here is an example of what to use in the future.

[essa]

LGTM 👍

Great investigation!
Thank you