Add possibility to build API and WEB servers along with regular ones

[ArseniiPetrovich]

Closes #133

DETAILS

• Most variables (except those related to DB) will no longer be deployed during the infrastructure deployment process as this takes time and duplicate efforts when deploying the software.
• Major Ansible role refactor for better understanding and hierarchy:
  • S3, S3_debug and S3_config were merged into single role with different tasks subsets inside.
  • DRY improvements - main_infra, destroy and attach_existing_rds were merged into single infrastructure role as they depend on the same subset of tasks.
  • main_software role were split into three roles - build, deploy and ssm_vars to keep each role nice and small.
    Thoughts

    I've decided to keep these three as a separate roles because of their low coupling and independence (tags: build, deploy, update_vars), but if there are any objections I can possibly merge them into single software role with different tasks as I made with main_infra role;
• Major refactor of the Terraform scripts;
  • Create a module that will be repeatedly used to deploy API, WEB and regular servers.
    Details

    I've used a workaround with creating the Ansible template on top of one of the TF files which is used to call newly created module because of current Terraform limitations, that will be addressed in the near future.
    API and WEB servers are served by the common loadbalancer. Due to known bug some of the API are still addressed to the WEB instance along with the write API requests.
• Bug fixing;
  • [Major] vpc_zone_identifier were set along with the availability_zones in TF ASG resource leads to plan inconsistency;
• New features
  • Users can independently select a number of "regulars_servers", "web_servers" and "api_servers" they want to deploy for each chain;
• Cosmetic README improvements;

STATUS

As for now PR is in MVP (minimal viable product) stage, which means it is still not tested well, but all the functionality are in place. Also it is NOT backward compatible with master because of major changes in TF and Ansible scripts. I'll work more to see if there are any workaround to make it compatible.

TODO

• Make PR backward compatible
• Test how this PR works when performing the parallel deployment
• Add information about new variables to the README.
• Improve README by splitting it on self-sustaining parts.
• Test attach_existing_rds script
• Add possibility to create a DB with read replica
• Identify the root of the issue with empty tuples while attempting to destroy TF infra.

[ArseniiPetrovich]

Update 1: Found bugs.

• Load balancer rules were not created properly.
• TF didn't output human-readable info.

[ArseniiPetrovich]

Update 2: As hashicorp/terraform#21096 and hashicorp/terraform#21333 mentions there is a bug in a Terraform 0.12 core which does not allow to destroy infrastructure. As such I propose the following:

• Create workaround which will grep particular words from error message and bypass the infrastructure destroy task if it match to the following catchphrase "The given key does not identify an element in this collection value."
• Notify user that he should check his infrastructure and ensure it was deleted.
• Watch the Un-destroyable partially applied infrastructure hashicorp/terraform#21096 and Impossible to destroy resources hashicorp/terraform#21333 issues and fix bug once the issued will be resolved.

[ArseniiPetrovich]

I have tested the attach_existing_rds.yml script and found out a limitation caused by the indexation that is currently used in the deployment scripts. Due to this limitation only writer instances will be imported. But this should be more than enough to bypass database backup&recover procedure when migrating the infrastructure. I'll think a bit about if there any possibility to workaround the limitation, but as for now I suggest we can simply:

• Add info about attach_existing_rds.yml script limitation to README

[ArseniiPetrovich]

@vbaranov @symanovich-a
Please, kindly check the pull request.

[ArseniiPetrovich]

As for now this could not be merged, at least until hashicorp/terraform-provider-aws#8268 is not merged.

[sergey989]

• check infrastructure deployment
  TASK [dynamodb : dynamodb_table] - during multi-chain deployment from scratch.

error "boto.exception.JSONResponseError: JSONResponseError: 400 Bad Request\n{'__type': 'com.amazon.coral.availability#ThrottlingException', 'message': 'The rate of control plane requests made by this account is too high'}"

• check software deployment

[ArseniiPetrovich]

Finally, the hashicorp/terraform-provider-aws#8268 was merged. We can merge this PR next week.

[ArseniiPetrovich]

@vbaranov However, I can see that the documentation migrated to docs.blockscout.org. How to resolve merge conflict?

[vbaranov]

@vbaranov However, I can see that the documentation migrated to docs.blockscout.org. How to resolve merge conflict?

@ArseniiPetrovich yes, indeed, we migrated documentation to the docs.blockscout.com. It is managed by GitBook and lives here https://github.com/blockscout/docs. In order to solve merging conflicts, I would suggest to remove updates of docs section in this PR and create a separate one to the docs repo.

[ArseniiPetrovich]

Terraform provider version released. Removing "do not merge" label.