multiple environments + droneci #63

lrvick · 2017-05-20T21:58:06Z

Initial stab at droneci deployment for #! in a dedicated VPC.

Most apps should probably end up in a production k8s cluster or similar, but CI is a bit special in that it it will have god rights to manage other environments and peer to all environments/regions.

One of the features of DroneCI is the ability to manage terraform changes so long as droneCI itself is not managed by terraform because... yeah.

Ideally in the end the CI environment should end up the only one we have to maintain by hand, and very seldom so.

In the mean time however I am not really clear how to make the Makefile make sense with multiple environments.

None of this is deployed yet, and we also will need to rename terraform buckets/topics by hand to make this work.

Mostly looking for feedback atm. I did deploy a very similar setup on my personal AWS account and all seems to work as expected.

KellerFuchs

Cool stuff, and it goes way beyond what the title indicates.

Comments inline, but for me the main issues are:

Deploying again CoreOS, when we all agreed multiple time not to use this for the k8s cluster.
Baking KMS in several places (Terraform, Drone's config, ...); I would be happier with something vendor-neutral, rather than going with vendor-lockin in the initial setup.

KellerFuchs · 2017-05-22T00:15:48Z

terraform/environments/ci/files/decrypt.sh

@@ -0,0 +1,19 @@
+#!/bin/bash


#!/bin/sh set -e

Would that not list potentially logging plaintext secrets?

Oh -e != -x

KellerFuchs · 2017-05-22T00:16:31Z

terraform/environments/ci/files/decrypt.sh

+#!/bin/bash
+unset IFS
+out_path=${1:-/out/secrets.env}
+rm "$out_path" 2> /dev/null


rm -f (so that set -e doesn't make it explode when $out_path doesn't exist yet).

KellerFuchs · 2017-05-22T00:17:42Z

terraform/environments/ci/files/decrypt.sh

+for line in $(env | egrep '^KMS_'); do
+	kms_key="${line%%=*}"
+	key=${kms_key/KMS_/}
+	encrypted_value=${line#*=}


What does ${line%%=*} and ${line#*=} do? Is it POSIX?

It is bash variable substitution %%=* says "strip the equals sign and anything after it" and #*= says "remove the equals sign and anything before it". When using bash there is little need for things like sed/awk for truncation and replacement.

I am only injecting the bash binary into the container so KISS.

KellerFuchs · 2017-05-22T00:19:21Z

terraform/environments/ci/main.tf

+}
+
+module "vpc" {
+    source = "github.com/terraform-community-modules/tf_aws_vpc?ref=v1.0.6"


Does that mean that anyone with push access to github.com/terraform-community-modules/tf_aws_vpc gets code execution as the admin running terraform (or as our CD infra, eventually)?

Could easily make this a submodule.

KellerFuchs · 2017-05-22T00:20:37Z

terraform/environments/ci/main.tf

+module "vpc" {
+    source = "github.com/terraform-community-modules/tf_aws_vpc?ref=v1.0.6"
+    name = "ci"
+    cidr = "10.0.0.0/16"


Why are we allocating the whole of 10/16 to the CI?
What's actually in use there? Please document the network segmentation.

Just giving a 10/16 to each enviornment of which will initally only be Production and CI. Keeping them far apart to have plenty of addresses for throw-away workers etc in the future.

To further clarify, we can later (with a bit more $) upgrade this with the drone agent containers in a dedicated autoscale group as dumb on-demand workers of which they will churn IP addresses pretty quick when doing a bunch of builds/jobs at once.

KellerFuchs · 2017-05-22T00:21:42Z

terraform/modules/coreos_asg/main.tf

+    name = "${var.name}"
+}
+
+data "aws_ami" "coreos_stable" {


Nope, not doubling down on the CoreOS mistake.
We had this discussion already...

This is not a k8s cluster. This is a standalone single-server solution that will manage the k8s cluster, terraform, automated testing etc.

If you want to write and maintian your own standalone baseimage that provides the ability to fully bootstrap everything I need via user-data I am happy to use that. We could also just switch to your drop-in replacement later once it exists with a 5 line PR here.

In this case being super minimal is pretty ideal imo. All of this is is totally fine to nuke&pave which lends well to the benefits of ContainerLinux.

Also ContainerLinux is the officially supported base OS for k8s so I will probably opt to use that there too until your replacement exists.

@daurnimator We would just need to make a base image that is all set up for that use case, and maintain that image, and the pipeline for that image.

That image would also probably need to be maintained by CI... and now we ended up with a strange recursion situation.

IMO having CI itself use an upstream maintained distro that has everything we need to be hands-off is not a bad thing.

This is a standalone single-server solution that will manage the k8s cluster, terraform, automated testing etc.

Can you explain that part?

@KellerFuchs droneci has the ability to deploy k8s jobs, run ansible playbooks, apply terraform configuration on merge to master, check signatures, build containers, and do pretty much anything we want to automate via the many available modules. The drone.yml files included in projects are ~ a superset of travis.

That is why I opted to put this in a dedicated VPC/Enviornment with full machine/network isolation away from the production environments.

If we were to later have a develop environment or a stanging enviornment etc we would allow this VPC to have "VPC Peering" to any other environment, without giving them access to talk to each other.

Also since this module will be managing terraform templates, it can't manage itself via terraform. Given that this enviornment has a dedicated terraform root-config to manage drone and all the AWS resources it needs to function and generally maintain itself.

From there, it could then manage any other VPCs/environments via other terraform configs or ansible playbooks.

Really shooting for this to be very hands-off thus the auto-dns updating etc so it can just terminate and rebuild itself on failure or for updates.

KellerFuchs · 2017-05-22T00:23:23Z

terraform/modules/route53_balancer/lambda/asg_ip_attach/index.py

+def handler(event, context):
+    ec2 = boto3.resource('ec2')
+    route53 = boto3.client('route53')
+    message = json.loads(event['Records'][0]['Sns']['Message'])


How are we ensuring that only legit messages land in the SNS queue?

Via the IAM rules that are also automatically deployed with this module.

https://github.com/hashbang/admin-tools/pull/63/files/0b2fefab2d1f7f15d3cb0c19e742495cd4630186#diff-39ceb2360f3ad2c62e6b1444254fe8d6R114

That is the IAM policy which is only made available to the AWS autoscale group API to report to it.

It is whitelist only, and nothing else is given write access to it.

I had missed that part, thanks :3

KellerFuchs · 2017-05-22T00:26:05Z

terraform/modules/security_group/main.tf

+    }
+}
+
+resource "aws_security_group_rule" "all_internal_ingress" {


Is there a way we can avoid allowing everything as ingress?
(Not a priority, and not required for initial dpeloyment.)

We are free to add any rules that we want. This is just one of many rules we can include for any given use case.

This is just meant as a holding module for very common rules for simpler syntax.

Custom rules can always be created in-place.

lrvick · 2017-05-23T04:43:28Z

@KellerFuchs I think you may of misunderstood the scope of this.

This is meant to be the single server CI environment and all attempts have been made to keep it as lean as possible.

I could use Vault but then we need to maintain a vault cluster for 2 secrets.

I could use pass but terraform preserves rendered configs in plaintext. KMS is the easiest/cheapest way to bootstrap secrets out-of-band to be made available only to a given AWS instance role JIT.

Once k8s is actually deployed in the production environment/vpc the built in etcd based secret store is probably good enough for all of our services with the exception of this one.

CI is the only machine with god rights so I wanted it to be very much isolated and self-contained away from the production environment.

lrvick · 2017-05-23T04:58:04Z

Re vendor lock-in: this is taking advantage of AWS built-ins in order to minimize cost since this is standalone. We could develop alternate more expensive secret handling such as vault and deploy this same cloud-init to most any other provider though. But then vault itself needs to be bootstrapped. The very top of the chain can't avoid having some lock-in, but we can make sure everything down-stream in this case does not know about anything but drone, which is portable.

The production cluster will be k8s and the apps in that environment won't know anything about AWS.

KellerFuchs · 2017-05-23T05:01:20Z

I think you may of misunderstood the scope of this.

I indeed did!

Misunderstood scope, should re-review.

singlerider · 2018-04-04T21:29:42Z

I'm afraid it's been... nine years. Is this going anywhere?

mayli · 2018-04-04T22:28:07Z

nope

lrvick · 2018-04-04T23:09:56Z

I don't think anyone but me can test or review this, so it will be blocked for the forseeable future. I am focused on K8S deployment now and Ansible conversion which will solve a lot of our current blockers and allow things like nginx to ship finally. Once those are done, I'll follow up with a new version of this which will be much smaller in scope. In particular DroneCI will be handled as a separate scope of work. We will deploy k8s with terraform in the short term.

…

On Wed, Apr 4, 2018 at 3:28 PM, Mengyang Li ***@***.***> wrote: nope — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEOUKBapGWvBPqRCTU4uQ5V2cngXouhks5tlUj4gaJpZM4NhbPD> .

-- Lance R. Vick

__________________________________________________ Cell - 650.686.8819 IRC - [email protected] Website - http://lrvick.net PGP Key - http://lrvick.net/0x36C8AAA9.asc

__________________________________________________

multiple environments + droneci

0b2fefa

lrvick requested review from DeviaVir, singlerider, KellerFuchs and RyanSquared May 20, 2017 21:59

KellerFuchs previously requested changes May 22, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multiple environments + droneci #63

multiple environments + droneci #63

lrvick commented May 20, 2017 •

edited

Loading

KellerFuchs left a comment

KellerFuchs May 22, 2017

lrvick May 23, 2017

lrvick May 23, 2017

KellerFuchs May 22, 2017

KellerFuchs May 22, 2017

lrvick May 23, 2017 •

edited

Loading

KellerFuchs May 22, 2017

lrvick May 23, 2017

KellerFuchs May 22, 2017

lrvick May 23, 2017

lrvick May 23, 2017 •

edited

Loading

KellerFuchs May 22, 2017

lrvick May 23, 2017 •

edited

Loading

lrvick May 23, 2017

daurnimator May 23, 2017 •

edited by lrvick

Loading

KellerFuchs May 23, 2017

lrvick May 23, 2017 •

edited

Loading

KellerFuchs May 22, 2017

lrvick May 23, 2017

lrvick May 23, 2017

KellerFuchs May 23, 2017

KellerFuchs May 22, 2017

lrvick May 23, 2017 •

edited

Loading

lrvick commented May 23, 2017 •

edited

Loading

lrvick commented May 23, 2017 •

edited

Loading

KellerFuchs commented May 23, 2017

singlerider commented Apr 4, 2018

mayli commented Apr 4, 2018

lrvick commented Apr 4, 2018 via email

multiple environments + droneci #63

Are you sure you want to change the base?

multiple environments + droneci #63

Conversation

lrvick commented May 20, 2017 • edited Loading

KellerFuchs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lrvick May 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lrvick May 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lrvick May 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daurnimator May 23, 2017 • edited by lrvick Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lrvick May 23, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lrvick May 23, 2017 • edited Loading

Choose a reason for hiding this comment

lrvick commented May 23, 2017 • edited Loading

lrvick commented May 23, 2017 • edited Loading

KellerFuchs commented May 23, 2017

singlerider commented Apr 4, 2018

mayli commented Apr 4, 2018

lrvick commented Apr 4, 2018 via email

lrvick commented May 20, 2017 •

edited

Loading

lrvick May 23, 2017 •

edited

Loading

lrvick May 23, 2017 •

edited

Loading

lrvick May 23, 2017 •

edited

Loading

daurnimator May 23, 2017 •

edited by lrvick

Loading

lrvick May 23, 2017 •

edited

Loading

lrvick May 23, 2017 •

edited

Loading

lrvick commented May 23, 2017 •

edited

Loading

lrvick commented May 23, 2017 •

edited

Loading