add deployment scripts and docs (#51)

README.md

@@ -2,23 +2,64 @@
 # Remote Key Server (RKS)
 > Centralized key server for management and distribution of TLS certificates and private keys to edge servers
 
+## Table of content
+  - [What is the Remote Key Server](#what-is-the-remote-key-server)  
+  - [Getting started](#getting-started)  
+  - [Usage](#usage)  
+  - [Links](#links)  
+  - [Authors](#authors)  
+
+
 ## What is the Remote Key Server
 The Remote Key Server is a solution to store TLS certificates and private keys and give secure access to these secrets to remote nodes.
 The main use case is enabling distributed servers to serve HTTPS traffic while securing the TLS keys storage and delivery.
 
 Here are a few selling points
-- Change from a push model where private keys are provisionned on nodes to a pull model where nodes ask for the private key only when they need it
-- By allowing nodes to request private keys on demand, they don't need to store it on disk and can keep it in memory
-- Ease certificates updates by only having to update the certificate on the RKS. This is especially useful in the context of short lived certificates
+- Change from a push model where private keys are provisionned on nodes to a pull model where nodes ask for the private key only when they need it  
+- By allowing nodes to request private keys on demand, they don't need to store it on disk and can keep it in memory  
+- Ease certificates updates by only having to update the certificate on the RKS. This is especially useful in the context of short lived certificates  
 
-The **RKS** is an API wrapper around Hashicorp Vault.
+The **RKS** is an API wrapper around [Hashicorp Vault](https://github.com/hashicorp/vault) secret store.
 It restricts and simplifies Vault to model interactions between servers who need access to secret TLS keys and Vault which stores them.
 
-Vault being a big toolbox with a lot of functionalities, we decided to implement an API on top of it with higher level functionalities.
-That way we can simplify its usage while hiding Vault intricacies like backend setup, token generation, policies.
+It wraps Vault API to manage, store and deliver TLS private keys and certificates in the context of HTTPS content delivery
+
+It builds upon Vault to provide a simple API to:
+- Manage certificate/private key pairs (Create/Read/Update/Delete)  
+- Manage a group of nodes and permissions to access secrets  
+- Manage node registration and authorization when requesting secrets  
+
+It consists of 4 sub API defined in the [OpenAPI specification](./rks-openapi.yaml):
+- [Initialization](./api/initialize) to setup the RKS  
+- [Administration](./api/admin) for configuration operations (Group creation, secret provisionning...)  
+- [Node](./api/node) for node registration  
+- [Secret](./api/secret) for secret access by the nodes  
 
 The API revolves around *Nodes* which need access to certain certificates/keys and *Group* of Nodes which represent logical grouping of nodes with same access to secrets
 
+### Concepts
+The RKS is based on two concepts: nodes and group of nodes
+
+A node is an entity requiring access to certificates and private keys in order to establish sessions with clients.
+It can be an edge server, a cache server, a load balancer...
+
+A group of nodes represent a system like a Content Delivery Network, Edge computing sites, distributed HTTP proxies...
+All nodes in a group share the same secret access permissions.
+
+On creation, a group is given a **group token**.
+This **group token** has to be provisionned onto group nodes so that they can request their **node token** to the RKS.
+
+For a node to get it's node token, it has to make a call to the registration endpoint of the rks using the group token.
+If a callback url has been configured on group creation it will be called on each node registration.
+An example callback URL implementation is available in [./tests/mock-callback-server/server.py](./tests/mock-callback-server/server.py). According to the callback server response the node will be provided a **node token** (HTTP 200) or be refused (40X, 50X)
+
+This allows groups to control which node can register to the RKS and protect against **group token** compromission by adding another layer of verification
+
+A node can access secrets authorized for his group on demand by querying the secret endpoint using its node token.
+If the node is allowed to access the secret, it is delivered along a Time To Live indicating for how long the node can keep the secret.
+When the TTL expires the node **must** destroy the secret and query the secret endpoint again.
+This TTL is there to avoid nodes storing secrets on disk or for a long time when it is not needed
+
 ## Getting started
 ### With docker image
 You could get our latest rks-aio docker image on the github package registry ([see how to configure docker for use with github package](https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-docker-for-use-with-github-packages))
@@ -40,6 +81,7 @@ make dev-env
 ```
 The vault root token needed for the RKS initialization is printed in the **root\_token** file
 
+For deployment without docker, please see [deploy documentation](./deploy/Deploy.md)
 
 ### In both cases
 
@@ -125,44 +167,7 @@ $ curl -k https://localhost:8080/rks/v1/secret/rks.local -H "X-Vault-Token: $NOD
 failed
 ```
 
-## Description
-The RKS is based on [Hashicorp Vault](https://github.com/hashicorp/vault) secret store.
-It wraps Vault API to manage, store and deliver TLS private keys and certificates in the context of HTTPS content delivery
-
-It builds upon Vault to provide a simple API to:
-- Manage certificate/private key pairs (Create/Read/Update/Delete)
-- Manage a group of nodes and permissions to access secrets
-- Manage node registration and authorization when requesting secrets
-
-It consists of 4 sub API defined in the [OpenAPI specification](./rks-openapi.yaml):
-- [Initialization](./api/initialize) to setup the RKS
-- [Administration](./api/admin) for configuration operations (Group creation, secret provisionning...)
-- [Node](./api/node) for node registration
-- [Secret](./api/secret) for secret access by the nodes
-
-
-### Concepts
-The RKS is based on two concepts: nodes and group of nodes
-
-A node is an entity requiring access to certificates and private keys in order to establish sessions with clients.
-It can be an edge server, a cache server, a load balancer...
-
-A group of nodes represent a system like a Content Delivery Network, Edge computing sites, distributed HTTP proxies...
-All nodes in a group share the same secret access permissions.
-
-On creation, a group is given a **group token**.
-This **group token** has to be provisionned onto group nodes so that they can request their **node token** to the RKS.
-
-For a node to get it's node token, it has to make a call to the registration endpoint of the rks using the group token.
-If a callback url has been configured on group creation it will be called on each node registration.
-An example callback URL implementation is available in [./tests/mock-callback-server/server.py](./tests/mock-callback-server/server.py). According to the callback server response the node will be provided a **node token** (HTTP 200) or be refused (40X, 50X)
-
-This allows groups to control which node can register to the RKS and protect against **group token** compromission by adding another layer of verification
-
-A node can access secrets authorized for his group on demand by querying the secret endpoint using its node token.
-If the node is allowed to access the secret, it is delivered along a Time To Live indicating for how long the node can keep the secret.
-When the TTL expires the node **must** destroy the secret and query the secret endpoint again.
-This TTL is there to avoid nodes storing secrets on disk or for a long time when it is not needed
+For more information about Usage please see [rks-handguide](./docs/rks-handguide.md)
 
 ## Links
 - Project Homepage: https://github.com/Orange-OpenSource/remote-key-server

deploy/Deploy.md

@@ -0,0 +1,70 @@
+# Deployment guide
+
+This an example deployment, which is using ansible roles to deploy consul and vault.
+
+By default, rks-server is deployed with rks.local.pem and rks.local.key, if you want to use a custom certificate, please put certificate and private key in 'roles/ansible-rks-server/files/'. 
+
+Then you will have to set to right values in:
+
+'./vars/vars.yml'
+
+And set target host IP in './hosts' file (multiple hosts for a cluster)
+
+You will also have to set ANSIBLE_SSH_USER to target host user in 'Makefile'.
+
+Then run:
+
+```bash
+make rks-install
+```
+For more documentations on consul and vault ansible deployment roles, please see:
+
+[ansible-vault](https://github.com/ansible-community/ansible-vault)  
+[ansible-consul](https://github.com/ansible-community/ansible-consul)
+
+##Set up deployed components
+
+In commands below, please replace 'rks-vault', 'rks-server' with your target host ip.
+
+If this is the **first** install of : Consul, Vault, RKS on this target host, 
+you have to **initialize Vault**. You can do this by calling the init endpoint on vault:
+
+```bash
+curl http://rks-vault:8200/v1/sys/init -X PUT -d '{"secret_shares": 1, "secret_threshold": 1}'
+{"keys":["ef8243a53feccc73001812950d9822d77975fab20beff5e3946c076bd972ae6f"],"keys_base64":["74JDpT/szHMAGBKVDZgi13l1+rIL7/XjlGwHa9lyrm8="],"root_token":"s.YXBSZBpCqwjizLekuONvzsRe"}
+export KEY=ef8243a53feccc73001812950d9822d77975fab20beff5e3946c076bd972ae6f
+export root_token=s.YXBSZBpCqwjizLekuONvzsRe
+```
+This will generate a master key used to unseal Vault.
+There could be several master keys which will all be required to unseal Vault, please, see [Vault documentation](https://www.vaultproject.io/api-docs/system/init#start-initialization).
+This will also generate a root_token that we will use to configure vault.
+
+Next step is to **unseal vault**:
+
+```bash
+curl http://rks-vault:8200/v1/sys/unseal -X PUT -d "{\"key\": \"$KEY\"}"
+{"type":"shamir","initialized":true,"sealed":false,"t":1,"n":1,"progress":0,"nonce":"","version":"1.3.1","migration":false,"cluster_name":"dc1","cluster_id":"16b06562-2417-935f-8edc-0155d06334c0","recovery_seal":false,"storage_type":"consul"}
+```
+
+Please see [Vault documentation](https://www.vaultproject.io/api/system/unseal.html) to unseal with several master keys.
+
+Vault initialization is done.
+You are now able to authenticate to vault with **root_token**.
+
+Once Consul and Vault are installed, initialized and unsealed, you need to call the **RKS Initialization API** endpoint, to **configure vault for RKS use and create an RKS admin user**:
+
+```bash
+curl -k -X POST -H "X-Vault-Token: $root_token" https://rks-server/rks/v1/init
+
+```
+## What about a cluster install
+
+Vault and Consul can be configured to run in cluster mode, that is, running several instances of each component. It is the recommended production setup because it is more resilient to failure.
+
+For more information about a cluster installation (see [Vault documentation](https://learn.hashicorp.com/vault/getting-started/deploy)), you will need to call init on one vault node, then unseal each vault node.
+
+
+## Start scenario
+
+please see [rks-handguide](../docs/rks-handguide.md)
+

deploy/Makefile

@@ -0,0 +1,18 @@
+ANSIBLE_SSH_USER := <TARGET_HOST_USER>
+
+.PHONY: rks_install
+
+roles/ansible-consul:
+	git clone https://github.com/brianshumate/ansible-consul roles/ansible-consul
+	cd roles/ansible-consul/ && git checkout v2.5.4
+
+roles/ansible-vault:
+	git clone https://github.com/brianshumate/ansible-vault roles/ansible-vault
+	cp ./vault_systemd_unconfined.service.j2 ./roles/ansible-vault/templates/
+	cd roles/ansible-vault/ && git checkout v2.5.2
+build_rks:
+	cd ../ && export CGO_ENABLED=0 && go mod download && go build -o deploy/roles/ansible-rks-server/files/rks-server ./cmd/remote-key-server/
+
+rks_install: build_rks roles/ansible-consul roles/ansible-vault
+	ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook -v -i ./hosts --user=${ANSIBLE_SSH_USER} --become ./rks_install.yaml
+

deploy/hosts

@@ -0,0 +1,3 @@
+[consul_instances]
+node0 ansible_host=<IP_target_host> consul_node_role=server consul_bootstrap_expect=true
+

deploy/rks_install.yaml

@@ -0,0 +1,23 @@
+---
+
+- hosts: consul_instances
+  tasks:
+    - name: test
+      debug:
+        msg: "Hello"
+
+- hosts: consul_instances
+  environment:
+    http_proxy: "{{ lookup('env', 'http_proxy') or  lookup('env', 'HTTP_PROXY')}}"
+    https_proxy: "{{ lookup('env', 'https_proxy') or lookup('env', 'HTTPS_PROXY')}}"
+    no_proxy: "localhost,127.0.0.1,{{ groups['consul_instances'] | \
+               map('extract', hostvars, ['ansible_host']) | join(',') }}"
+    CONSUL_CONFIGURE_SYSLOGD: true
+
+  #we could define vars in vars.yml
+  vars_files:
+    - ./vars/vars.yml
+  roles:
+    - ansible-consul
+    - ansible-vault
+    - ansible-rks-server

deploy/roles/ansible-rks-server/defaults/main.yml

@@ -0,0 +1,8 @@
+---
+rks_admin_login: admin
+rks_admin_pwd: 12345
+rks_listen_address: ":443"
+rks_config_path: /etc/rks/
+rks_cert_name: rks.local.pem
+rks_pkey_name: rks.local.key
+rks_vault_address: http://{{ ansible_default_ipv4.address }}:8200

deploy/roles/ansible-rks-server/files/rks.local.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCnD5+tBWvR40wL
+ViA2qDoyT0T/g7nLh+yhAtFl2u/hMDBO9cBc6Vo/S2PfhD+vxv64PvEKX27e/A1D
+PCW3y5k+to8l0CgURX093mgv+nLvqJxFW9YZHEjUv1OGQ2clsKU/flia+Cmke3W9
+oPQNN8Gv4eKWHnnATqQgcBKJ4EkBRhH2y+8yNbSIQuLh0hgdPSOQpvBA7PttoLLR
+KWC2TRCN6KlZyHJKCuwv1JtDgPFyK4HWf9Fl/ChYXgdO6XIUJNopAFoIU8fjNdTB
+fI37p2cYvFnEr7NiDZVRH/rnu9i1UNqsBnN6b09nGzBSsiSn/s0brTSwp2EBFxLH
+1JXNr5NjAgMBAAECggEAL0fotRs3rKtbtkmlzFJIRjzRFk+9obDSc69MfDs+cMkm
+ovJCgwk4FXnMxlGzhJgZNyP5LCpeQpWrW0AGnMHumSUp1lXXQgV4sK3lZs7HALUD
+pU2chnbO1gdgKDujUbNw0Ut84D8j6f1XEVggOi1xj8mqSzpM0vRu1w9g78yeXNF2
+2iY7rvwRYfZs9f/ewXhUOl7BYA6579njf/z3EFfrI0PCZJUHZ28QK+djOsRqWsnr
+yN0oD6+hTup/xeXfRSB7oAdAhiz22asFULIBWqJpLGvSjB9W83qEVsW9kTQ3Bfzr
+/0m0TksR9PHUhyFZF80LU6VUCVx9ijEP1MPBZaVjIQKBgQDWRI2jXRPbe/CKPfQN
+5hwQLR7CpBGsQHpQiiKI0hYVEZlK2DChQn+xWHLRGUNKVU46szZf1Ai9IqjBm+JF
+EAvERf3rGrI/szaFKs0R1LG3dtN8tyUo25HWEG41+hGI5olZbr5xO1hLma83HKoP
+1ALlUlWVYnq6o9JSYDylFxw4zwKBgQDHmVRaqq48q7XKeS4Uy/ufBy7MvSbqt51B
+g1ze4oRyTgFymOYaEDuR6zUxikCsepqbLm0KQ51kDYTtlOq6QI3lPcmBkaI8S5NW
+flkcd7t/1p6fcQAJaO1B6Zo+8+81VSgXdS8BTtU0DdpCSxhdWwx62fNsiDDcYETj
+qE2Zr5S5LQKBgQDQk2MdR5nw6jE5IR3V+c4PoRx0dKoPd6SZih4b+gYs9Di4tsJ/
+9++9f1AlY+YxEeFZGuMvXYQ0OOz1qv0UJY6OvWNz3UTu5VGJSCTf2qA066LroIy3
+vSW1r8DZIli3X1MMj9Uq/I3y6eO6Tljtl4BKI1/UT00RKi+uqKfgPSLn2QKBgHxP
+Eiolgr5mcqHXNsvc/qjcYFlCtKoaEOZsWbq2eKZ3zo8Ais5QrckDI+3mnMDO2tsq
+3t41niBiCxesV0QIUFclVApOGSxUJT8JEZaZOz0y/TdvuQYjQyB5zIbIhYJBtuZl
+JRE7d60js3r2KgZ6XhtlAiXl7AhULJNcjnfY0ldBAoGAPHnxf8mkZW2ThQMpidlf
+ydausZgNkgw3UKwjWk/rItDKbBLua5e91fRI4Womb0VR0+9PrQ0KjjnDCFWCLkhC
+QkLfoZTP4gI5eZJz20rgCKJA+6sHOaOvEFDGLYc940jKrvgAaPHO5DlUObdkHjrP
+crqmIBjrrMjW9Sker/rFjMY=
+-----END PRIVATE KEY-----

deploy/roles/ansible-rks-server/files/rks.local.pem

@@ -0,0 +1,21 @@
+-----BEGIN CERTIFICATE-----
+MIIDhTCCAm2gAwIBAgIUeHKnHOUKd3VvlXQ9wEKfgXMxBmIwDQYJKoZIhvcNAQEL
+BQAwLjELMAkGA1UEBhMCRlIxHzAdBgkqhkiG9w0BCQEWEHRlc3RAZXhhbXBsZS5j
+b20wHhcNMTkwNTA3MDk1OTAwWhcNMjkwNTA0MDk1OTAwWjBCMQswCQYDVQQGEwJG
+UjESMBAGA1UEAwwJcmtzLmxvY2FsMR8wHQYJKoZIhvcNAQkBFhB0ZXN0QGV4YW1w
+bGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApw+frQVr0eNM
+C1YgNqg6Mk9E/4O5y4fsoQLRZdrv4TAwTvXAXOlaP0tj34Q/r8b+uD7xCl9u3vwN
+Qzwlt8uZPraPJdAoFEV9Pd5oL/py76icRVvWGRxI1L9ThkNnJbClP35YmvgppHt1
+vaD0DTfBr+Hilh55wE6kIHASieBJAUYR9svvMjW0iELi4dIYHT0jkKbwQOz7baCy
+0Slgtk0QjeipWchySgrsL9SbQ4DxciuB1n/RZfwoWF4HTulyFCTaKQBaCFPH4zXU
+wXyN+6dnGLxZxK+zYg2VUR/657vYtVDarAZzem9PZxswUrIkp/7NG600sKdhARcS
+x9SVza+TYwIDAQABo4GGMIGDMFMGA1UdIwRMMEqhMqQwMC4xCzAJBgNVBAYTAkZS
+MR8wHQYJKoZIhvcNAQkBFhB0ZXN0QGV4YW1wbGUuY29tghQPNroTcjy3KkVWybBy
+0grPI22gbDAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAUBgNVHREEDTALgglya3Mu
+bG9jYWwwDQYJKoZIhvcNAQELBQADggEBADp1WF97f7VTOR5HSQcobjYIOvqkrUUk
+g818BeOyh6YH5VT0lGF45VVU025cFAPg8qnMiRu/Vz4YdWBARyzutP2cEQ5cSL+V
+hb0POL+SPnoIf/MnKUDwuGMWzOY1IIeQqNDrEvYLqgXMJFUwgBAUpSQv7ndE8X5E
+yFd9Is5mkN3xVBsS+Anf24i3KomM6Kf20sXUu257lbsMapO9ppe5z9WyqAvu5jNy
+4xjI1Yyk+rWQsIuJql8trpnkSl8raRg+3+qhoxdWAu79JYX8NM9nAuOaO1ijWXfR
+ig6kjCqw+C9A9EpxNxDPa1m5hmt/z0O9VKGdmhF5CieHW0KXXXcRYIA=
+-----END CERTIFICATE-----

deploy/roles/ansible-rks-server/tasks/main.yml

@@ -0,0 +1,33 @@
+---
+
+- name: Copy systemd unit
+  template:
+    src: rks-server.service.j2
+    dest: /lib/systemd/system/rks-server.service
+
+- name: Create rks config directory
+  file:
+    path: "{{ rks_config_path }}"
+    state: directory
+
+- name: Copy rks-server binary
+  copy:
+    src: rks-server
+    dest: /usr/bin/rks-server
+    mode: u+x
+
+- name: Copy cert and key
+  copy:
+    src: "{{ item }}"
+    dest: "{{ rks_config_path }}/{{ item }}"
+    mode: u+x
+  loop:
+    - "{{rks_cert_name}}"
+    - "{{rks_pkey_name}}"
+
+- name: Start RKS server
+  systemd:
+    name: rks-server.service
+    daemon_reload: true
+    enabled: true
+    state: restarted

deploy/roles/ansible-rks-server/templates/rks-server.service.j2

@@ -0,0 +1,16 @@
+[Unit]
+Description=RKS Server
+Requires=network-online.target consul.service vault.service
+After=network-online.target consul.service vault.service
+
+[Service]
+ExecStart=/usr/bin/rks-server --adminLogin {{ rks_admin_login }} --adminPwd {{ rks_admin_pwd }} --listenAddress "{{ rks_listen_address }}" --cert {{ rks_config_path }}/{{ rks_cert_name }} --pkey {{ rks_config_path }}/{{ rks_pkey_name }} --vaultaddr {{ rks_vault_address }}
+KillMode=process
+KillSignal=SIGTERM
+Restart=on-failure
+LimitNOFILE=50000:50000
+LimitNPROC=500000
+
+
+[Install]
+WantedBy=multi-user.target

deploy/vars/vars.yml

@@ -0,0 +1,16 @@
+consul_syslog_enable: true
+consul_version: 1.6.1
+vault_version: 1.3.1
+vault_backend: consul
+vault_iface: ens192
+#may be deleted
+vault_awskms: false
+vault_log_level: debug
+rks_admin_login: admin
+rks_admin_pwd: 12345
+rks_listen_address: ":443"
+rks_config_path: /etc/rks/
+rks_cert_name: rks.local.pem
+rks_pkey_name: rks.local.key
+rks_vault_address: http://{{ ansible_default_ipv4.address }}:8200
+