From a9f4b9f9545b21c040a29fb7a9264d7da58a0d1c Mon Sep 17 00:00:00 2001 From: Fredrik Hoem Grelland <40291976+fredrikhgrelland@users.noreply.github.com> Date: Thu, 25 Jun 2020 08:12:12 +0200 Subject: [PATCH 01/10] Initial commit --- LICENSE | 201 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 LICENSE diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..261eeb9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. From 9d0dc24c4866c5698934c092fe72b9da9125650d Mon Sep 17 00:00:00 2001 From: fredrikhgrelland Date: Thu, 25 Jun 2020 06:17:23 +0000 Subject: [PATCH 02/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/afd02e54d4eda5d86d384d0205e2cf8e8646e0b4) --- HOWTO.md | 12 +++ LICENSE | 201 ------------------------------------ Makefile | 28 +++++ Vagrantfile | 13 +++ conf/consul/99-override.hcl | 0 conf/nomad/99-override.hcl | 0 conf/vault/99-override.hcl | 0 7 files changed, 53 insertions(+), 201 deletions(-) create mode 100644 HOWTO.md delete mode 100644 LICENSE create mode 100644 Makefile create mode 100644 Vagrantfile create mode 100644 conf/consul/99-override.hcl create mode 100644 conf/nomad/99-override.hcl create mode 100644 conf/vault/99-override.hcl diff --git a/HOWTO.md b/HOWTO.md new file mode 100644 index 0000000..fedecac --- /dev/null +++ b/HOWTO.md @@ -0,0 +1,12 @@ +# Development template for `fredrikhgrelland/hashistack` + +This template can be used as a base image for developing services on the hashistack. + +## Change configuration of hashistack + +- consul `conf/consul/99-override.hcl` +- nomad `conf/nomad/99-override.hcl` +- vault `conf/vault/99-override.hcl` + +You may edit the `99-override.hcl` or add you own. +Any valid configuration added to these directories will be added and lexically merged. diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 261eeb9..0000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..dd6c487 --- /dev/null +++ b/Makefile @@ -0,0 +1,28 @@ +include .env +export +export PATH := $(shell pwd)/tmp:$(PATH) + +#### Development #### +# start commands +up: clean update-box + SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant up --provision + +update-box: + @SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant box update || (echo '\n\nIf you get an SSL error you might be behind a transparent proxy. \nMore info https://github.com/fredrikhgrelland/vagrant-hashistack/blob/master/README.md#if-you-are-behind-a-transparent-proxy\n\n' && exit 2) + +# clean commands +destroy-box: + vagrant destroy -f + +remove-tmp: + rm -rf ./tmp + +clean: destroy-box remove-tmp + +copy-consul: + if [ ! -f "./tmp/consul" ]; then mkdir -p ./tmp; vagrant ssh -c "cp /usr/local/bin/consul /vagrant/tmp/consul"; fi; + +#### Test #### TODO: move to test template + +test: up + $(MAKE) -C test test diff --git a/Vagrantfile b/Vagrantfile new file mode 100644 index 0000000..ec78cdc --- /dev/null +++ b/Vagrantfile @@ -0,0 +1,13 @@ +Vagrant.configure("2") do |config| + config.vm.box = "fredrikhgrelland/hashistack" + + config.vm.provider "virtualbox" do |vb| + vb.linked_clone = true + vb.memory = 2048 + end + + config.vm.provision "ansible_local" do |startup| + run = "always" + startup.playbook = "/etc/ansible/startup.yml" + end +end diff --git a/conf/consul/99-override.hcl b/conf/consul/99-override.hcl new file mode 100644 index 0000000..e69de29 diff --git a/conf/nomad/99-override.hcl b/conf/nomad/99-override.hcl new file mode 100644 index 0000000..e69de29 diff --git a/conf/vault/99-override.hcl b/conf/vault/99-override.hcl new file mode 100644 index 0000000..e69de29 From 291cd25624ddfb996087f438b00dc10eb60134d3 Mon Sep 17 00:00:00 2001 From: fredrikhgrelland Date: Fri, 26 Jun 2020 19:35:36 +0000 Subject: [PATCH 03/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/9f1367fa46f61690be689739e8da41d6a0b18cde) --- HOWTO.md | 12 ---- Makefile | 9 +-- README.md | 26 ++++++++ Vagrantfile | 7 +-- conf/consul/99-override.hcl | 0 conf/nomad/99-override.hcl | 0 conf/vault/99-override.hcl | 0 test/.env | 2 + test/Makefile | 15 +++++ test/Vagrantfile | 7 +++ test/ansible/playbook.yml | 14 +++++ test/nomad/countdash.hcl | 60 +++++++++++++++++++ test/terraform/main.tf | 8 +++ .../ansible/playbooks/poststart/0-example.yml | 3 + .../ansible/playbooks/prestart/0-example.yml | 3 + .../conf/hashistack/consul/99-override.hcl | 1 + vagrant/conf/hashistack/nomad/99-override.hcl | 1 + vagrant/conf/hashistack/vault/99-override.hcl | 1 + 18 files changed, 147 insertions(+), 22 deletions(-) delete mode 100644 HOWTO.md create mode 100644 README.md delete mode 100644 conf/consul/99-override.hcl delete mode 100644 conf/nomad/99-override.hcl delete mode 100644 conf/vault/99-override.hcl create mode 100644 test/.env create mode 100644 test/Makefile create mode 100644 test/Vagrantfile create mode 100644 test/ansible/playbook.yml create mode 100644 test/nomad/countdash.hcl create mode 100644 test/terraform/main.tf create mode 100644 vagrant/conf/ansible/playbooks/poststart/0-example.yml create mode 100644 vagrant/conf/ansible/playbooks/prestart/0-example.yml create mode 100644 vagrant/conf/hashistack/consul/99-override.hcl create mode 100644 vagrant/conf/hashistack/nomad/99-override.hcl create mode 100644 vagrant/conf/hashistack/vault/99-override.hcl diff --git a/HOWTO.md b/HOWTO.md deleted file mode 100644 index fedecac..0000000 --- a/HOWTO.md +++ /dev/null @@ -1,12 +0,0 @@ -# Development template for `fredrikhgrelland/hashistack` - -This template can be used as a base image for developing services on the hashistack. - -## Change configuration of hashistack - -- consul `conf/consul/99-override.hcl` -- nomad `conf/nomad/99-override.hcl` -- vault `conf/vault/99-override.hcl` - -You may edit the `99-override.hcl` or add you own. -Any valid configuration added to these directories will be added and lexically merged. diff --git a/Makefile b/Makefile index dd6c487..b5919aa 100644 --- a/Makefile +++ b/Makefile @@ -2,6 +2,9 @@ include .env export export PATH := $(shell pwd)/tmp:$(PATH) +.ONESHELL .PHONY: up update-box destroy-box remove-tmp clean copy-consul test +.DEFAULT_GOAL := up + #### Development #### # start commands up: clean update-box @@ -22,7 +25,5 @@ clean: destroy-box remove-tmp copy-consul: if [ ! -f "./tmp/consul" ]; then mkdir -p ./tmp; vagrant ssh -c "cp /usr/local/bin/consul /vagrant/tmp/consul"; fi; -#### Test #### TODO: move to test template - -test: up - $(MAKE) -C test test +test: + $(MAKE) -C test diff --git a/README.md b/README.md new file mode 100644 index 0000000..1a31ef1 --- /dev/null +++ b/README.md @@ -0,0 +1,26 @@ +# Starter template for `fredrikhgrelland/hashistack` + +This repository can be used as a base for developing services on the hashistack. +On github, you may use the ["Use this template"](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to generate a new repository from this template. + +If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in this separate repository [vagrant-hashistack-template-dev](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to start a new repository from this repo +. + +Documentation on [parent repository](https://github.com/fredrikhgrelland/vagrant-hashistack#usage). + +## Customizing the vagrant box +The vagrant box ships with a default startup scheme. It will run an ansible playbook to start all services. +You may change the hashistack configuration or add aditional pre and post steps to the startup procedure to match your needs. + +### Overriding and extending the configuration of the hashistack + +- consul [vagrant/conf/hashistack/consul/99-override.hcl](vagrant/conf/hashistack/consul/99-override.hcl) +- nomad [vagrant/conf/hashistack/nomad/99-override.hcl](vagrant/conf/hashistack/nomad/99-override.hcl) +- vault [vagrant/conf/hashistack/vault/99-override.hcl](vagrant/conf/hashistack/vault/99-override.hcl) + +You may edit the `99-override.hcl` or add your own. +Any valid configuration added to these directories will be added to their respective services' configuration, in lexical order. + +### Pre- and post-startup ansible playbooks +This vagrant box will execute ansible playbooks put in two special directories [vagrant/conf/ansible/playbooks/prestart](vagrant/conf/ansible/playbooks/prestart) and [vagrant/conf/ansible/playbooks/poststart](vagrant/conf/ansible/playbooks/poststart). These playbooks will be executed before and after the box's bundled startup sequence, respectively. This gives the flexibility to configure all aspects of the hashistack as well as run tasks needed for tests or demo purposes as part of `vagrant up` Note; The playbooks are included into the main run, so the syntax in the [example](vagrant/conf/ansible/playbooks/prestart/0-example.yml) must be followed.. +They will be run in lexical order, and prefixing with numbers is a good way to get the order you want. diff --git a/Vagrantfile b/Vagrantfile index ec78cdc..a6e7528 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -1,13 +1,8 @@ Vagrant.configure("2") do |config| config.vm.box = "fredrikhgrelland/hashistack" - + config.vm.box_version = "~> 0.2" config.vm.provider "virtualbox" do |vb| vb.linked_clone = true vb.memory = 2048 end - - config.vm.provision "ansible_local" do |startup| - run = "always" - startup.playbook = "/etc/ansible/startup.yml" - end end diff --git a/conf/consul/99-override.hcl b/conf/consul/99-override.hcl deleted file mode 100644 index e69de29..0000000 diff --git a/conf/nomad/99-override.hcl b/conf/nomad/99-override.hcl deleted file mode 100644 index e69de29..0000000 diff --git a/conf/vault/99-override.hcl b/conf/vault/99-override.hcl deleted file mode 100644 index e69de29..0000000 diff --git a/test/.env b/test/.env new file mode 100644 index 0000000..272cdce --- /dev/null +++ b/test/.env @@ -0,0 +1,2 @@ +#NOMAD_TOKEN=NOT_IN_USE +#VAULT_TOKEN=NOT_IN_USE \ No newline at end of file diff --git a/test/Makefile b/test/Makefile new file mode 100644 index 0000000..e2490de --- /dev/null +++ b/test/Makefile @@ -0,0 +1,15 @@ +include .env +export + +.ONESHELL .PHONY: test clean up +.DEFAULT_GOAL := test + +up: + vagrant up --provision + $(MAKE) clean + +clean: + vagrant destroy -f + rm -rf .vagrant + +test: clean up diff --git a/test/Vagrantfile b/test/Vagrantfile new file mode 100644 index 0000000..6cc19e8 --- /dev/null +++ b/test/Vagrantfile @@ -0,0 +1,7 @@ +Vagrant.configure("2") do |config| + config.vm.box = "fredrikhgrelland/hashistack" + config.vm.provider "virtualbox" do |vb| + vb.linked_clone = true + vb.memory = 2048 + end +end diff --git a/test/ansible/playbook.yml b/test/ansible/playbook.yml new file mode 100644 index 0000000..2d8ff68 --- /dev/null +++ b/test/ansible/playbook.yml @@ -0,0 +1,14 @@ +--- +- hosts: all + become: yes + tasks: + - name: Terraform + terraform: + project_path: ../terraform + force_init: true + state: present + register: terraform + + - name: Terraform stdout + debug: + msg: "{{terraform.stdout}}" \ No newline at end of file diff --git a/test/nomad/countdash.hcl b/test/nomad/countdash.hcl new file mode 100644 index 0000000..9314ddf --- /dev/null +++ b/test/nomad/countdash.hcl @@ -0,0 +1,60 @@ +job "countdash" { + datacenters = ["dc1"] + group "api" { + network { + mode = "bridge" + } + + service { + name = "count-api" + port = "9001" + + connect { + sidecar_service {} + } + } + + task "web" { + driver = "docker" + config { + image = "hashicorpnomad/counter-api:v1" + } + } + } + + group "dashboard" { + network { + mode ="bridge" + port "http" { + static = 9002 + to = 9002 + } + } + + service { + name = "count-dashboard" + port = "9002" + + connect { + sidecar_service { + proxy { + upstreams { + destination_name = "count-api" + local_bind_port = 8080 + } + } + } + } + } + + task "dashboard" { + driver = "docker" + env { + COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}" + } + config { + image = "hashicorpnomad/counter-dashboard:v1" + } + } + } +} diff --git a/test/terraform/main.tf b/test/terraform/main.tf new file mode 100644 index 0000000..f09ba72 --- /dev/null +++ b/test/terraform/main.tf @@ -0,0 +1,8 @@ +provider "nomad" { + address = "http://127.0.0.1:4646" +} + +resource "nomad_job" "countdash" { + jobspec = file("${path.cwd}/../nomad/countdash.hcl") + detach = false +} \ No newline at end of file diff --git a/vagrant/conf/ansible/playbooks/poststart/0-example.yml b/vagrant/conf/ansible/playbooks/poststart/0-example.yml new file mode 100644 index 0000000..0709769 --- /dev/null +++ b/vagrant/conf/ansible/playbooks/poststart/0-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of poststart + debug: + msg: This would be a poststart task diff --git a/vagrant/conf/ansible/playbooks/prestart/0-example.yml b/vagrant/conf/ansible/playbooks/prestart/0-example.yml new file mode 100644 index 0000000..cc03e7f --- /dev/null +++ b/vagrant/conf/ansible/playbooks/prestart/0-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of prestart + debug: + msg: This would be a prestart task diff --git a/vagrant/conf/hashistack/consul/99-override.hcl b/vagrant/conf/hashistack/consul/99-override.hcl new file mode 100644 index 0000000..d2b8041 --- /dev/null +++ b/vagrant/conf/hashistack/consul/99-override.hcl @@ -0,0 +1 @@ +#Any vaild configuration from https://www.consul.io/docs/agent/options.html#configuration_files \ No newline at end of file diff --git a/vagrant/conf/hashistack/nomad/99-override.hcl b/vagrant/conf/hashistack/nomad/99-override.hcl new file mode 100644 index 0000000..d817340 --- /dev/null +++ b/vagrant/conf/hashistack/nomad/99-override.hcl @@ -0,0 +1 @@ +#Any vaild configuration from https://www.nomadproject.io/docs/configuration#general-parameters \ No newline at end of file diff --git a/vagrant/conf/hashistack/vault/99-override.hcl b/vagrant/conf/hashistack/vault/99-override.hcl new file mode 100644 index 0000000..2a828c6 --- /dev/null +++ b/vagrant/conf/hashistack/vault/99-override.hcl @@ -0,0 +1 @@ +#Any vaild configuration from https://www.vaultproject.io/docs/configuration \ No newline at end of file From 53467eebf7cb8c1ff5a406f5a583dbe8769aa47c Mon Sep 17 00:00:00 2001 From: fredrikhgrelland <> Date: Mon, 29 Jun 2020 13:23:20 +0000 Subject: [PATCH 04/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/19d28e673cee303c926a462a2f2920a0e089d2a8) --- template/.env | 1 + .../.github/workflows/on_pr_push_master.yml | 14 +++++ template/Makefile | 29 +++++++++ template/README.md | 26 ++++++++ template/Vagrantfile | 8 +++ template/test/.env | 2 + template/test/Makefile | 15 +++++ template/test/Vagrantfile | 7 +++ template/test/ansible/playbook.yml | 14 +++++ template/test/nomad/countdash.hcl | 60 +++++++++++++++++++ template/test/terraform/main.tf | 8 +++ .../ansible/playbooks/poststart/0-example.yml | 3 + .../ansible/playbooks/prestart/0-example.yml | 3 + .../conf/hashistack/consul/99-override.hcl | 1 + .../conf/hashistack/nomad/99-override.hcl | 1 + .../conf/hashistack/vault/99-override.hcl | 1 + 16 files changed, 193 insertions(+) create mode 100644 template/.env create mode 100644 template/.github/workflows/on_pr_push_master.yml create mode 100644 template/Makefile create mode 100644 template/README.md create mode 100644 template/Vagrantfile create mode 100644 template/test/.env create mode 100644 template/test/Makefile create mode 100644 template/test/Vagrantfile create mode 100644 template/test/ansible/playbook.yml create mode 100644 template/test/nomad/countdash.hcl create mode 100644 template/test/terraform/main.tf create mode 100644 template/vagrant/conf/ansible/playbooks/poststart/0-example.yml create mode 100644 template/vagrant/conf/ansible/playbooks/prestart/0-example.yml create mode 100644 template/vagrant/conf/hashistack/consul/99-override.hcl create mode 100644 template/vagrant/conf/hashistack/nomad/99-override.hcl create mode 100644 template/vagrant/conf/hashistack/vault/99-override.hcl diff --git a/template/.env b/template/.env new file mode 100644 index 0000000..975bbb5 --- /dev/null +++ b/template/.env @@ -0,0 +1 @@ +#FOO=BAR \ No newline at end of file diff --git a/template/.github/workflows/on_pr_push_master.yml b/template/.github/workflows/on_pr_push_master.yml new file mode 100644 index 0000000..45148b3 --- /dev/null +++ b/template/.github/workflows/on_pr_push_master.yml @@ -0,0 +1,14 @@ +name: CI + +on: + pull_request: + branches: [ master ] + push: + branches: [ master ] +jobs: + test: + runs-on: macos-latest + steps: + - uses: actions/checkout@v2 + - name: Run vagrant box with tests + run: make test \ No newline at end of file diff --git a/template/Makefile b/template/Makefile new file mode 100644 index 0000000..b5919aa --- /dev/null +++ b/template/Makefile @@ -0,0 +1,29 @@ +include .env +export +export PATH := $(shell pwd)/tmp:$(PATH) + +.ONESHELL .PHONY: up update-box destroy-box remove-tmp clean copy-consul test +.DEFAULT_GOAL := up + +#### Development #### +# start commands +up: clean update-box + SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant up --provision + +update-box: + @SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant box update || (echo '\n\nIf you get an SSL error you might be behind a transparent proxy. \nMore info https://github.com/fredrikhgrelland/vagrant-hashistack/blob/master/README.md#if-you-are-behind-a-transparent-proxy\n\n' && exit 2) + +# clean commands +destroy-box: + vagrant destroy -f + +remove-tmp: + rm -rf ./tmp + +clean: destroy-box remove-tmp + +copy-consul: + if [ ! -f "./tmp/consul" ]; then mkdir -p ./tmp; vagrant ssh -c "cp /usr/local/bin/consul /vagrant/tmp/consul"; fi; + +test: + $(MAKE) -C test diff --git a/template/README.md b/template/README.md new file mode 100644 index 0000000..1a31ef1 --- /dev/null +++ b/template/README.md @@ -0,0 +1,26 @@ +# Starter template for `fredrikhgrelland/hashistack` + +This repository can be used as a base for developing services on the hashistack. +On github, you may use the ["Use this template"](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to generate a new repository from this template. + +If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in this separate repository [vagrant-hashistack-template-dev](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to start a new repository from this repo +. + +Documentation on [parent repository](https://github.com/fredrikhgrelland/vagrant-hashistack#usage). + +## Customizing the vagrant box +The vagrant box ships with a default startup scheme. It will run an ansible playbook to start all services. +You may change the hashistack configuration or add aditional pre and post steps to the startup procedure to match your needs. + +### Overriding and extending the configuration of the hashistack + +- consul [vagrant/conf/hashistack/consul/99-override.hcl](vagrant/conf/hashistack/consul/99-override.hcl) +- nomad [vagrant/conf/hashistack/nomad/99-override.hcl](vagrant/conf/hashistack/nomad/99-override.hcl) +- vault [vagrant/conf/hashistack/vault/99-override.hcl](vagrant/conf/hashistack/vault/99-override.hcl) + +You may edit the `99-override.hcl` or add your own. +Any valid configuration added to these directories will be added to their respective services' configuration, in lexical order. + +### Pre- and post-startup ansible playbooks +This vagrant box will execute ansible playbooks put in two special directories [vagrant/conf/ansible/playbooks/prestart](vagrant/conf/ansible/playbooks/prestart) and [vagrant/conf/ansible/playbooks/poststart](vagrant/conf/ansible/playbooks/poststart). These playbooks will be executed before and after the box's bundled startup sequence, respectively. This gives the flexibility to configure all aspects of the hashistack as well as run tasks needed for tests or demo purposes as part of `vagrant up` Note; The playbooks are included into the main run, so the syntax in the [example](vagrant/conf/ansible/playbooks/prestart/0-example.yml) must be followed.. +They will be run in lexical order, and prefixing with numbers is a good way to get the order you want. diff --git a/template/Vagrantfile b/template/Vagrantfile new file mode 100644 index 0000000..a6e7528 --- /dev/null +++ b/template/Vagrantfile @@ -0,0 +1,8 @@ +Vagrant.configure("2") do |config| + config.vm.box = "fredrikhgrelland/hashistack" + config.vm.box_version = "~> 0.2" + config.vm.provider "virtualbox" do |vb| + vb.linked_clone = true + vb.memory = 2048 + end +end diff --git a/template/test/.env b/template/test/.env new file mode 100644 index 0000000..272cdce --- /dev/null +++ b/template/test/.env @@ -0,0 +1,2 @@ +#NOMAD_TOKEN=NOT_IN_USE +#VAULT_TOKEN=NOT_IN_USE \ No newline at end of file diff --git a/template/test/Makefile b/template/test/Makefile new file mode 100644 index 0000000..e2490de --- /dev/null +++ b/template/test/Makefile @@ -0,0 +1,15 @@ +include .env +export + +.ONESHELL .PHONY: test clean up +.DEFAULT_GOAL := test + +up: + vagrant up --provision + $(MAKE) clean + +clean: + vagrant destroy -f + rm -rf .vagrant + +test: clean up diff --git a/template/test/Vagrantfile b/template/test/Vagrantfile new file mode 100644 index 0000000..6cc19e8 --- /dev/null +++ b/template/test/Vagrantfile @@ -0,0 +1,7 @@ +Vagrant.configure("2") do |config| + config.vm.box = "fredrikhgrelland/hashistack" + config.vm.provider "virtualbox" do |vb| + vb.linked_clone = true + vb.memory = 2048 + end +end diff --git a/template/test/ansible/playbook.yml b/template/test/ansible/playbook.yml new file mode 100644 index 0000000..2d8ff68 --- /dev/null +++ b/template/test/ansible/playbook.yml @@ -0,0 +1,14 @@ +--- +- hosts: all + become: yes + tasks: + - name: Terraform + terraform: + project_path: ../terraform + force_init: true + state: present + register: terraform + + - name: Terraform stdout + debug: + msg: "{{terraform.stdout}}" \ No newline at end of file diff --git a/template/test/nomad/countdash.hcl b/template/test/nomad/countdash.hcl new file mode 100644 index 0000000..9314ddf --- /dev/null +++ b/template/test/nomad/countdash.hcl @@ -0,0 +1,60 @@ +job "countdash" { + datacenters = ["dc1"] + group "api" { + network { + mode = "bridge" + } + + service { + name = "count-api" + port = "9001" + + connect { + sidecar_service {} + } + } + + task "web" { + driver = "docker" + config { + image = "hashicorpnomad/counter-api:v1" + } + } + } + + group "dashboard" { + network { + mode ="bridge" + port "http" { + static = 9002 + to = 9002 + } + } + + service { + name = "count-dashboard" + port = "9002" + + connect { + sidecar_service { + proxy { + upstreams { + destination_name = "count-api" + local_bind_port = 8080 + } + } + } + } + } + + task "dashboard" { + driver = "docker" + env { + COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}" + } + config { + image = "hashicorpnomad/counter-dashboard:v1" + } + } + } +} diff --git a/template/test/terraform/main.tf b/template/test/terraform/main.tf new file mode 100644 index 0000000..f09ba72 --- /dev/null +++ b/template/test/terraform/main.tf @@ -0,0 +1,8 @@ +provider "nomad" { + address = "http://127.0.0.1:4646" +} + +resource "nomad_job" "countdash" { + jobspec = file("${path.cwd}/../nomad/countdash.hcl") + detach = false +} \ No newline at end of file diff --git a/template/vagrant/conf/ansible/playbooks/poststart/0-example.yml b/template/vagrant/conf/ansible/playbooks/poststart/0-example.yml new file mode 100644 index 0000000..0709769 --- /dev/null +++ b/template/vagrant/conf/ansible/playbooks/poststart/0-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of poststart + debug: + msg: This would be a poststart task diff --git a/template/vagrant/conf/ansible/playbooks/prestart/0-example.yml b/template/vagrant/conf/ansible/playbooks/prestart/0-example.yml new file mode 100644 index 0000000..cc03e7f --- /dev/null +++ b/template/vagrant/conf/ansible/playbooks/prestart/0-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of prestart + debug: + msg: This would be a prestart task diff --git a/template/vagrant/conf/hashistack/consul/99-override.hcl b/template/vagrant/conf/hashistack/consul/99-override.hcl new file mode 100644 index 0000000..d2b8041 --- /dev/null +++ b/template/vagrant/conf/hashistack/consul/99-override.hcl @@ -0,0 +1 @@ +#Any vaild configuration from https://www.consul.io/docs/agent/options.html#configuration_files \ No newline at end of file diff --git a/template/vagrant/conf/hashistack/nomad/99-override.hcl b/template/vagrant/conf/hashistack/nomad/99-override.hcl new file mode 100644 index 0000000..d817340 --- /dev/null +++ b/template/vagrant/conf/hashistack/nomad/99-override.hcl @@ -0,0 +1 @@ +#Any vaild configuration from https://www.nomadproject.io/docs/configuration#general-parameters \ No newline at end of file diff --git a/template/vagrant/conf/hashistack/vault/99-override.hcl b/template/vagrant/conf/hashistack/vault/99-override.hcl new file mode 100644 index 0000000..2a828c6 --- /dev/null +++ b/template/vagrant/conf/hashistack/vault/99-override.hcl @@ -0,0 +1 @@ +#Any vaild configuration from https://www.vaultproject.io/docs/configuration \ No newline at end of file From 2db61b64e8c4a6c9ca5db98db6caab8e4fa2ebc2 Mon Sep 17 00:00:00 2001 From: fredrikhgrelland <> Date: Fri, 3 Jul 2020 07:50:44 +0000 Subject: [PATCH 05/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/19cbba14643af7c33bb034bb43feeaadf436ff6c) --- Makefile | 29 --------- README.md | 26 -------- Vagrantfile | 8 --- template/README.md | 31 +++++++++- template/ansible/README.md | 4 ++ {test => template}/ansible/playbook.yml | 0 template/nomad/README.md | 3 + template/nomad/your_nomad_job.hcl | 1 + template/terraform/README.md | 8 +++ template/terraform/main.tf | 9 +++ template/test/Makefile | 3 + .../ansible/playbooks/poststart/0-example.yml | 0 .../ansible/playbooks/prestart/0-example.yml | 2 +- .../conf/hashistack/consul/99-override.hcl | 0 .../conf/hashistack/nomad/99-override.hcl | 0 .../conf/hashistack/vault/99-override.hcl | 0 .../conf/ansible/playbooks/prestart/1-acl.yml | 3 + test/.env | 2 - test/Makefile | 15 ----- test/Vagrantfile | 7 --- test/nomad/countdash.hcl | 60 ------------------- test/terraform/main.tf | 8 --- 22 files changed, 62 insertions(+), 157 deletions(-) delete mode 100644 Makefile delete mode 100644 README.md delete mode 100644 Vagrantfile create mode 100644 template/ansible/README.md rename {test => template}/ansible/playbook.yml (100%) create mode 100644 template/nomad/README.md create mode 100644 template/nomad/your_nomad_job.hcl create mode 100644 template/terraform/README.md create mode 100644 template/terraform/main.tf rename {vagrant => template/test/vagrant}/conf/ansible/playbooks/poststart/0-example.yml (100%) rename {vagrant => template/test/vagrant}/conf/ansible/playbooks/prestart/0-example.yml (56%) rename {vagrant => template/test/vagrant}/conf/hashistack/consul/99-override.hcl (100%) rename {vagrant => template/test/vagrant}/conf/hashistack/nomad/99-override.hcl (100%) rename {vagrant => template/test/vagrant}/conf/hashistack/vault/99-override.hcl (100%) create mode 100644 template/vagrant/conf/ansible/playbooks/prestart/1-acl.yml delete mode 100644 test/.env delete mode 100644 test/Makefile delete mode 100644 test/Vagrantfile delete mode 100644 test/nomad/countdash.hcl delete mode 100644 test/terraform/main.tf diff --git a/Makefile b/Makefile deleted file mode 100644 index b5919aa..0000000 --- a/Makefile +++ /dev/null @@ -1,29 +0,0 @@ -include .env -export -export PATH := $(shell pwd)/tmp:$(PATH) - -.ONESHELL .PHONY: up update-box destroy-box remove-tmp clean copy-consul test -.DEFAULT_GOAL := up - -#### Development #### -# start commands -up: clean update-box - SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant up --provision - -update-box: - @SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant box update || (echo '\n\nIf you get an SSL error you might be behind a transparent proxy. \nMore info https://github.com/fredrikhgrelland/vagrant-hashistack/blob/master/README.md#if-you-are-behind-a-transparent-proxy\n\n' && exit 2) - -# clean commands -destroy-box: - vagrant destroy -f - -remove-tmp: - rm -rf ./tmp - -clean: destroy-box remove-tmp - -copy-consul: - if [ ! -f "./tmp/consul" ]; then mkdir -p ./tmp; vagrant ssh -c "cp /usr/local/bin/consul /vagrant/tmp/consul"; fi; - -test: - $(MAKE) -C test diff --git a/README.md b/README.md deleted file mode 100644 index 1a31ef1..0000000 --- a/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Starter template for `fredrikhgrelland/hashistack` - -This repository can be used as a base for developing services on the hashistack. -On github, you may use the ["Use this template"](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to generate a new repository from this template. - -If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in this separate repository [vagrant-hashistack-template-dev](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to start a new repository from this repo -. - -Documentation on [parent repository](https://github.com/fredrikhgrelland/vagrant-hashistack#usage). - -## Customizing the vagrant box -The vagrant box ships with a default startup scheme. It will run an ansible playbook to start all services. -You may change the hashistack configuration or add aditional pre and post steps to the startup procedure to match your needs. - -### Overriding and extending the configuration of the hashistack - -- consul [vagrant/conf/hashistack/consul/99-override.hcl](vagrant/conf/hashistack/consul/99-override.hcl) -- nomad [vagrant/conf/hashistack/nomad/99-override.hcl](vagrant/conf/hashistack/nomad/99-override.hcl) -- vault [vagrant/conf/hashistack/vault/99-override.hcl](vagrant/conf/hashistack/vault/99-override.hcl) - -You may edit the `99-override.hcl` or add your own. -Any valid configuration added to these directories will be added to their respective services' configuration, in lexical order. - -### Pre- and post-startup ansible playbooks -This vagrant box will execute ansible playbooks put in two special directories [vagrant/conf/ansible/playbooks/prestart](vagrant/conf/ansible/playbooks/prestart) and [vagrant/conf/ansible/playbooks/poststart](vagrant/conf/ansible/playbooks/poststart). These playbooks will be executed before and after the box's bundled startup sequence, respectively. This gives the flexibility to configure all aspects of the hashistack as well as run tasks needed for tests or demo purposes as part of `vagrant up` Note; The playbooks are included into the main run, so the syntax in the [example](vagrant/conf/ansible/playbooks/prestart/0-example.yml) must be followed.. -They will be run in lexical order, and prefixing with numbers is a good way to get the order you want. diff --git a/Vagrantfile b/Vagrantfile deleted file mode 100644 index a6e7528..0000000 --- a/Vagrantfile +++ /dev/null @@ -1,8 +0,0 @@ -Vagrant.configure("2") do |config| - config.vm.box = "fredrikhgrelland/hashistack" - config.vm.box_version = "~> 0.2" - config.vm.provider "virtualbox" do |vb| - vb.linked_clone = true - vb.memory = 2048 - end -end diff --git a/template/README.md b/template/README.md index 1a31ef1..370c4be 100644 --- a/template/README.md +++ b/template/README.md @@ -3,7 +3,7 @@ This repository can be used as a base for developing services on the hashistack. On github, you may use the ["Use this template"](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to generate a new repository from this template. -If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in this separate repository [vagrant-hashistack-template-dev](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to start a new repository from this repo +If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in this separate repository [vagrant-hashistack-template](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to start a new repository from this repo . Documentation on [parent repository](https://github.com/fredrikhgrelland/vagrant-hashistack#usage). @@ -21,6 +21,35 @@ You may change the hashistack configuration or add aditional pre and post steps You may edit the `99-override.hcl` or add your own. Any valid configuration added to these directories will be added to their respective services' configuration, in lexical order. +#### Config variations +We provide some config variations as additional functionality, which you may want to use. +Current supported variations: +```text +* ACL default policy for consul (deny or allow) +``` +To choose other config variation you need to override ansible variables. There are two options how to override ansible variables: +* First option via `ANSIBLE_AGRS` +```bash +ANSIBLE_ARGS='--extra-vars "consul_agent_acl_default_policy=deny"' vagrant up --provision +``` +* Second option via prestart, adding fact in `vagrant/conf/ansible/playbooks/prestart/0-example.yml` +```yaml +# Option 2 how to turn on ACL +# [ACL] Set 'deny' default policy for consul agent +- set_fact: consul_agent_acl_default_policy="deny" +``` + +Full set of extra_variables could be found [default_vars.yml](../ansible/default_vars.yml) + ### Pre- and post-startup ansible playbooks This vagrant box will execute ansible playbooks put in two special directories [vagrant/conf/ansible/playbooks/prestart](vagrant/conf/ansible/playbooks/prestart) and [vagrant/conf/ansible/playbooks/poststart](vagrant/conf/ansible/playbooks/poststart). These playbooks will be executed before and after the box's bundled startup sequence, respectively. This gives the flexibility to configure all aspects of the hashistack as well as run tasks needed for tests or demo purposes as part of `vagrant up` Note; The playbooks are included into the main run, so the syntax in the [example](vagrant/conf/ansible/playbooks/prestart/0-example.yml) must be followed.. They will be run in lexical order, and prefixing with numbers is a good way to get the order you want. + +### Vagrant box life-cycle +![img](../docs/life-cycle.png) +1. `install.yml` - installing required software +2. `default_vars.yml` - setup default ansible variables +3. `prestart/*.yml` - prestart scripts, running before hashistack software will start +4. `bootstrap.yml` - verify ansible variables and software configuration, run hashistack software & verify that it started correctly +5. `poststart/*.yml` - poststart scripts, running after hasistack software runs and ready +6. `ansible/playbook.yml` - user's provisioning playbook diff --git a/template/ansible/README.md b/template/ansible/README.md new file mode 100644 index 0000000..8c4f715 --- /dev/null +++ b/template/ansible/README.md @@ -0,0 +1,4 @@ +# Ansible + +This is a good starting point where you may run a series of sequential steps. +In the example, we use ansible to initialize and start a terraform job which in turn starts nomad jobs in parallel. \ No newline at end of file diff --git a/test/ansible/playbook.yml b/template/ansible/playbook.yml similarity index 100% rename from test/ansible/playbook.yml rename to template/ansible/playbook.yml diff --git a/template/nomad/README.md b/template/nomad/README.md new file mode 100644 index 0000000..7162afa --- /dev/null +++ b/template/nomad/README.md @@ -0,0 +1,3 @@ +# Nomad + +Put your nomad jobs in this directory and reference it in [../terraform](../terraform) code. \ No newline at end of file diff --git a/template/nomad/your_nomad_job.hcl b/template/nomad/your_nomad_job.hcl new file mode 100644 index 0000000..aede7f0 --- /dev/null +++ b/template/nomad/your_nomad_job.hcl @@ -0,0 +1 @@ +#Placeholder \ No newline at end of file diff --git a/template/terraform/README.md b/template/terraform/README.md new file mode 100644 index 0000000..edd3d0e --- /dev/null +++ b/template/terraform/README.md @@ -0,0 +1,8 @@ +# Terraform + +Any configuration of the hashistack is best performed through terraform providers. +- [nomad](https://www.terraform.io/docs/providers/nomad/index.html) +- [vault](https://www.terraform.io/docs/providers/vault/index.html) +- [consul](https://www.terraform.io/docs/providers/consul/index.html) + +You put them in this directory and use [../ansible/playbook.yml](../ansible/playbook.yml) to init and run terraform. \ No newline at end of file diff --git a/template/terraform/main.tf b/template/terraform/main.tf new file mode 100644 index 0000000..e16257c --- /dev/null +++ b/template/terraform/main.tf @@ -0,0 +1,9 @@ +provider "nomad" { + address = "http://127.0.0.1:4646" +} +/* +resource "nomad_job" "your_nomad_job" { + jobspec = file("${path.cwd}/../nomad/your_nomad_job.hcl") + detach = false +} +*/ \ No newline at end of file diff --git a/template/test/Makefile b/template/test/Makefile index e2490de..82abd63 100644 --- a/template/test/Makefile +++ b/template/test/Makefile @@ -7,6 +7,9 @@ export up: vagrant up --provision $(MAKE) clean + # Option 1 how to turn on ACL + ANSIBLE_ARGS='--extra-vars "consul_agent_acl_default_policy=deny"' vagrant up --provision + $(MAKE) clean clean: vagrant destroy -f diff --git a/vagrant/conf/ansible/playbooks/poststart/0-example.yml b/template/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml similarity index 100% rename from vagrant/conf/ansible/playbooks/poststart/0-example.yml rename to template/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml diff --git a/vagrant/conf/ansible/playbooks/prestart/0-example.yml b/template/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml similarity index 56% rename from vagrant/conf/ansible/playbooks/prestart/0-example.yml rename to template/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml index cc03e7f..52f8e81 100644 --- a/vagrant/conf/ansible/playbooks/prestart/0-example.yml +++ b/template/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml @@ -1,3 +1,3 @@ - name: Task that shows usage of prestart debug: - msg: This would be a prestart task + msg: This would be a prestart task \ No newline at end of file diff --git a/vagrant/conf/hashistack/consul/99-override.hcl b/template/test/vagrant/conf/hashistack/consul/99-override.hcl similarity index 100% rename from vagrant/conf/hashistack/consul/99-override.hcl rename to template/test/vagrant/conf/hashistack/consul/99-override.hcl diff --git a/vagrant/conf/hashistack/nomad/99-override.hcl b/template/test/vagrant/conf/hashistack/nomad/99-override.hcl similarity index 100% rename from vagrant/conf/hashistack/nomad/99-override.hcl rename to template/test/vagrant/conf/hashistack/nomad/99-override.hcl diff --git a/vagrant/conf/hashistack/vault/99-override.hcl b/template/test/vagrant/conf/hashistack/vault/99-override.hcl similarity index 100% rename from vagrant/conf/hashistack/vault/99-override.hcl rename to template/test/vagrant/conf/hashistack/vault/99-override.hcl diff --git a/template/vagrant/conf/ansible/playbooks/prestart/1-acl.yml b/template/vagrant/conf/ansible/playbooks/prestart/1-acl.yml new file mode 100644 index 0000000..ab73d9e --- /dev/null +++ b/template/vagrant/conf/ansible/playbooks/prestart/1-acl.yml @@ -0,0 +1,3 @@ +## Option 2 how to turn on ACL +## [ACL] Set 'deny' default policy for consul agent +#- set_fact: consul_agent_acl_default_policy="deny" \ No newline at end of file diff --git a/test/.env b/test/.env deleted file mode 100644 index 272cdce..0000000 --- a/test/.env +++ /dev/null @@ -1,2 +0,0 @@ -#NOMAD_TOKEN=NOT_IN_USE -#VAULT_TOKEN=NOT_IN_USE \ No newline at end of file diff --git a/test/Makefile b/test/Makefile deleted file mode 100644 index e2490de..0000000 --- a/test/Makefile +++ /dev/null @@ -1,15 +0,0 @@ -include .env -export - -.ONESHELL .PHONY: test clean up -.DEFAULT_GOAL := test - -up: - vagrant up --provision - $(MAKE) clean - -clean: - vagrant destroy -f - rm -rf .vagrant - -test: clean up diff --git a/test/Vagrantfile b/test/Vagrantfile deleted file mode 100644 index 6cc19e8..0000000 --- a/test/Vagrantfile +++ /dev/null @@ -1,7 +0,0 @@ -Vagrant.configure("2") do |config| - config.vm.box = "fredrikhgrelland/hashistack" - config.vm.provider "virtualbox" do |vb| - vb.linked_clone = true - vb.memory = 2048 - end -end diff --git a/test/nomad/countdash.hcl b/test/nomad/countdash.hcl deleted file mode 100644 index 9314ddf..0000000 --- a/test/nomad/countdash.hcl +++ /dev/null @@ -1,60 +0,0 @@ -job "countdash" { - datacenters = ["dc1"] - group "api" { - network { - mode = "bridge" - } - - service { - name = "count-api" - port = "9001" - - connect { - sidecar_service {} - } - } - - task "web" { - driver = "docker" - config { - image = "hashicorpnomad/counter-api:v1" - } - } - } - - group "dashboard" { - network { - mode ="bridge" - port "http" { - static = 9002 - to = 9002 - } - } - - service { - name = "count-dashboard" - port = "9002" - - connect { - sidecar_service { - proxy { - upstreams { - destination_name = "count-api" - local_bind_port = 8080 - } - } - } - } - } - - task "dashboard" { - driver = "docker" - env { - COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}" - } - config { - image = "hashicorpnomad/counter-dashboard:v1" - } - } - } -} diff --git a/test/terraform/main.tf b/test/terraform/main.tf deleted file mode 100644 index f09ba72..0000000 --- a/test/terraform/main.tf +++ /dev/null @@ -1,8 +0,0 @@ -provider "nomad" { - address = "http://127.0.0.1:4646" -} - -resource "nomad_job" "countdash" { - jobspec = file("${path.cwd}/../nomad/countdash.hcl") - detach = false -} \ No newline at end of file From d08bde6ef1443d3c53f28f60d787ea8d17419289 Mon Sep 17 00:00:00 2001 From: fredrikhgrelland <> Date: Fri, 3 Jul 2020 08:23:48 +0000 Subject: [PATCH 06/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/19cbba14643af7c33bb034bb43feeaadf436ff6c) --- template/.env => .env | 0 {template/.github => .github}/workflows/on_pr_push_master.yml | 0 template/Makefile => Makefile | 0 template/README.md => README.md | 0 template/Vagrantfile => Vagrantfile | 0 {template/ansible => ansible}/README.md | 0 {template/ansible => ansible}/playbook.yml | 0 {template/nomad => nomad}/README.md | 0 {template/nomad => nomad}/your_nomad_job.hcl | 0 {template/terraform => terraform}/README.md | 0 {template/terraform => terraform}/main.tf | 0 {template/test => test}/.env | 0 {template/test => test}/Makefile | 0 {template/test => test}/Vagrantfile | 0 {template/test => test}/ansible/playbook.yml | 0 {template/test => test}/nomad/countdash.hcl | 0 {template/test => test}/terraform/main.tf | 0 .../vagrant/conf/ansible/playbooks/poststart/0-example.yml | 0 .../vagrant/conf/ansible/playbooks/prestart/0-example.yml | 0 .../test => test}/vagrant/conf/hashistack/consul/99-override.hcl | 0 .../test => test}/vagrant/conf/hashistack/nomad/99-override.hcl | 0 .../test => test}/vagrant/conf/hashistack/vault/99-override.hcl | 0 .../conf/ansible/playbooks/poststart/0-example.yml | 0 .../conf/ansible/playbooks/prestart/0-example.yml | 0 .../vagrant => vagrant}/conf/ansible/playbooks/prestart/1-acl.yml | 0 .../vagrant => vagrant}/conf/hashistack/consul/99-override.hcl | 0 .../vagrant => vagrant}/conf/hashistack/nomad/99-override.hcl | 0 .../vagrant => vagrant}/conf/hashistack/vault/99-override.hcl | 0 28 files changed, 0 insertions(+), 0 deletions(-) rename template/.env => .env (100%) rename {template/.github => .github}/workflows/on_pr_push_master.yml (100%) rename template/Makefile => Makefile (100%) rename template/README.md => README.md (100%) rename template/Vagrantfile => Vagrantfile (100%) rename {template/ansible => ansible}/README.md (100%) rename {template/ansible => ansible}/playbook.yml (100%) rename {template/nomad => nomad}/README.md (100%) rename {template/nomad => nomad}/your_nomad_job.hcl (100%) rename {template/terraform => terraform}/README.md (100%) rename {template/terraform => terraform}/main.tf (100%) rename {template/test => test}/.env (100%) rename {template/test => test}/Makefile (100%) rename {template/test => test}/Vagrantfile (100%) rename {template/test => test}/ansible/playbook.yml (100%) rename {template/test => test}/nomad/countdash.hcl (100%) rename {template/test => test}/terraform/main.tf (100%) rename {template/test => test}/vagrant/conf/ansible/playbooks/poststart/0-example.yml (100%) rename {template/test => test}/vagrant/conf/ansible/playbooks/prestart/0-example.yml (100%) rename {template/test => test}/vagrant/conf/hashistack/consul/99-override.hcl (100%) rename {template/test => test}/vagrant/conf/hashistack/nomad/99-override.hcl (100%) rename {template/test => test}/vagrant/conf/hashistack/vault/99-override.hcl (100%) rename {template/vagrant => vagrant}/conf/ansible/playbooks/poststart/0-example.yml (100%) rename {template/vagrant => vagrant}/conf/ansible/playbooks/prestart/0-example.yml (100%) rename {template/vagrant => vagrant}/conf/ansible/playbooks/prestart/1-acl.yml (100%) rename {template/vagrant => vagrant}/conf/hashistack/consul/99-override.hcl (100%) rename {template/vagrant => vagrant}/conf/hashistack/nomad/99-override.hcl (100%) rename {template/vagrant => vagrant}/conf/hashistack/vault/99-override.hcl (100%) diff --git a/template/.env b/.env similarity index 100% rename from template/.env rename to .env diff --git a/template/.github/workflows/on_pr_push_master.yml b/.github/workflows/on_pr_push_master.yml similarity index 100% rename from template/.github/workflows/on_pr_push_master.yml rename to .github/workflows/on_pr_push_master.yml diff --git a/template/Makefile b/Makefile similarity index 100% rename from template/Makefile rename to Makefile diff --git a/template/README.md b/README.md similarity index 100% rename from template/README.md rename to README.md diff --git a/template/Vagrantfile b/Vagrantfile similarity index 100% rename from template/Vagrantfile rename to Vagrantfile diff --git a/template/ansible/README.md b/ansible/README.md similarity index 100% rename from template/ansible/README.md rename to ansible/README.md diff --git a/template/ansible/playbook.yml b/ansible/playbook.yml similarity index 100% rename from template/ansible/playbook.yml rename to ansible/playbook.yml diff --git a/template/nomad/README.md b/nomad/README.md similarity index 100% rename from template/nomad/README.md rename to nomad/README.md diff --git a/template/nomad/your_nomad_job.hcl b/nomad/your_nomad_job.hcl similarity index 100% rename from template/nomad/your_nomad_job.hcl rename to nomad/your_nomad_job.hcl diff --git a/template/terraform/README.md b/terraform/README.md similarity index 100% rename from template/terraform/README.md rename to terraform/README.md diff --git a/template/terraform/main.tf b/terraform/main.tf similarity index 100% rename from template/terraform/main.tf rename to terraform/main.tf diff --git a/template/test/.env b/test/.env similarity index 100% rename from template/test/.env rename to test/.env diff --git a/template/test/Makefile b/test/Makefile similarity index 100% rename from template/test/Makefile rename to test/Makefile diff --git a/template/test/Vagrantfile b/test/Vagrantfile similarity index 100% rename from template/test/Vagrantfile rename to test/Vagrantfile diff --git a/template/test/ansible/playbook.yml b/test/ansible/playbook.yml similarity index 100% rename from template/test/ansible/playbook.yml rename to test/ansible/playbook.yml diff --git a/template/test/nomad/countdash.hcl b/test/nomad/countdash.hcl similarity index 100% rename from template/test/nomad/countdash.hcl rename to test/nomad/countdash.hcl diff --git a/template/test/terraform/main.tf b/test/terraform/main.tf similarity index 100% rename from template/test/terraform/main.tf rename to test/terraform/main.tf diff --git a/template/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml b/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml similarity index 100% rename from template/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml rename to test/vagrant/conf/ansible/playbooks/poststart/0-example.yml diff --git a/template/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml b/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml similarity index 100% rename from template/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml rename to test/vagrant/conf/ansible/playbooks/prestart/0-example.yml diff --git a/template/test/vagrant/conf/hashistack/consul/99-override.hcl b/test/vagrant/conf/hashistack/consul/99-override.hcl similarity index 100% rename from template/test/vagrant/conf/hashistack/consul/99-override.hcl rename to test/vagrant/conf/hashistack/consul/99-override.hcl diff --git a/template/test/vagrant/conf/hashistack/nomad/99-override.hcl b/test/vagrant/conf/hashistack/nomad/99-override.hcl similarity index 100% rename from template/test/vagrant/conf/hashistack/nomad/99-override.hcl rename to test/vagrant/conf/hashistack/nomad/99-override.hcl diff --git a/template/test/vagrant/conf/hashistack/vault/99-override.hcl b/test/vagrant/conf/hashistack/vault/99-override.hcl similarity index 100% rename from template/test/vagrant/conf/hashistack/vault/99-override.hcl rename to test/vagrant/conf/hashistack/vault/99-override.hcl diff --git a/template/vagrant/conf/ansible/playbooks/poststart/0-example.yml b/vagrant/conf/ansible/playbooks/poststart/0-example.yml similarity index 100% rename from template/vagrant/conf/ansible/playbooks/poststart/0-example.yml rename to vagrant/conf/ansible/playbooks/poststart/0-example.yml diff --git a/template/vagrant/conf/ansible/playbooks/prestart/0-example.yml b/vagrant/conf/ansible/playbooks/prestart/0-example.yml similarity index 100% rename from template/vagrant/conf/ansible/playbooks/prestart/0-example.yml rename to vagrant/conf/ansible/playbooks/prestart/0-example.yml diff --git a/template/vagrant/conf/ansible/playbooks/prestart/1-acl.yml b/vagrant/conf/ansible/playbooks/prestart/1-acl.yml similarity index 100% rename from template/vagrant/conf/ansible/playbooks/prestart/1-acl.yml rename to vagrant/conf/ansible/playbooks/prestart/1-acl.yml diff --git a/template/vagrant/conf/hashistack/consul/99-override.hcl b/vagrant/conf/hashistack/consul/99-override.hcl similarity index 100% rename from template/vagrant/conf/hashistack/consul/99-override.hcl rename to vagrant/conf/hashistack/consul/99-override.hcl diff --git a/template/vagrant/conf/hashistack/nomad/99-override.hcl b/vagrant/conf/hashistack/nomad/99-override.hcl similarity index 100% rename from template/vagrant/conf/hashistack/nomad/99-override.hcl rename to vagrant/conf/hashistack/nomad/99-override.hcl diff --git a/template/vagrant/conf/hashistack/vault/99-override.hcl b/vagrant/conf/hashistack/vault/99-override.hcl similarity index 100% rename from template/vagrant/conf/hashistack/vault/99-override.hcl rename to vagrant/conf/hashistack/vault/99-override.hcl From 560bbef50780b41e040680a22fe4f113022d3081 Mon Sep 17 00:00:00 2001 From: fredrikhgrelland <> Date: Mon, 17 Aug 2020 15:06:27 +0000 Subject: [PATCH 07/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/f72fa86322b6c50b0e2b02df33e879b92a71610b) --- .env | 1 - .github/action/create-env.py | 15 +++ .github/linters/.markdown-lint.yml | 37 ++++++ .github/workflows/on_pr_push_master.yml | 107 +++++++++++++++++- .gitignore | 8 ++ Makefile | 36 ++++-- README.md | 107 ++++++++++++------ Vagrantfile | 6 +- ansible/playbook.yml | 14 --- conf/nomad/README.md | 3 + dev/.env | 1 + dev/README.md | 3 + {ansible => dev/ansible}/README.md | 4 +- dev/ansible/playbook.yml | 7 ++ dev/vagrant/conf/README.md | 31 +++++ dev/vagrant/conf/consul/README.md | 8 ++ dev/vagrant/conf/nomad/README.md | 19 ++++ dev/vagrant/conf/post_bootstrap/README.md | 1 + dev/vagrant/conf/pre_bootstrap/README.md | 15 +++ dev/vagrant/conf/vault/README.md | 9 ++ docker/.dockerignore | 4 + docker/README.md | 16 +++ docker/conf/certificates/.gitignore | 1 + docker/conf/certificates/README.md | 2 + example/README.md | 3 + main.tf | 0 nomad/README.md | 3 - nomad/your_nomad_job.hcl | 1 - outputs.tf | 0 terraform/README.md | 8 -- terraform/main.tf | 9 -- test/.env | 2 - test/Makefile | 18 --- test/Vagrantfile | 7 -- test/ansible/playbook.yml | 14 --- test/terraform/main.tf | 8 -- .../ansible/playbooks/poststart/0-example.yml | 3 - .../ansible/playbooks/prestart/0-example.yml | 3 - test_example/.env | 4 + test_example/Vagrantfile | 11 ++ .../conf}/nomad/countdash.hcl | 26 ++++- .../dev/ansible/01_build_docker_image.yml | 32 ++++++ test_example/dev/ansible/02_run_terraform.yml | 10 ++ test_example/dev/ansible/playbook.yml | 7 ++ .../dev/vagrant/conf}/consul/99-override.hcl | 0 .../dev/vagrant/conf}/nomad/99-override.hcl | 0 test_example/dev/vagrant/conf/post_ansible.sh | 3 + .../post_bootstrap/00-poststart-example.yml | 3 + .../post_bootstrap/01-poststart-example.yml | 3 + .../post_bootstrap/02-poststart-example.yml | 3 + test_example/dev/vagrant/conf/pre_ansible.sh | 3 + .../pre_bootstrap/00-prestart-example.yml | 3 + .../pre_bootstrap/01-prestart-example.yml | 3 + .../pre_bootstrap/02-prestart-example.yml | 3 + .../dev/vagrant/conf}/vault/99-override.hcl | 0 test_example/docker/.dockerignore | 4 + test_example/docker/Dockerfile | 56 +++++++++ .../docker/conf/certificates/.gitignore | 1 + .../docker/conf/certificates/README.md | 2 + test_example/example/main.tf | 7 ++ test_example/example/nomad_acl_test.tf | 12 ++ test_example/example/variables.tf | 3 + test_example/main.tf | 4 + test_example/outputs.tf | 3 + test_example/variables.tf | 0 .../ansible/playbooks/poststart/0-example.yml | 3 - .../ansible/playbooks/prestart/0-example.yml | 3 - .../conf/ansible/playbooks/prestart/1-acl.yml | 3 - .../conf/hashistack/consul/99-override.hcl | 1 - vagrant/conf/hashistack/nomad/99-override.hcl | 1 - vagrant/conf/hashistack/vault/99-override.hcl | 1 - variables.tf | 0 72 files changed, 597 insertions(+), 155 deletions(-) delete mode 100644 .env create mode 100644 .github/action/create-env.py create mode 100644 .github/linters/.markdown-lint.yml create mode 100644 .gitignore delete mode 100644 ansible/playbook.yml create mode 100644 conf/nomad/README.md create mode 100644 dev/.env create mode 100644 dev/README.md rename {ansible => dev/ansible}/README.md (61%) create mode 100644 dev/ansible/playbook.yml create mode 100644 dev/vagrant/conf/README.md create mode 100644 dev/vagrant/conf/consul/README.md create mode 100644 dev/vagrant/conf/nomad/README.md create mode 100644 dev/vagrant/conf/post_bootstrap/README.md create mode 100644 dev/vagrant/conf/pre_bootstrap/README.md create mode 100644 dev/vagrant/conf/vault/README.md create mode 100644 docker/.dockerignore create mode 100644 docker/README.md create mode 100644 docker/conf/certificates/.gitignore create mode 100644 docker/conf/certificates/README.md create mode 100644 example/README.md create mode 100644 main.tf delete mode 100644 nomad/README.md delete mode 100644 nomad/your_nomad_job.hcl create mode 100644 outputs.tf delete mode 100644 terraform/README.md delete mode 100644 terraform/main.tf delete mode 100644 test/.env delete mode 100644 test/Makefile delete mode 100644 test/Vagrantfile delete mode 100644 test/ansible/playbook.yml delete mode 100644 test/terraform/main.tf delete mode 100644 test/vagrant/conf/ansible/playbooks/poststart/0-example.yml delete mode 100644 test/vagrant/conf/ansible/playbooks/prestart/0-example.yml create mode 100644 test_example/.env create mode 100644 test_example/Vagrantfile rename {test => test_example/conf}/nomad/countdash.hcl (58%) create mode 100644 test_example/dev/ansible/01_build_docker_image.yml create mode 100644 test_example/dev/ansible/02_run_terraform.yml create mode 100644 test_example/dev/ansible/playbook.yml rename {test/vagrant/conf/hashistack => test_example/dev/vagrant/conf}/consul/99-override.hcl (100%) rename {test/vagrant/conf/hashistack => test_example/dev/vagrant/conf}/nomad/99-override.hcl (100%) create mode 100644 test_example/dev/vagrant/conf/post_ansible.sh create mode 100644 test_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml create mode 100644 test_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml create mode 100644 test_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml create mode 100644 test_example/dev/vagrant/conf/pre_ansible.sh create mode 100644 test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml create mode 100644 test_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml create mode 100644 test_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml rename {test/vagrant/conf/hashistack => test_example/dev/vagrant/conf}/vault/99-override.hcl (100%) create mode 100644 test_example/docker/.dockerignore create mode 100644 test_example/docker/Dockerfile create mode 100644 test_example/docker/conf/certificates/.gitignore create mode 100644 test_example/docker/conf/certificates/README.md create mode 100644 test_example/example/main.tf create mode 100644 test_example/example/nomad_acl_test.tf create mode 100644 test_example/example/variables.tf create mode 100644 test_example/main.tf create mode 100644 test_example/outputs.tf create mode 100644 test_example/variables.tf delete mode 100644 vagrant/conf/ansible/playbooks/poststart/0-example.yml delete mode 100644 vagrant/conf/ansible/playbooks/prestart/0-example.yml delete mode 100644 vagrant/conf/ansible/playbooks/prestart/1-acl.yml delete mode 100644 vagrant/conf/hashistack/consul/99-override.hcl delete mode 100644 vagrant/conf/hashistack/nomad/99-override.hcl delete mode 100644 vagrant/conf/hashistack/vault/99-override.hcl create mode 100644 variables.tf diff --git a/.env b/.env deleted file mode 100644 index 975bbb5..0000000 --- a/.env +++ /dev/null @@ -1 +0,0 @@ -#FOO=BAR \ No newline at end of file diff --git a/.github/action/create-env.py b/.github/action/create-env.py new file mode 100644 index 0000000..56ce9a4 --- /dev/null +++ b/.github/action/create-env.py @@ -0,0 +1,15 @@ +import os + +env_keys = list(dict(os.environ).keys()) + +out_file = "" + +for key in env_keys: + if key.startswith("ENVKEY_"): + out_file += key.split("ENVKEY_")[1] + "=" + os.environ.get(key) + "\n" + +with open( str(os.environ.get("GITHUB_WORKSPACE")) + "/" + str(os.environ.get("FILE_NAME")), "w") as text_file: + text_file.write(out_file) + +with open( str(os.environ.get("GITHUB_WORKSPACE")) + "/" + str(os.environ.get("FILE_NAME")), "r") as text_file: + print(text_file.read()) diff --git a/.github/linters/.markdown-lint.yml b/.github/linters/.markdown-lint.yml new file mode 100644 index 0000000..0e10fac --- /dev/null +++ b/.github/linters/.markdown-lint.yml @@ -0,0 +1,37 @@ +--- +########################### +########################### +## Markdown Linter rules ## +########################### +########################### + +# Linter rules doc: +# - https://github.com/DavidAnson/markdownlint +# +# Note: +# To comment out a single error: +# +# any violations you want +# +# + +############### +# Rules by id # +############### +MD004: false # Unordered list style +MD007: + indent: 2 # Unordered list indentation +MD013: + line_length: 808 # Line length +MD024: + allow_different_nesting: true # Multiple headers with the same content +MD026: + punctuation: ".,;:!。，；:" # List of not allowed +MD029: false # Ordered list item prefix +MD033: false # Allow inline HTML +MD036: false # Emphasis used instead of a heading + +################# +# Rules by tags # +################# +blank_lines: false # Error on blank lines \ No newline at end of file diff --git a/.github/workflows/on_pr_push_master.yml b/.github/workflows/on_pr_push_master.yml index 45148b3..1fd691c 100644 --- a/.github/workflows/on_pr_push_master.yml +++ b/.github/workflows/on_pr_push_master.yml @@ -1,14 +1,113 @@ -name: CI - +name: CI/CD on: pull_request: branches: [ master ] + types: [opened, synchronize, reopened, edited, closed] push: branches: [ master ] + jobs: + linter: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - name: Super-Linter + uses: github/super-linter@latest + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + VALIDATE_ANSIBLE: true + VALIDATE_BASH: true + VALIDATE_DOCKER_HADOLINT: true + VALIDATE_GO: true + VALIDATE_HTML: true + VALIDATE_JAVA: true + VALIDATE_JSON: true + VALIDATE_MD: true + VALIDATE_OPENAPI: true + VALIDATE_PYTHON: true + VALIDATE_PYTHON_PYLINT: true + VALIDATE_PYTHON_FLAKE8: true + VALIDATE_RUBY: true + VALIDATE_SQL: true + VALIDATE_TERRAFORM: true + VALIDATE_XML: true + test: runs-on: macos-latest + + strategy: + matrix: + consul_acl: ["consul_acl_enabled", "consul_acl_disabled"] + consul_acl_default_policy: ["consul_acl_deny"] + nomad_acl: ["nomad_acl_enabled", "nomad_acl_disabled"] + hashicorp_binary: ["hashicorp_oss", "hashicorp_enterprise"] + steps: - uses: actions/checkout@v2 - - name: Run vagrant box with tests - run: make test \ No newline at end of file + + - name: set consul acl (enabled/disabled) + id: consul_acl + run: | + if [ "$test_consul_acl" == 'consul_acl_enabled' ] + then + echo "::set-output name=consul_acl::true" + else + echo "::set-output name=consul_acl::false" + fi + env: + test_consul_acl: ${{ matrix.consul_acl }} + + - name: set consul acl default policy (allow/deny) + id: consul_acl_default_policy + run: | + if [ "$test_consul_acl_default_policy" == 'consul_acl_allow' ] + then + echo "::set-output name=default_policy::allow" + else + echo "::set-output name=default_policy::deny" + fi + env: + test_consul_acl_default_policy: ${{ matrix.consul_acl_default_policy }} + + - name: set nomad acl (enabled/disabled) + id: nomad_acl + run: | + if [ "$test_nomad_acl" == 'nomad_acl_enabled' ] + then + echo "::set-output name=nomad_acl::true" + else + echo "::set-output name=nomad_acl::false" + fi + env: + test_nomad_acl: ${{ matrix.nomad_acl }} + + - name: set hashicorp binary (oss/enterprise) + id: hashicorp_binary + run: | + if [ "$test_hashicorp_binary" == 'hashicorp_oss' ] + then + echo "::set-output name=consul_enterprise::false" + echo "::set-output name=nomad_enterprise::false" + echo "::set-output name=vault_enterprise::false" + else + echo "::set-output name=consul_enterprise::true" + echo "::set-output name=nomad_enterprise::true" + echo "::set-output name=vault_enterprise::true" + fi + env: + test_hashicorp_binary: ${{ matrix.hashicorp_binary }} + + - name: Make .env_override in order to matrix test + run: python .github/action/create-env.py + env: + ENVKEY_env_override: true + ENVKEY_consul_acl: ${{ steps.consul_acl.outputs.consul_acl }} + ENVKEY_consul_acl_default_policy: ${{ steps.consul_acl_default_policy.outputs.default_policy }} + ENVKEY_nomad_acl: ${{ steps.nomad_acl.outputs.nomad_acl }} + ENVKEY_consul_enterprise: ${{ steps.hashicorp_binary.outputs.consul_enterprise }} + ENVKEY_nomad_enterprise: ${{ steps.hashicorp_binary.outputs.nomad_enterprise }} + ENVKEY_vault_enterprise: ${{ steps.hashicorp_binary.outputs.vault_enterprise }} + FILE_NAME: .env_override + + - name: Run make test + run: make test diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3be1e9f --- /dev/null +++ b/.gitignore @@ -0,0 +1,8 @@ +.vagrant +.idea +packer/output-hashistack +terraform.tfstate* +.terraform +.minio.sys +tmp +docker/conf/certificates/*.crt \ No newline at end of file diff --git a/Makefile b/Makefile index b5919aa..261cecf 100644 --- a/Makefile +++ b/Makefile @@ -1,17 +1,36 @@ -include .env +include dev/.env export export PATH := $(shell pwd)/tmp:$(PATH) -.ONESHELL .PHONY: up update-box destroy-box remove-tmp clean copy-consul test +.ONESHELL .PHONY: up update-box destroy-box remove-tmp clean example .DEFAULT_GOAL := up #### Development #### # start commands -up: clean update-box +dev: update-box + SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} ANSIBLE_ARGS='--skip-tags "test"' vagrant up --provision + +custom_ca: +ifdef CUSTOM_CA + cp -f ${CUSTOM_CA} docker/conf/certificates/ +endif + +up: update-box custom_ca +ifdef CI # CI is set in Github Actions SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant up --provision +else + SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} CUSTOM_CA=${CUSTOM_CA} ANSIBLE_ARGS='--extra-vars "local_test=true"' vagrant up --provision +endif -update-box: - @SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant box update || (echo '\n\nIf you get an SSL error you might be behind a transparent proxy. \nMore info https://github.com/fredrikhgrelland/vagrant-hashistack/blob/master/README.md#if-you-are-behind-a-transparent-proxy\n\n' && exit 2) +test: clean up + +example: custom_ca +ifdef CI # CI is set in Github Actions + cd test_example; SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant up --provision +else + cp -f docker/conf/certificates/*.crt test_example/docker/conf/certificates + cd test_example; SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} CUSTOM_CA=${CUSTOM_CA} ANSIBLE_ARGS='--extra-vars "local_test=true"' vagrant up --provision +endif # clean commands destroy-box: @@ -22,8 +41,7 @@ remove-tmp: clean: destroy-box remove-tmp -copy-consul: - if [ ! -f "./tmp/consul" ]; then mkdir -p ./tmp; vagrant ssh -c "cp /usr/local/bin/consul /vagrant/tmp/consul"; fi; +# helper commands +update-box: + @SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant box update || (echo '\n\nIf you get an SSL error you might be behind a transparent proxy. \nMore info https://github.com/fredrikhgrelland/vagrant-hashistack/blob/master/README.md#if-you-are-behind-a-transparent-proxy\n\n' && exit 2) -test: - $(MAKE) -C test diff --git a/README.md b/README.md index 370c4be..95c07e4 100644 --- a/README.md +++ b/README.md @@ -8,48 +8,83 @@ If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be intereste Documentation on [parent repository](https://github.com/fredrikhgrelland/vagrant-hashistack#usage). -## Customizing the vagrant box -The vagrant box ships with a default startup scheme. It will run an ansible playbook to start all services. +## Customizing and using the vagrant box + +### Building and testing docker image +See docker [README.md](docker/README.md). + +### Starting a box +The vagrant box ships with a default startup scheme. `make` from this directory will start the box, and it will run all books in [dev/ansible](dev/ansible) in lexical order (NB: `playbook.yml` is run first, but is only used to run all other playbooks) after the bootstrap-process for the hashistack is done. In the [example](test_example/dev/ansible/playbook.yml) we use it to start terraform which then starts a nomad-job. + +### Pre and post hashistack procedure You may change the hashistack configuration or add aditional pre and post steps to the startup procedure to match your needs. +Detailed documentation in [dev/vagrant/conf/README.md](dev/vagrant/conf/README.md) + +### Pre packaged configuration switches + +The box comes standard with a set of environment switches to simplify testing of different scenarios and enable staged development efforts. + +NB: All lowercase variables will automatically get a corresponding TF_VAR_ prepended variant for use directly in terraform. +To change from the default value, you may add the environment variable to [.env](dev/.env) + +#### Enterprise vs Open Source Software (OSS) +As long as Enterprise is not set to `true` the box will utilise OSS version of the binaries. + +#### Nomad + +| default | environment variable | value | +|:---------:|:----------------------|:-------:| +| | nomad_enterprise | true | +| x | nomad_enterprise | false | +| | nomad_acl | true | +| x | nomad_acl | false | -### Overriding and extending the configuration of the hashistack +When ACLs in Nomad are enabled the bootstrap token will be available in vault under `secret/nomad/management-token` with the two key-value pairs `accessor-id` and `secret-id`. `secret-id` is the token itself. These can be accessed in several ways: +- From inside the vagrant box with `vault kv get secret/nomad-bootstrap-token` +- From local machine with `vagrant ssh -c vault kv get secret/nomad-bootstrap-token"` +- By going to vault's UI on `localhost:8200`, and signing in with the root token. -- consul [vagrant/conf/hashistack/consul/99-override.hcl](vagrant/conf/hashistack/consul/99-override.hcl) -- nomad [vagrant/conf/hashistack/nomad/99-override.hcl](vagrant/conf/hashistack/nomad/99-override.hcl) -- vault [vagrant/conf/hashistack/vault/99-override.hcl](vagrant/conf/hashistack/vault/99-override.hcl) +#### Consul -You may edit the `99-override.hcl` or add your own. -Any valid configuration added to these directories will be added to their respective services' configuration, in lexical order. +| default | environment variable | value | +|:---------:|:---------------------------------|:-------:| +| | consul_enterprise | true | +| x | consul_enterprise | false | +| x | consul_acl | true | +| | consul_acl | false | +| x | consul_acl_default_policy | allow | +| | consul_acl_default_policy | deny | -#### Config variations -We provide some config variations as additional functionality, which you may want to use. -Current supported variations: +#### Vault + +| default | environment variable | value | +|:---------:|:---------------------------------|:-------:| +| | vault_enterprise | true | +| x | vault_enterprise | false | + +##### Consul secrets engine + +If `consul_acl_default_policy` has value `deny`, it will also enable [consul secrets engine](https://www.vaultproject.io/docs/secrets/consul) in vault. +Ansible will provision additional custom roles (admin-team, dev-team), [policies](../ansible/templates/consul-policies) and tokens for test purpose with different access level. + +How to generate token: ```text -* ACL default policy for consul (deny or allow) -``` -To choose other config variation you need to override ansible variables. There are two options how to override ansible variables: -* First option via `ANSIBLE_AGRS` -```bash -ANSIBLE_ARGS='--extra-vars "consul_agent_acl_default_policy=deny"' vagrant up --provision -``` -* Second option via prestart, adding fact in `vagrant/conf/ansible/playbooks/prestart/0-example.yml` -```yaml -# Option 2 how to turn on ACL -# [ACL] Set 'deny' default policy for consul agent -- set_fact: consul_agent_acl_default_policy="deny" -``` +# generate token for dev team member +vagrant ssh -c 'vault read consul/creds/dev-team' -Full set of extra_variables could be found [default_vars.yml](../ansible/default_vars.yml) +# generate token for admin team member +vagrant ssh -c 'vault read consul/creds/admin-team' +``` -### Pre- and post-startup ansible playbooks -This vagrant box will execute ansible playbooks put in two special directories [vagrant/conf/ansible/playbooks/prestart](vagrant/conf/ansible/playbooks/prestart) and [vagrant/conf/ansible/playbooks/poststart](vagrant/conf/ansible/playbooks/poststart). These playbooks will be executed before and after the box's bundled startup sequence, respectively. This gives the flexibility to configure all aspects of the hashistack as well as run tasks needed for tests or demo purposes as part of `vagrant up` Note; The playbooks are included into the main run, so the syntax in the [example](vagrant/conf/ansible/playbooks/prestart/0-example.yml) must be followed.. -They will be run in lexical order, and prefixing with numbers is a good way to get the order you want. +*Tokens can be used to access UI (different access level depends on role) -### Vagrant box life-cycle -![img](../docs/life-cycle.png) -1. `install.yml` - installing required software -2. `default_vars.yml` - setup default ansible variables -3. `prestart/*.yml` - prestart scripts, running before hashistack software will start -4. `bootstrap.yml` - verify ansible variables and software configuration, run hashistack software & verify that it started correctly -5. `poststart/*.yml` - poststart scripts, running after hasistack software runs and ready -6. `ansible/playbook.yml` - user's provisioning playbook +## Vagrant box life-cycle +1. `/home/vagrant/.env_default` - _preloaded_ - default variables +1. `vagrant/.env` - _user provided_ - variables override +1. `vagrant/.env_override` - _system provided_ - variables are overridden for test purposes +1. `vagrant/dev/vagrant/conf/pre_ansible.sh` - _user provided_ - script running before ansible bootstrap procedure +1. `vagrant/dev/vagrant/conf/pre_bootstrap/*.yml` - _user provided_ - pre bootstrap tasks, running before hashistack software runs and ready +1. `/etc/ansible/bootstrap.yml` - _preloaded_ - verify ansible variables and software configuration, run hashistack software & verify that it started correctly +1. `vagrant/conf/post_bootstrap/*.yml` - _user provided_ - poststart scripts, running after hasistack software runs and ready +1. `vagrant/dev/conf/pre_ansible.sh` - _user provided_ - script running after ansible bootstrap procedure +1. `vagrant/ansible/*.yml` - _user provided_ - ansible tasks included in playbook diff --git a/Vagrantfile b/Vagrantfile index a6e7528..18ed093 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -1,8 +1,12 @@ Vagrant.configure("2") do |config| config.vm.box = "fredrikhgrelland/hashistack" - config.vm.box_version = "~> 0.2" + config.vm.box_version = ">= 0.2, < 0.3" config.vm.provider "virtualbox" do |vb| vb.linked_clone = true vb.memory = 2048 end + config.vm.provision "ansible_local" do |ansible| + ansible.provisioning_path = "/vagrant/ansible" + ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml + end end diff --git a/ansible/playbook.yml b/ansible/playbook.yml deleted file mode 100644 index 2d8ff68..0000000 --- a/ansible/playbook.yml +++ /dev/null @@ -1,14 +0,0 @@ ---- -- hosts: all - become: yes - tasks: - - name: Terraform - terraform: - project_path: ../terraform - force_init: true - state: present - register: terraform - - - name: Terraform stdout - debug: - msg: "{{terraform.stdout}}" \ No newline at end of file diff --git a/conf/nomad/README.md b/conf/nomad/README.md new file mode 100644 index 0000000..992762f --- /dev/null +++ b/conf/nomad/README.md @@ -0,0 +1,3 @@ +# Nomad + +Put your nomad jobs in this directory and reference it inside terraform module code. (root directory) \ No newline at end of file diff --git a/dev/.env b/dev/.env new file mode 100644 index 0000000..1ecbb1b --- /dev/null +++ b/dev/.env @@ -0,0 +1 @@ +# User provided environment variables \ No newline at end of file diff --git a/dev/README.md b/dev/README.md new file mode 100644 index 0000000..9c1bbf3 --- /dev/null +++ b/dev/README.md @@ -0,0 +1,3 @@ +# Development and test code and configuration + +This directory contains code related to building, testing and developing in the vagrant box \ No newline at end of file diff --git a/ansible/README.md b/dev/ansible/README.md similarity index 61% rename from ansible/README.md rename to dev/ansible/README.md index 8c4f715..256dfda 100644 --- a/ansible/README.md +++ b/dev/ansible/README.md @@ -1,4 +1,6 @@ # Ansible This is a good starting point where you may run a series of sequential steps. -In the example, we use ansible to initialize and start a terraform job which in turn starts nomad jobs in parallel. \ No newline at end of file +In the example, we use ansible to initialize and start a terraform job which in turn starts nomad jobs in parallel. + +Se [example](../../test_example/dev/ansible) \ No newline at end of file diff --git a/dev/ansible/playbook.yml b/dev/ansible/playbook.yml new file mode 100644 index 0000000..0cc2fc5 --- /dev/null +++ b/dev/ansible/playbook.yml @@ -0,0 +1,7 @@ +--- +- hosts: all + become: yes + tasks: + - name: Include and run tasks + include_tasks: "{{ item }}" + loop: "{{ query('fileglob', \"/vagrant/ansible/*.yml\", wantlist=true) | reject('search','playbook.yml') | list | sort }}" \ No newline at end of file diff --git a/dev/vagrant/conf/README.md b/dev/vagrant/conf/README.md new file mode 100644 index 0000000..7657570 --- /dev/null +++ b/dev/vagrant/conf/README.md @@ -0,0 +1,31 @@ +# Runtime configuration of the hashistack Vagrant box + +There are two layers of configuration built into the box. + +## Outer layer ( advanced ) + +### Pre or post ansible bootstrap procedure + +#### Pre +You may add a `pre_ansible.sh` script file to this directory to run any alterations **before** ansible bootstrap procedure will run. + +This might come handy if you need to change or replace that bootstrap process. For example you replacing the entire `/etc/ansible` directory. +For most cases, you are probably looking to add configuration in [pre/poststart bootstrap](pre_bootstrap/README.md) +#### Post +If you need to run additional commands after ansible bootstrap has happened, you may add a `post_ansible.sh`. +This might come in handy if you would like to pat your self on tha back or test a recent configuration change before anything you might add to your own Vagrantfile. + +## Inner layer ( easy ) + +There are two primary cases for customization. + +### Add or override hashistack configurations + +- [nomad/](nomad/README.md) +- [consul/](consul/README.md) +- [vault/](vault/README.md) + +### Add pre and post bootstrap tasks + +- [pre_bootstrap](pre_bootstrap/README.md) +- [post_bootstrap](post_bootstrap/README.md) \ No newline at end of file diff --git a/dev/vagrant/conf/consul/README.md b/dev/vagrant/conf/consul/README.md new file mode 100644 index 0000000..a9b927b --- /dev/null +++ b/dev/vagrant/conf/consul/README.md @@ -0,0 +1,8 @@ +# Overriding and appending consul configuration + +You may add any hcl-files to this directory in order to change the configuration. +Any valid configuration added to this directory will append the configuration, in lexical order. + +Adding a file `99-override.hcl` you will ensure it will be appended last, and 00-override.hcl will be read first. +Any valid configuration from [https://www.consul.io/docs/agent/options.html#configuration_files](https://www.consul.io/docs/agent/options.html#configuration_files) will work. +See [example](../../../../test_example/dev/vagrant/conf/consul/99-override.hcl) diff --git a/dev/vagrant/conf/nomad/README.md b/dev/vagrant/conf/nomad/README.md new file mode 100644 index 0000000..c08e492 --- /dev/null +++ b/dev/vagrant/conf/nomad/README.md @@ -0,0 +1,19 @@ +# Overriding and appending nomad configuration + +You may add any hcl-files to this directory in order to change the configuration. +Any valid configuration added to this directory will append the configuration, in lexical order. + +Adding a file `99-override.hcl` you will ensure it will be appended last, and 00-override.hcl will be read first. +Any valid configuration from [https://www.nomadproject.io/docs/configuration#general-parameters](https://www.nomadproject.io/docs/configuration#general-parameters) will work. + +## Example `98-template-plugin.hcl` +```hcl +client { + template { + #Remove blacklist in order for allow "plugins" to run. We need curl to run as a plugin in template + plugin_blacklist = [] + } +} +``` + +See [example](../../../../test_example/dev/vagrant/conf/nomad/99-override.hcl) \ No newline at end of file diff --git a/dev/vagrant/conf/post_bootstrap/README.md b/dev/vagrant/conf/post_bootstrap/README.md new file mode 100644 index 0000000..5e2db8d --- /dev/null +++ b/dev/vagrant/conf/post_bootstrap/README.md @@ -0,0 +1 @@ +# See [../post_boostrap](../post_bootstrap) \ No newline at end of file diff --git a/dev/vagrant/conf/pre_bootstrap/README.md b/dev/vagrant/conf/pre_bootstrap/README.md new file mode 100644 index 0000000..4350d9a --- /dev/null +++ b/dev/vagrant/conf/pre_bootstrap/README.md @@ -0,0 +1,15 @@ +# Customize the startup procedure of the Hashistack with pre- and post start ansible scripts + +You may put any number of script files in this directory for running ansible commands prior to bootstrapping the hashistack. +The bootstrap procedure is included/hardcoded in your box. +[bootstrap.yml](https://github.com/fredrikhgrelland/vagrant-hashistack/blob/master/ansible/bootstrap.yml) will start by running the scripts in this folder and end by running the scripts in [../post_bootstrap](../post_bootstrap) + + +The files e.g. 0-example.yml must only include pure ansible task syntax: +```yaml +- name: Task that shows usage of prestart + debug: + msg: This would be a prestart task +``` + +See [example](../../../../test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml) \ No newline at end of file diff --git a/dev/vagrant/conf/vault/README.md b/dev/vagrant/conf/vault/README.md new file mode 100644 index 0000000..1819c1b --- /dev/null +++ b/dev/vagrant/conf/vault/README.md @@ -0,0 +1,9 @@ +# Overriding and appending vault configuration + +You may add any hcl-files to this directory in order to change the configuration. +Any valid configuration added to this directory will append the configuration, in lexical order. + +Adding a file `99-override.hcl` you will ensure it will be appended last, and 00-override.hcl will be read first. +Any valid configuration from [https://www.vaultproject.io/docs/configuration](https://www.vaultproject.io/docs/configuration) will work. + +See [example](../../../../test_example/dev/vagrant/conf/vault/99-override.hcl) \ No newline at end of file diff --git a/docker/.dockerignore b/docker/.dockerignore new file mode 100644 index 0000000..bdef5a7 --- /dev/null +++ b/docker/.dockerignore @@ -0,0 +1,4 @@ +* +!bin +!lib +!conf \ No newline at end of file diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 0000000..2228af7 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,16 @@ +# Docker build directory + +Put your Dockerfile and other files relating to a docker-build here. + +## Building docker image locally + +If you have docker installed on your machine, you may `cd docker; docker build -t my_image:local .` and build the image. + +This image can be built and operated behind a corporate proxy where the base os needs to trust a custom CA. +While building locally using the Makefile, you may set the environment variable CUSTOM_CA to a custom .crt file in order to import it into the docker image. See [conf/certificates](conf/certificates) + +See [../test_example/docker/Dockerfile](../test_example/docker/Dockerfile) for examples on how to import and trust CA for centos/debian/alpine based docker images. + +## Building and testing the docker-image within the vagrant-hashistack box + +We advise you to build and test your docker image within the hashistack eco-system. Running `make test` will launch the [default playbook](../dev/ansible/playbook.yml) inside the box, and [test_example/](../test_example/) shows a simple build process for building and running the docker image using this. Refer to books in [test_example/dev/ansible](../test_example/dev/ansible) to see details. diff --git a/docker/conf/certificates/.gitignore b/docker/conf/certificates/.gitignore new file mode 100644 index 0000000..ee0cb61 --- /dev/null +++ b/docker/conf/certificates/.gitignore @@ -0,0 +1 @@ +*.crt \ No newline at end of file diff --git a/docker/conf/certificates/README.md b/docker/conf/certificates/README.md new file mode 100644 index 0000000..5829d33 --- /dev/null +++ b/docker/conf/certificates/README.md @@ -0,0 +1,2 @@ +# Custom CA certificates +If the env variable `CUSTOM_CA` is pointing to a certificate file, it will be copied into this directory by the Makefile. \ No newline at end of file diff --git a/example/README.md b/example/README.md new file mode 100644 index 0000000..37fe4f2 --- /dev/null +++ b/example/README.md @@ -0,0 +1,3 @@ +# This is a terraform module example + +TODO: explain \ No newline at end of file diff --git a/main.tf b/main.tf new file mode 100644 index 0000000..e69de29 diff --git a/nomad/README.md b/nomad/README.md deleted file mode 100644 index 7162afa..0000000 --- a/nomad/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Nomad - -Put your nomad jobs in this directory and reference it in [../terraform](../terraform) code. \ No newline at end of file diff --git a/nomad/your_nomad_job.hcl b/nomad/your_nomad_job.hcl deleted file mode 100644 index aede7f0..0000000 --- a/nomad/your_nomad_job.hcl +++ /dev/null @@ -1 +0,0 @@ -#Placeholder \ No newline at end of file diff --git a/outputs.tf b/outputs.tf new file mode 100644 index 0000000..e69de29 diff --git a/terraform/README.md b/terraform/README.md deleted file mode 100644 index edd3d0e..0000000 --- a/terraform/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Terraform - -Any configuration of the hashistack is best performed through terraform providers. -- [nomad](https://www.terraform.io/docs/providers/nomad/index.html) -- [vault](https://www.terraform.io/docs/providers/vault/index.html) -- [consul](https://www.terraform.io/docs/providers/consul/index.html) - -You put them in this directory and use [../ansible/playbook.yml](../ansible/playbook.yml) to init and run terraform. \ No newline at end of file diff --git a/terraform/main.tf b/terraform/main.tf deleted file mode 100644 index e16257c..0000000 --- a/terraform/main.tf +++ /dev/null @@ -1,9 +0,0 @@ -provider "nomad" { - address = "http://127.0.0.1:4646" -} -/* -resource "nomad_job" "your_nomad_job" { - jobspec = file("${path.cwd}/../nomad/your_nomad_job.hcl") - detach = false -} -*/ \ No newline at end of file diff --git a/test/.env b/test/.env deleted file mode 100644 index 272cdce..0000000 --- a/test/.env +++ /dev/null @@ -1,2 +0,0 @@ -#NOMAD_TOKEN=NOT_IN_USE -#VAULT_TOKEN=NOT_IN_USE \ No newline at end of file diff --git a/test/Makefile b/test/Makefile deleted file mode 100644 index 82abd63..0000000 --- a/test/Makefile +++ /dev/null @@ -1,18 +0,0 @@ -include .env -export - -.ONESHELL .PHONY: test clean up -.DEFAULT_GOAL := test - -up: - vagrant up --provision - $(MAKE) clean - # Option 1 how to turn on ACL - ANSIBLE_ARGS='--extra-vars "consul_agent_acl_default_policy=deny"' vagrant up --provision - $(MAKE) clean - -clean: - vagrant destroy -f - rm -rf .vagrant - -test: clean up diff --git a/test/Vagrantfile b/test/Vagrantfile deleted file mode 100644 index 6cc19e8..0000000 --- a/test/Vagrantfile +++ /dev/null @@ -1,7 +0,0 @@ -Vagrant.configure("2") do |config| - config.vm.box = "fredrikhgrelland/hashistack" - config.vm.provider "virtualbox" do |vb| - vb.linked_clone = true - vb.memory = 2048 - end -end diff --git a/test/ansible/playbook.yml b/test/ansible/playbook.yml deleted file mode 100644 index 2d8ff68..0000000 --- a/test/ansible/playbook.yml +++ /dev/null @@ -1,14 +0,0 @@ ---- -- hosts: all - become: yes - tasks: - - name: Terraform - terraform: - project_path: ../terraform - force_init: true - state: present - register: terraform - - - name: Terraform stdout - debug: - msg: "{{terraform.stdout}}" \ No newline at end of file diff --git a/test/terraform/main.tf b/test/terraform/main.tf deleted file mode 100644 index f09ba72..0000000 --- a/test/terraform/main.tf +++ /dev/null @@ -1,8 +0,0 @@ -provider "nomad" { - address = "http://127.0.0.1:4646" -} - -resource "nomad_job" "countdash" { - jobspec = file("${path.cwd}/../nomad/countdash.hcl") - detach = false -} \ No newline at end of file diff --git a/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml b/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml deleted file mode 100644 index 0709769..0000000 --- a/test/vagrant/conf/ansible/playbooks/poststart/0-example.yml +++ /dev/null @@ -1,3 +0,0 @@ -- name: Task that shows usage of poststart - debug: - msg: This would be a poststart task diff --git a/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml b/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml deleted file mode 100644 index 52f8e81..0000000 --- a/test/vagrant/conf/ansible/playbooks/prestart/0-example.yml +++ /dev/null @@ -1,3 +0,0 @@ -- name: Task that shows usage of prestart - debug: - msg: This would be a prestart task \ No newline at end of file diff --git a/test_example/.env b/test_example/.env new file mode 100644 index 0000000..16ca13c --- /dev/null +++ b/test_example/.env @@ -0,0 +1,4 @@ +#Control box features +#consul_acl=true +#consul_acl_default_policy=allow +#nomad_acl=false \ No newline at end of file diff --git a/test_example/Vagrantfile b/test_example/Vagrantfile new file mode 100644 index 0000000..e9d6a3d --- /dev/null +++ b/test_example/Vagrantfile @@ -0,0 +1,11 @@ +Vagrant.configure("2") do |config| + config.vm.box = "fredrikhgrelland/hashistack" + config.vm.provider "virtualbox" do |vb| + vb.linked_clone = true + vb.memory = 2048 + end + config.vm.provision "ansible_local" do |ansible| + ansible.provisioning_path = "/vagrant/dev/ansible" + ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml + end +end diff --git a/test/nomad/countdash.hcl b/test_example/conf/nomad/countdash.hcl similarity index 58% rename from test/nomad/countdash.hcl rename to test_example/conf/nomad/countdash.hcl index 9314ddf..d7a2890 100644 --- a/test/nomad/countdash.hcl +++ b/test_example/conf/nomad/countdash.hcl @@ -12,12 +12,28 @@ job "countdash" { connect { sidecar_service {} } + check { + expose = true + name = "api-alive" + type = "http" + path = "/health" + interval = "10s" + timeout = "2s" + } } task "web" { driver = "docker" + artifact { + source = "s3::http://127.0.0.1:9000/dev/tmp/docker_image.tar" + options { + aws_access_key_id = "minioadmin" + aws_access_key_secret = "minioadmin" + } + } config { - image = "hashicorpnomad/counter-api:v1" + load = "docker_image.tar" + image = "docker_image:local" } } } @@ -45,6 +61,14 @@ job "countdash" { } } } + check { + expose = true + name = "dashboard-alive" + type = "http" + path = "/health" + interval = "10s" + timeout = "2s" + } } task "dashboard" { diff --git a/test_example/dev/ansible/01_build_docker_image.yml b/test_example/dev/ansible/01_build_docker_image.yml new file mode 100644 index 0000000..db4e7cd --- /dev/null +++ b/test_example/dev/ansible/01_build_docker_image.yml @@ -0,0 +1,32 @@ +- name: Remove docker image + docker_image: + name: docker_image + tag: local + force_absent: true + state: absent + +- name: Build docker image + docker_image: + name: docker_image + tag: local + build: + path: /vagrant/docker + pull: false + args: + TEST_DOWNLOAD_BUILD_ARGUMENT: https://nrk.no + source: build + +- name: Create tmp if it does not exist + file: + path: /vagrant/dev/tmp + state: directory + mode: '0755' + owner: vagrant + group: vagrant + +- name: Archive docker image + docker_image: + name: docker_image + tag: local + archive_path: /vagrant/dev/tmp/docker_image.tar + source: local \ No newline at end of file diff --git a/test_example/dev/ansible/02_run_terraform.yml b/test_example/dev/ansible/02_run_terraform.yml new file mode 100644 index 0000000..cc381f2 --- /dev/null +++ b/test_example/dev/ansible/02_run_terraform.yml @@ -0,0 +1,10 @@ +- name: Terraform + terraform: + project_path: ../../example + force_init: true + state: present + register: terraform + +- name: Terraform stdout + debug: + msg: "{{terraform.stdout}}" \ No newline at end of file diff --git a/test_example/dev/ansible/playbook.yml b/test_example/dev/ansible/playbook.yml new file mode 100644 index 0000000..5ccb21b --- /dev/null +++ b/test_example/dev/ansible/playbook.yml @@ -0,0 +1,7 @@ +--- +- hosts: all + become: yes + tasks: + - name: Include and run tasks + include_tasks: "{{ item }}" + loop: "{{ query('fileglob', \"/vagrant/dev/ansible/*.yml\", wantlist=true) | reject('search','playbook.yml') | list | sort }}" \ No newline at end of file diff --git a/test/vagrant/conf/hashistack/consul/99-override.hcl b/test_example/dev/vagrant/conf/consul/99-override.hcl similarity index 100% rename from test/vagrant/conf/hashistack/consul/99-override.hcl rename to test_example/dev/vagrant/conf/consul/99-override.hcl diff --git a/test/vagrant/conf/hashistack/nomad/99-override.hcl b/test_example/dev/vagrant/conf/nomad/99-override.hcl similarity index 100% rename from test/vagrant/conf/hashistack/nomad/99-override.hcl rename to test_example/dev/vagrant/conf/nomad/99-override.hcl diff --git a/test_example/dev/vagrant/conf/post_ansible.sh b/test_example/dev/vagrant/conf/post_ansible.sh new file mode 100644 index 0000000..d09910c --- /dev/null +++ b/test_example/dev/vagrant/conf/post_ansible.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "Running /vagrant/dev/vagrant/conf/post_ansible.sh" \ No newline at end of file diff --git a/test_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml b/test_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml new file mode 100644 index 0000000..7bf4f56 --- /dev/null +++ b/test_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of poststart + debug: + msg: This is the first poststart task ( 00-poststart-example.yml ) \ No newline at end of file diff --git a/test_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml b/test_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml new file mode 100644 index 0000000..4fc2a1b --- /dev/null +++ b/test_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of poststart + debug: + msg: This is the second poststart task ( 01-poststart-example.yml ) \ No newline at end of file diff --git a/test_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml b/test_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml new file mode 100644 index 0000000..fc045ea --- /dev/null +++ b/test_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of poststart + debug: + msg: This is the third poststart task ( 02-poststart-example.yml ) \ No newline at end of file diff --git a/test_example/dev/vagrant/conf/pre_ansible.sh b/test_example/dev/vagrant/conf/pre_ansible.sh new file mode 100644 index 0000000..4695070 --- /dev/null +++ b/test_example/dev/vagrant/conf/pre_ansible.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +echo "Running /vagrant/dev/vagrant/conf/pre_ansible.sh" \ No newline at end of file diff --git a/test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml b/test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml new file mode 100644 index 0000000..4bcb7a7 --- /dev/null +++ b/test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of prestart + debug: + msg: This is the first prestart task ( 00-poststart-example.yml ) \ No newline at end of file diff --git a/test_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml b/test_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml new file mode 100644 index 0000000..07002cc --- /dev/null +++ b/test_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of prestart + debug: + msg: This is the second prestart task ( 01-poststart-example.yml ) \ No newline at end of file diff --git a/test_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml b/test_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml new file mode 100644 index 0000000..deeb4d3 --- /dev/null +++ b/test_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml @@ -0,0 +1,3 @@ +- name: Task that shows usage of prestart + debug: + msg: This is the third prestart task ( 02-poststart-example.yml ) \ No newline at end of file diff --git a/test/vagrant/conf/hashistack/vault/99-override.hcl b/test_example/dev/vagrant/conf/vault/99-override.hcl similarity index 100% rename from test/vagrant/conf/hashistack/vault/99-override.hcl rename to test_example/dev/vagrant/conf/vault/99-override.hcl diff --git a/test_example/docker/.dockerignore b/test_example/docker/.dockerignore new file mode 100644 index 0000000..bdef5a7 --- /dev/null +++ b/test_example/docker/.dockerignore @@ -0,0 +1,4 @@ +* +!bin +!lib +!conf \ No newline at end of file diff --git a/test_example/docker/Dockerfile b/test_example/docker/Dockerfile new file mode 100644 index 0000000..b8ca278 --- /dev/null +++ b/test_example/docker/Dockerfile @@ -0,0 +1,56 @@ +########################################## +########### READ THIS FIRST ############## +########################################## +## This docker build is special... ## +## It wil leverage multi stage builds ## +## to test centos/debian/alpine ## +## certificate trust before building ## +## the docker image used in the example ## +########################################## + +FROM centos:8 + +# Allow buildtime config +ARG TEST_DOWNLOAD_BUILD_ARGUMENT=https://nrk.no + +#Add ca_certificates to the image ( if trust is not already added through base image ) +COPY conf/certificates /usr/share/pki/ca-trust-source/anchors/ + +#Install certs +RUN \ + #Update CA_Certs + update-ca-trust 2>/dev/null || true && echo "NOTE: CA warnings suppressed." \ + #Test download ( does ssl trust work ) + && curl -s -L -o /dev/null ${TEST_DOWNLOAD_BUILD_ARGUMENT} || printf "\n###############\nERROR: You are probably behind a corporate proxy. Add your custom ca .crt in the conf/certificates docker build folder\n###############\n" + +FROM debian:stretch + +# Allow buildtime config +ARG TEST_DOWNLOAD_BUILD_ARGUMENT=https://nrk.no + +#Add ca_certificates to the image ( if trust is not already added through base image ) +COPY conf/certificates /usr/local/share/ca-certificates + +#Install certs +# hadolint ignore=DL3015 +RUN \ + #Update CA_Certs + apt-get update && apt-get install -y curl=7.52.1-5+deb9u11 && rm -rf /var/lib/apt/lists/* \ + && update-ca-certificates 2>/dev/null || true && echo "NOTE: CA warnings suppressed." \ + #Test download ( does ssl trust work ) + && curl -s -L -o /dev/null ${TEST_DOWNLOAD_BUILD_ARGUMENT} || printf "\n###############\nERROR: You are probably behind a corporate proxy. Add your custom ca .crt in the conf/certificates docker build folder\n###############\n" + + +FROM hashicorpnomad/counter-api:v1 + +# Allow buildtime config +ARG TEST_DOWNLOAD_BUILD_ARGUMENT=https://nrk.no + +#Add ca_certificates to the image ( if trust is not already added through base image ) +COPY conf/certificates /usr/local/share/ca-certificates + +RUN apk --no-cache add curl=~7 ca-certificates=~20190108 \ + && find /usr/local/share/ca-certificates -not -name "*.crt" -type f -delete \ + && update-ca-certificates 2>/dev/null || true && echo "NOTE: CA warnings suppressed." \ + # Test download + && curl -s -L -o /dev/null ${TEST_DOWNLOAD_BUILD_ARGUMENT} || printf "\n###############\nERROR: You are probably behind a corporate proxy. Add your custom ca .crt in the conf/certificates docker build folder\n###############\n" diff --git a/test_example/docker/conf/certificates/.gitignore b/test_example/docker/conf/certificates/.gitignore new file mode 100644 index 0000000..ee0cb61 --- /dev/null +++ b/test_example/docker/conf/certificates/.gitignore @@ -0,0 +1 @@ +*.crt \ No newline at end of file diff --git a/test_example/docker/conf/certificates/README.md b/test_example/docker/conf/certificates/README.md new file mode 100644 index 0000000..5829d33 --- /dev/null +++ b/test_example/docker/conf/certificates/README.md @@ -0,0 +1,2 @@ +# Custom CA certificates +If the env variable `CUSTOM_CA` is pointing to a certificate file, it will be copied into this directory by the Makefile. \ No newline at end of file diff --git a/test_example/example/main.tf b/test_example/example/main.tf new file mode 100644 index 0000000..6bb90a1 --- /dev/null +++ b/test_example/example/main.tf @@ -0,0 +1,7 @@ +module "countdash" { + source = "./.." +} + +provider "vault" { + address = "http://127.0.0.1:8200" +} \ No newline at end of file diff --git a/test_example/example/nomad_acl_test.tf b/test_example/example/nomad_acl_test.tf new file mode 100644 index 0000000..2ed9a35 --- /dev/null +++ b/test_example/example/nomad_acl_test.tf @@ -0,0 +1,12 @@ +data "vault_generic_secret" "nomad_secret_id" { + # Set count of this data source to 1 if ACLs are enabled in Nomad, and 0 if not + count = var.nomad_acl ? 1 : 0 + path = "nomad/creds/write" +} + +provider "nomad" { + address = "http://127.0.0.1:4646" + # Add a secret_id if ACLs are enabled in nomad + secret_id = var.nomad_acl ? data.vault_generic_secret.nomad_secret_id[0].data.secret_id : null +} + diff --git a/test_example/example/variables.tf b/test_example/example/variables.tf new file mode 100644 index 0000000..55cbd1b --- /dev/null +++ b/test_example/example/variables.tf @@ -0,0 +1,3 @@ +variable "nomad_acl" { + type = bool +} \ No newline at end of file diff --git a/test_example/main.tf b/test_example/main.tf new file mode 100644 index 0000000..4a066ce --- /dev/null +++ b/test_example/main.tf @@ -0,0 +1,4 @@ +resource "nomad_job" "countdash" { + jobspec = file("${path.module}/conf/nomad/countdash.hcl") + detach = false +} \ No newline at end of file diff --git a/test_example/outputs.tf b/test_example/outputs.tf new file mode 100644 index 0000000..a711302 --- /dev/null +++ b/test_example/outputs.tf @@ -0,0 +1,3 @@ +output "nomad_job" { + value = nomad_job.countdash +} \ No newline at end of file diff --git a/test_example/variables.tf b/test_example/variables.tf new file mode 100644 index 0000000..e69de29 diff --git a/vagrant/conf/ansible/playbooks/poststart/0-example.yml b/vagrant/conf/ansible/playbooks/poststart/0-example.yml deleted file mode 100644 index 0709769..0000000 --- a/vagrant/conf/ansible/playbooks/poststart/0-example.yml +++ /dev/null @@ -1,3 +0,0 @@ -- name: Task that shows usage of poststart - debug: - msg: This would be a poststart task diff --git a/vagrant/conf/ansible/playbooks/prestart/0-example.yml b/vagrant/conf/ansible/playbooks/prestart/0-example.yml deleted file mode 100644 index cc03e7f..0000000 --- a/vagrant/conf/ansible/playbooks/prestart/0-example.yml +++ /dev/null @@ -1,3 +0,0 @@ -- name: Task that shows usage of prestart - debug: - msg: This would be a prestart task diff --git a/vagrant/conf/ansible/playbooks/prestart/1-acl.yml b/vagrant/conf/ansible/playbooks/prestart/1-acl.yml deleted file mode 100644 index ab73d9e..0000000 --- a/vagrant/conf/ansible/playbooks/prestart/1-acl.yml +++ /dev/null @@ -1,3 +0,0 @@ -## Option 2 how to turn on ACL -## [ACL] Set 'deny' default policy for consul agent -#- set_fact: consul_agent_acl_default_policy="deny" \ No newline at end of file diff --git a/vagrant/conf/hashistack/consul/99-override.hcl b/vagrant/conf/hashistack/consul/99-override.hcl deleted file mode 100644 index d2b8041..0000000 --- a/vagrant/conf/hashistack/consul/99-override.hcl +++ /dev/null @@ -1 +0,0 @@ -#Any vaild configuration from https://www.consul.io/docs/agent/options.html#configuration_files \ No newline at end of file diff --git a/vagrant/conf/hashistack/nomad/99-override.hcl b/vagrant/conf/hashistack/nomad/99-override.hcl deleted file mode 100644 index d817340..0000000 --- a/vagrant/conf/hashistack/nomad/99-override.hcl +++ /dev/null @@ -1 +0,0 @@ -#Any vaild configuration from https://www.nomadproject.io/docs/configuration#general-parameters \ No newline at end of file diff --git a/vagrant/conf/hashistack/vault/99-override.hcl b/vagrant/conf/hashistack/vault/99-override.hcl deleted file mode 100644 index 2a828c6..0000000 --- a/vagrant/conf/hashistack/vault/99-override.hcl +++ /dev/null @@ -1 +0,0 @@ -#Any vaild configuration from https://www.vaultproject.io/docs/configuration \ No newline at end of file diff --git a/variables.tf b/variables.tf new file mode 100644 index 0000000..e69de29 From 3784fa0c8b231b8c0eb575b1d1f8cdc250cb0d0b Mon Sep 17 00:00:00 2001 From: fredrikhgrelland <> Date: Wed, 19 Aug 2020 08:22:55 +0000 Subject: [PATCH 08/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/4de1874cf71126f6b7f628ec8511c92e02041bad) --- Vagrantfile | 4 ++-- dev/ansible/playbook.yml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 18ed093..3ad3025 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -1,12 +1,12 @@ Vagrant.configure("2") do |config| config.vm.box = "fredrikhgrelland/hashistack" - config.vm.box_version = ">= 0.2, < 0.3" + config.vm.box_version = ">= 0.3, < 0.4" config.vm.provider "virtualbox" do |vb| vb.linked_clone = true vb.memory = 2048 end config.vm.provision "ansible_local" do |ansible| - ansible.provisioning_path = "/vagrant/ansible" + ansible.provisioning_path = "/vagrant/dev/ansible" ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml end end diff --git a/dev/ansible/playbook.yml b/dev/ansible/playbook.yml index 0cc2fc5..5ccb21b 100644 --- a/dev/ansible/playbook.yml +++ b/dev/ansible/playbook.yml @@ -4,4 +4,4 @@ tasks: - name: Include and run tasks include_tasks: "{{ item }}" - loop: "{{ query('fileglob', \"/vagrant/ansible/*.yml\", wantlist=true) | reject('search','playbook.yml') | list | sort }}" \ No newline at end of file + loop: "{{ query('fileglob', \"/vagrant/dev/ansible/*.yml\", wantlist=true) | reject('search','playbook.yml') | list | sort }}" \ No newline at end of file From 4329c51de860a481f7d4c20f56d40c756f4a6764 Mon Sep 17 00:00:00 2001 From: fredrikhgrelland <> Date: Fri, 28 Aug 2020 08:54:11 +0000 Subject: [PATCH 09/10] Update from https://github.com/fredrikhgrelland/vagrant-hashistack/commit/62a1d8db737b865892a3a905f6d9367603e9d5d3) --- LICENSE | 201 +++++++++++ Makefile | 12 +- README.md | 339 ++++++++++++++++-- README_template.md | 51 +++ Vagrantfile | 14 +- Vagrantfile.default | 12 + dev/ansible/README.md | 2 +- dev/vagrant/conf/README.md | 1 + dev/vagrant/conf/consul/README.md | 2 +- dev/vagrant/conf/nomad/README.md | 2 +- dev/vagrant/conf/pre_bootstrap/README.md | 2 +- dev/vagrant/conf/vault/README.md | 2 +- docker/README.md | 4 +- example/README.md | 4 +- {test_example => template_example}/.env | 0 template_example/Vagrantfile | 2 + .../conf/nomad/countdash.hcl | 0 .../dev/ansible/01_build_docker_image.yml | 5 +- .../dev/ansible/02_run_terraform.yml | 0 .../dev/ansible/playbook.yml | 0 .../dev/vagrant/conf/consul/99-override.hcl | 0 .../dev/vagrant/conf/nomad/99-override.hcl | 0 .../dev/vagrant/conf/post_ansible.sh | 0 .../post_bootstrap/00-poststart-example.yml | 0 .../post_bootstrap/01-poststart-example.yml | 0 .../post_bootstrap/02-poststart-example.yml | 0 .../dev/vagrant/conf/pre_ansible.sh | 0 .../pre_bootstrap/00-prestart-example.yml | 0 .../pre_bootstrap/01-prestart-example.yml | 0 .../pre_bootstrap/02-prestart-example.yml | 0 .../dev/vagrant/conf/vault/99-override.hcl | 0 .../docker/.dockerignore | 0 .../docker/Dockerfile | 0 .../docker/conf/certificates/.gitignore | 0 .../docker/conf/certificates/README.md | 0 .../example/main.tf | 0 .../example/nomad_acl_test.tf | 0 .../example/variables.tf | 0 {test_example => template_example}/main.tf | 0 {test_example => template_example}/outputs.tf | 0 .../variables.tf | 0 test_example/Vagrantfile | 11 - 42 files changed, 598 insertions(+), 68 deletions(-) create mode 100644 LICENSE create mode 100644 README_template.md create mode 100644 Vagrantfile.default rename {test_example => template_example}/.env (100%) create mode 100644 template_example/Vagrantfile rename {test_example => template_example}/conf/nomad/countdash.hcl (100%) rename {test_example => template_example}/dev/ansible/01_build_docker_image.yml (83%) rename {test_example => template_example}/dev/ansible/02_run_terraform.yml (100%) rename {test_example => template_example}/dev/ansible/playbook.yml (100%) rename {test_example => template_example}/dev/vagrant/conf/consul/99-override.hcl (100%) rename {test_example => template_example}/dev/vagrant/conf/nomad/99-override.hcl (100%) rename {test_example => template_example}/dev/vagrant/conf/post_ansible.sh (100%) rename {test_example => template_example}/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml (100%) rename {test_example => template_example}/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml (100%) rename {test_example => template_example}/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml (100%) rename {test_example => template_example}/dev/vagrant/conf/pre_ansible.sh (100%) rename {test_example => template_example}/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml (100%) rename {test_example => template_example}/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml (100%) rename {test_example => template_example}/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml (100%) rename {test_example => template_example}/dev/vagrant/conf/vault/99-override.hcl (100%) rename {test_example => template_example}/docker/.dockerignore (100%) rename {test_example => template_example}/docker/Dockerfile (100%) rename {test_example => template_example}/docker/conf/certificates/.gitignore (100%) rename {test_example => template_example}/docker/conf/certificates/README.md (100%) rename {test_example => template_example}/example/main.tf (100%) rename {test_example => template_example}/example/nomad_acl_test.tf (100%) rename {test_example => template_example}/example/variables.tf (100%) rename {test_example => template_example}/main.tf (100%) rename {test_example => template_example}/outputs.tf (100%) rename {test_example => template_example}/variables.tf (100%) delete mode 100644 test_example/Vagrantfile diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..261eeb9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/Makefile b/Makefile index 261cecf..ebfe6c8 100644 --- a/Makefile +++ b/Makefile @@ -5,6 +5,10 @@ export PATH := $(shell pwd)/tmp:$(PATH) .ONESHELL .PHONY: up update-box destroy-box remove-tmp clean example .DEFAULT_GOAL := up +#### Pre requisites #### +install: + mkdir -p tmp;(cd tmp; git clone --depth=1 https://github.com/fredrikhgrelland/vagrant-hashistack.git; cd vagrant-hashistack; make install); rm -rf tmp/vagrant-hashistack + #### Development #### # start commands dev: update-box @@ -24,12 +28,12 @@ endif test: clean up -example: custom_ca +template-example: custom_ca ifdef CI # CI is set in Github Actions - cd test_example; SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant up --provision + cd template_example; SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant up --provision else - cp -f docker/conf/certificates/*.crt test_example/docker/conf/certificates - cd test_example; SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} CUSTOM_CA=${CUSTOM_CA} ANSIBLE_ARGS='--extra-vars "local_test=true"' vagrant up --provision + if [ -f "docker/conf/certificates/*.crt" ]; then cp -f docker/conf/certificates/*.crt template_example/docker/conf/certificates; fi + cd template_example; SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} CUSTOM_CA=${CUSTOM_CA} ANSIBLE_ARGS='--extra-vars "local_test=true"' vagrant up --provision endif # clean commands diff --git a/README.md b/README.md index 95c07e4..f98ba4c 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,169 @@ -# Starter template for `fredrikhgrelland/hashistack` + +

+ Vagrant-hashistack +

Vagrant-hashistack Template

+

Starter template for fredrikhgrelland/vagrant-hashistack

+

+ + Build + + + Releases + + + Updated + +
+
+

+ + + +

+

-This repository can be used as a base for developing services on the hashistack. -On github, you may use the ["Use this template"](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to generate a new repository from this template. +## Content +1. [Description - What & Why](#description---what--why) + 1. [Why Does This Exist?](#why-does-this-exist) + 2. [Services](#services) +2. [Install Prerequisites](#install-prerequisites) + 1. [Packages that needs to be pre-installed](#packages-that-needs-to-be-pre-installed) + 1. [MacOS Specific](#macos-specific) + 2. [Ubuntu Specific](#ubuntu-specific) +3. [Configuration](#configuration) + 1. [Startup Scheme](#startup-scheme) + 1. [Detailed Startup Procedure](#detailed-startup-procedure) + 2. [Pre and Post Hashistack Startup Procedure](#pre-and-post-hashistack-startup-procedure) + 1. [Ansible Playbooks Pre and Post Hashistack Startup](#ansible-playbooks-pre-and-post-hashistack-startup) + 2. [Bash Scripts Pre and Post Ansible Playbook](#bash-scripts-pre-and-post-ansible-playbook) + 3. [Pre-packaged Configuration Switches](#pre-packaged-configuration-switches) + 1. [Enterprise vs Open Source Software (OSS)](#enterprise-vs-open-source-software-oss) + 2. [Nomad](#nomad) + 3. [Consul](#consul) + 4. [Vault](#vault) + 1. [Consul Secrets Engine](#consul-secrets-engine) + 2. [Vagrant Box Resources](#vagrant-box-resources) +4. [Usage](#usage) + 1. [Commands](#commands) + 2. [MinIO](#minio) + 1. [Pushing Resources To MinIO With Ansible (Docker image)](#pushing-resources-to-minio-with-ansible-docker-image) + 2. [Fetching Resources From MinIO With Nomad (Docker image)](#fetching-resources-from-minio-with-nomad-docker-image) + 3. [Iteration of the Development Process](#iteration-of-the-development-process) +5. [Test Configuration and Execution](#test-configuration-and-execution) -If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in this separate repository [vagrant-hashistack-template](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to start a new repository from this repo -. -Documentation on [parent repository](https://github.com/fredrikhgrelland/vagrant-hashistack#usage). +## Description - What & Why +This template is a starting point, and example, on how to take advantage of the [Hashistack vagrant-box](https://app.vagrantup.com/fredrikhgrelland/boxes/hashistack) to create, develop, and test Terraform-modules within the Hashistack ecosystem. -## Customizing and using the vagrant box +**Hashistack**, in current repository context, is a set of software products by [HashiCorp](https://www.hashicorp.com/). -### Building and testing docker image -See docker [README.md](docker/README.md). -### Starting a box -The vagrant box ships with a default startup scheme. `make` from this directory will start the box, and it will run all books in [dev/ansible](dev/ansible) in lexical order (NB: `playbook.yml` is run first, but is only used to run all other playbooks) after the bootstrap-process for the hashistack is done. In the [example](test_example/dev/ansible/playbook.yml) we use it to start terraform which then starts a nomad-job. +> :bulb: If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in the separate repository [vagrant-hashistack-template](https://github.com/fredrikhgrelland/vagrant-hashistack-template/). -### Pre and post hashistack procedure -You may change the hashistack configuration or add aditional pre and post steps to the startup procedure to match your needs. +> :warning: If you are reading this in your own repository, go to [If This Is in Your Own Repository](#if-this-is-in-your-own-repository) + +### Why Does This Exist? + This template aims to standardize workflow for building and testing terraform-nomad-modules, using the [fredrikhgrelland/hashistack](https://github.com/fredrikhgrelland/vagrant-hashistack) vagrant-box. + + +### Services +The default box will start Nomad, Vault, Consul and MinIO bound to loopback and advertising on the IP `10.0.3.10`, which should be available on your local machine. +Port-forwarding for `nomad` on port `4646` should bind to `127.0.0.1` and should allow you to use the nomad binary to post jobs directly. +Consul and Vault have also been port-forwarded and are available on `127.0.0.1` on ports `8500` and `8200` respectively. +Minio is started on port `9000` and shares the `/vagrant` (your repo) from within the vagrant box. + +|Service|URL|Token(s)| +|:---|:---:|:---:| +|Nomad| [http://10.0.3.10:4646](http://10.0.3.10:4646)|| +|Consul| [http://10.0.3.10:8500](http://10.0.3.10:8500)|master| +|Vault| [http://10.0.3.10:8200](http://10.0.3.10:8200)|master| +|Minio| [http://10.0.3.10:9000](http://10.0.3.10:9000)|minioadmin : minioadmin| + + +## Install Prerequisites + +```text +make install +``` + +The command, will install: +- [VirtualBox](https://www.virtualbox.org/) +- [Packer](https://www.packer.io/) +- [Vagrant](https://www.vagrantup.com/) with additional plugins +- [Additional software dependent on the OS (Linux, MacOS)](../install/Makefile) + +### Packages that needs to be pre-installed + +- [Make](https://man7.org/linux/man-pages/man1/make.1.html) +- [Git CLI](https://git-scm.com/book/en/v2/Getting-Started-The-Command-Line) + +#### MacOS Specific +- Virtualization must be enabled. [This is enabled by default on MacOS.](https://support.apple.com/en-us/HT203296) +- [Homebrew](https://brew.sh/) must be installed. + +#### Ubuntu Specific +- Virtualization must be enabled. [Error if it is not.](https://github.com/fredrikhgrelland/vagrant-hashistack/issues/136) +- Packages [gpg](http://manpages.ubuntu.com/manpages/xenial/man1/gpg.1.html) and [apt](http://manpages.ubuntu.com/manpages/bionic/man8/apt.8.html) must be installed. + +--- + +`NB` _Post installation you might need to reboot your system in order to start the virtual-provider (VirtualBox)_ + +--- + + +## Configuration + +### Startup Scheme +From a thousand foot view the startup scheme will: +1. Start the hashistack and MinIO +2. Run [playbook.yml](dev/ansible/playbook.yml), which in turn runs all ansible-playbooks inside [dev/ansible/](dev/ansible). + +> :bulb: Vagrantfile lines 8-11 run the first playbook on startup, and can be changed. + +> :bulb: Below is a detailed description of the _whole_ startup procedure, both user changeable and not. + +--- + +#### Detailed Startup Procedure +_box_ - Comes bundled with the box, not possible to change + +_system_ - Provided by the system in automated processes, not possible to change + +_user_ - Provided by the user to alter the box or template in some way + +|Seq number| What | Provided by | Description | +|:--:|:------------|:------------:|:-----| +|1 |`/home/vagrant/.env_default`|[ _box_ ]| default variables | +|2 |`/vagrant/.env`|[ _user_ ]| variables override, see [Pre-packaged Configuration Switches](#pre-packaged-configuration-switches) for details | +|3 |`/vagrant/.env_override`|[ _system_ ]| variables are overridden for test purposes | +|4 |`/vagrant/dev/vagrant/conf/pre_ansible.sh`|[ _user_ ]| script running before ansible bootstrap procedure, [details](dev/vagrant/conf/pre_bootstrap/README.md) | +|5 |`/vagrant/dev/vagrant/conf/pre_bootstrap/*.yml`|[ _user_ ]| pre bootstrap tasks, running before hashistack software starts, [details](dev/vagrant/conf/README.md) | +|6 |`/etc/ansible/bootstrap.yml`|[ _box_ ]| verify ansible variables and software configuration, run hashistack software and MinIO, & verify that it started correctly, [link](../ansible/bootstrap.yml) | +|7 |`/vagrant/conf/post_bootstrap/*.yml`|[ _user_ ]| poststart scripts, running after hashistack software has started, [details](dev/vagrant/conf/pre_bootstrap/README.md) | +|8 |`/vagrant/dev/conf/post_ansible.sh`|[ _user_ ]| script running after ansible bootstrap procedure, [details](dev/vagrant/conf/README.md) | +|9 |`/vagrant/ansible/*.yml`|[ _user_ ]| ansible tasks included in playbook, see [Pre-packaged Configuration Switches](#pre-packaged-configuration-switches) for details | + +--- + +### Pre and Post Hashistack Startup Procedure +#### Ansible Playbooks Pre and Post Hashistack Startup +You may change the hashistack configuration or add additional pre and post steps to the ansible startup procedure to match your needs. Detailed documentation in [dev/vagrant/conf/README.md](dev/vagrant/conf/README.md) -### Pre packaged configuration switches +#### Bash Scripts Pre and Post Ansible Playbook +In addition to ansible playbooks, you can also add bash-scripts that will be run before and/or after the ansible provisioning step. This is useful for doing deeper changes to the box pertaining to your needs. Detailed documentation in [dev/vagrant/conf/README.md](dev/vagrant/conf/README.md) + + +### Pre-packaged Configuration Switches -The box comes standard with a set of environment switches to simplify testing of different scenarios and enable staged development efforts. +The box comes [with a set of configuration switches controlled by env variables](https://github.com/fredrikhgrelland/vagrant-hashistack#configuration) to simplify testing of different scenarios and enable staged development efforts. +To change any of these values from their defaults, you may add the environment variable to [.env](dev/.env). -NB: All lowercase variables will automatically get a corresponding TF_VAR_ prepended variant for use directly in terraform. -To change from the default value, you may add the environment variable to [.env](dev/.env) +NB: All lowercase variables will automatically get a corresponding `TF_VAR_` prepended variant for use directly in terraform. [Script](../.github/action/create-env.py) #### Enterprise vs Open Source Software (OSS) -As long as Enterprise is not set to `true` the box will utilise OSS version of the binaries. +To use enterprise versions of the hashistack components set the software's corresponding Enterprise-variable to `true` (see below). #### Nomad @@ -39,7 +174,7 @@ As long as Enterprise is not set to `true` the box will utilise OSS version of t | | nomad_acl | true | | x | nomad_acl | false | -When ACLs in Nomad are enabled the bootstrap token will be available in vault under `secret/nomad/management-token` with the two key-value pairs `accessor-id` and `secret-id`. `secret-id` is the token itself. These can be accessed in several ways: +When ACLs are enabled in Nomad the bootstrap token will be available in vault under `secret/nomad/management-token` with the two key-value pairs `accessor-id` and `secret-id`. `secret-id` is the token itself. These can be accessed in several ways: - From inside the vagrant box with `vault kv get secret/nomad-bootstrap-token` - From local machine with `vagrant ssh -c vault kv get secret/nomad-bootstrap-token"` - By going to vault's UI on `localhost:8200`, and signing in with the root token. @@ -62,7 +197,7 @@ When ACLs in Nomad are enabled the bootstrap token will be available in vault un | | vault_enterprise | true | | x | vault_enterprise | false | -##### Consul secrets engine +##### Consul Secrets Engine If `consul_acl_default_policy` has value `deny`, it will also enable [consul secrets engine](https://www.vaultproject.io/docs/secrets/consul) in vault. Ansible will provision additional custom roles (admin-team, dev-team), [policies](../ansible/templates/consul-policies) and tokens for test purpose with different access level. @@ -76,15 +211,155 @@ vagrant ssh -c 'vault read consul/creds/dev-team' vagrant ssh -c 'vault read consul/creds/admin-team' ``` -*Tokens can be used to access UI (different access level depends on role) - -## Vagrant box life-cycle -1. `/home/vagrant/.env_default` - _preloaded_ - default variables -1. `vagrant/.env` - _user provided_ - variables override -1. `vagrant/.env_override` - _system provided_ - variables are overridden for test purposes -1. `vagrant/dev/vagrant/conf/pre_ansible.sh` - _user provided_ - script running before ansible bootstrap procedure -1. `vagrant/dev/vagrant/conf/pre_bootstrap/*.yml` - _user provided_ - pre bootstrap tasks, running before hashistack software runs and ready -1. `/etc/ansible/bootstrap.yml` - _preloaded_ - verify ansible variables and software configuration, run hashistack software & verify that it started correctly -1. `vagrant/conf/post_bootstrap/*.yml` - _user provided_ - poststart scripts, running after hasistack software runs and ready -1. `vagrant/dev/conf/pre_ansible.sh` - _user provided_ - script running after ansible bootstrap procedure -1. `vagrant/ansible/*.yml` - _user provided_ - ansible tasks included in playbook +> :bulb: Tokens can be used to access UI (different access level depends on policy attached to the token) + +### Vagrant Box Resources +If you get the error message `Dimension memory exhausted on 1 node` or `Dimension CPU exhausted on 1 node`, you might want to increase resources dedicated to your vagrant-box. +To overwrite the default resource-configuration you can add the lines +```hcl +Vagrant.configure("2") do |config| + config.vm.provider "virtualbox" do |vb| + vb.memory = 2048 + vb.cpu = 2 + end +end +``` +to the bottom of your [Vagrantfile](Vagrantfile), and change `vb.memory` and `vb.cpu` to suit your needs. Any configuration in [Vagrantfile](Vagrantfile) will overwrite the defaults if there is any. [More configuration options](https://www.vagrantup.com/docs/providers/virtualbox/configuration.html). + +> :bulb: The defaults can be found in [Vagrantfile.default](Vagrantfile.default). + + +## Usage +### Commands +There are several commands that help to run the vagrant-box: +- `make install` installs all prerequisites. Run once. + +- `make up` provisions a [vagrant-hashistack](https://github.com/fredrikhgrelland/vagrant-hashistack/) box on your machine. After the machine and hashistack are set up it will run the [Startup Scheme](#startup-scheme). + +- `make clean` takes down the provisioned box if there is any. + +- `make update` downloads the newest version of the [vagrant-hashistack box](https://github.com/fredrikhgrelland/vagrant-hashistack/) from [vagrantcloud](https://vagrantcloud.com/fredrikhgrelland/hashistack). + +- `make example` runs the example in [template_example/](template_example) + +> :bulb: For full info, check [`template/Makefile`](./Makefile). +> :warning: Makefile commands are not idempotent in the context of vagrant-box. You could face the error of port collisions. Most of the cases it could happen because of the vagrant box has already been running. Run `vagrant destroy -f` to destroy the box. + +Once vagrant-box is running, you can use other [options like the Nomad- and Terraform-CLIs to iterate over the deployment in the development stage](#iteration-of-the-development-process). + +### MinIO +Minio S3 can be used as a general artifact repository while building and testing within the scope of the vagrantbox to push, pull and store resources for further deployments. + +> :warning: Directory `/vagrant` is mounted to minio. Only first level of sub-directories become bucket names. + +Resource examples: +- docker images +- compiled binaries +- jar files +- etc... + +#### Pushing Resources To MinIO With Ansible (Docker image) +Push(archive) of docker image. +```yaml +# NB! Folder /vagrant is mounted to Minio +# Folder `dev` is going to be a bucket name +- name: Create tmp if it does not exist + file: + path: /vagrant/dev/tmp + state: directory + mode: '0755' + owner: vagrant + group: vagrant + +- name: Archive docker image + docker_image: + name: docker_image + tag: local + archive_path: /vagrant/dev/tmp/docker_image.tar + source: local +``` +[Full example](template_example/dev/ansible/01_build_docker_image.yml) + +#### Fetching Resources From MinIO With Nomad (Docker image) +> :bulb: [The artifact stanza](https://www.nomadproject.io/docs/job-specification/artifact) instructs Nomad to fetch and unpack a remote resource, such as a file, tarball, or binary. + +Example: +```hcl +task "web" { + driver = "docker" + artifact { + source = "s3::http://127.0.0.1:9000/dev/tmp/docker_image.tar" + options { + aws_access_key_id = "minioadmin" + aws_access_key_secret = "minioadmin" + } + } + config { + load = "docker_image.tar" + image = "docker_image:local" + } +} +``` +[Full example](./template_example/conf/nomad/countdash.hcl) + +### Iteration of the Development Process + +Once you start the box with one of the commands `make dev`, `make up` or `make example`, +you need a simple way how to continuously deploy development changes. + +There are several options: + +1. **From the local machine**. You can install Hashicorp binaries on the local machine, such as terraform and nomad. +Then you can deploy changes to the vagrant-box using these binaries. + +Example terraform: +```text +terraform init +terraform apply +``` + +Example nomad: +```text +nomad job run countdash.hcl +``` + +> :warning: _Your local binaries and the binaries in the box might not be the same versions, and may behave differently. [Box versions.](../ansible/group_vars/all/variables.yml) + +2. **Using vagrant**. Box instance has all binaries are installed and available in the PATH. +You can use `vagrant ssh` to place yourself inside of the vagrantbox and run commands. + +```text +# remote command execution +vagrant ssh default -c 'cd /vagrant; terraform init; terraform apply' + +# ssh inside the box, local command execution +vagrant ssh default +cd /vagrant +terraform init +terraform apply +``` + +> :bulb: `default` is the name of running VM. You could also use VM `id`. +To get vm `id` check `vagrant global-status`. + +## Test Configuration and Execution +The tests are run using [Github Actions](https://github.com/features/actions) feature which makes it possible to automate, customize, and execute the software development workflows right in the repository. We utilize the **matrix testing strategy** to cover all the possible and logical combinations of the different properties and values that the components support. The .env_override file is used by the tests to override the values that are available in the .env_default file, as well as the user configurable .env file. + + +As of today, the following tests are executed: + +| Test name | Consul Acl | Consul Acl Policy | Nomad Acl | Hashicorp binary +|:------------------------------------------------------------------------------------------:|:------------|:-------------------:|:-------------:|:---------------:| +| test (consul_acl_enabled, consul_acl_deny, nomad_acl_enabled, hashicorp_oss) | true | deny | true | Open source | +| test (consul_acl_enabled, consul_acl_deny, nomad_acl_enabled, hashicorp_enterprise) | true | deny | true | enterprise | +| test (consul_acl_enabled, consul_acl_deny, nomad_acl_disabled, hashicorp_oss) | true | deny | false | Open source | +| test (consul_acl_enabled, consul_acl_deny, nomad_acl_disabled, hashicorp_enterprise) | true | deny | false | enterprise | +| test (consul_acl_disabled, consul_acl_deny, nomad_acl_enabled, hashicorp_oss) | false | deny | true | Open source | +| test (consul_acl_disabled, consul_acl_deny, nomad_acl_enabled, hashicorp_enterprise) | false | deny | true | enterprise | +| test (consul_acl_disabled, consul_acl_deny, nomad_acl_disabled, hashicorp_oss) | false | deny | false | Open source | +| test (consul_acl_disabled, consul_acl_deny, nomad_acl_disabled, hashicorp_enterprise) | false | deny | false | enterprise | + +The latest test results can be looked up under the **Actions** tab. + +## If This Is in Your Own Repository +If you are reading this from your own repository you should _delete_ this `README.md`, fill out `README_template.md`, and rename `README_template.md` to `README.md`. \ No newline at end of file diff --git a/README_template.md b/README_template.md new file mode 100644 index 0000000..fb0e273 --- /dev/null +++ b/README_template.md @@ -0,0 +1,51 @@ + +

+ +> :warning: The text under headlines are examples, and should be removed. +> +> :warning: Remove this line and the one above! + +# Terraform-module name + +## Compatibility +List of compatible versions. E.g. +1. Terraform 0.12.2 or newer +2. Nomad 0.12 or newer + +## Usage + +### Requirements +#### Required software +1. Software 1 +2. Software 2 + +#### Other +Any other requirements. E.g. "This needs to be run on a Debian system" + +### Providers +A description of the providers that the module uses. E.g. "This module uses the [Nomad](https://registry.terraform.io/providers/hashicorp/nomad/latest/docs) and [Vault](https://registry.terraform.io/providers/hashicorp/vault/latest/docs) providers" + +## Inputs +|Name |Description |Type |Default |Required | +|:--|:--|:--|:-:|:-:| +| | |bool |true |yes | + +## Outputs +|Name |Description |Type |Default |Required | +|:--|:--|:--|:-:|:-:| +| | |bool |true |yes | + +### Example +Example-code that shows how to use the module, and, if applicable, its different use cases. +```hcl-terraform +module "example"{ + source = "./" +} +``` + +### Verifying setup +Description of expected end result and how to check it. E.g. "After a successful run Presto should be available at localhost:8080". + +## Authors + +## License \ No newline at end of file diff --git a/Vagrantfile b/Vagrantfile index 3ad3025..308f255 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -1,12 +1,2 @@ -Vagrant.configure("2") do |config| - config.vm.box = "fredrikhgrelland/hashistack" - config.vm.box_version = ">= 0.3, < 0.4" - config.vm.provider "virtualbox" do |vb| - vb.linked_clone = true - vb.memory = 2048 - end - config.vm.provision "ansible_local" do |ansible| - ansible.provisioning_path = "/vagrant/dev/ansible" - ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml - end -end +default_vagrantfile = "Vagrantfile.default" +load default_vagrantfile if File.exists?(default_vagrantfile) \ No newline at end of file diff --git a/Vagrantfile.default b/Vagrantfile.default new file mode 100644 index 0000000..bbcaf77 --- /dev/null +++ b/Vagrantfile.default @@ -0,0 +1,12 @@ +Vagrant.configure("2") do |config| + config.vm.box = "fredrikhgrelland/hashistack" + config.vm.box_version = ">= 0.3, < 0.4" + config.vm.provider "virtualbox" do |vb| + vb.linked_clone = true + vb.memory = 2048 + end + config.vm.provision "ansible_local" do |ansible| + ansible.provisioning_path = "/vagrant/dev/ansible" + ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml + end +end \ No newline at end of file diff --git a/dev/ansible/README.md b/dev/ansible/README.md index 256dfda..66e30be 100644 --- a/dev/ansible/README.md +++ b/dev/ansible/README.md @@ -3,4 +3,4 @@ This is a good starting point where you may run a series of sequential steps. In the example, we use ansible to initialize and start a terraform job which in turn starts nomad jobs in parallel. -Se [example](../../test_example/dev/ansible) \ No newline at end of file +See [example](../../template_example/dev/ansible) \ No newline at end of file diff --git a/dev/vagrant/conf/README.md b/dev/vagrant/conf/README.md index 7657570..dc83a86 100644 --- a/dev/vagrant/conf/README.md +++ b/dev/vagrant/conf/README.md @@ -11,6 +11,7 @@ You may add a `pre_ansible.sh` script file to this directory to run any alterati This might come handy if you need to change or replace that bootstrap process. For example you replacing the entire `/etc/ansible` directory. For most cases, you are probably looking to add configuration in [pre/poststart bootstrap](pre_bootstrap/README.md) + #### Post If you need to run additional commands after ansible bootstrap has happened, you may add a `post_ansible.sh`. This might come in handy if you would like to pat your self on tha back or test a recent configuration change before anything you might add to your own Vagrantfile. diff --git a/dev/vagrant/conf/consul/README.md b/dev/vagrant/conf/consul/README.md index a9b927b..6f53275 100644 --- a/dev/vagrant/conf/consul/README.md +++ b/dev/vagrant/conf/consul/README.md @@ -5,4 +5,4 @@ Any valid configuration added to this directory will append the configuration, i Adding a file `99-override.hcl` you will ensure it will be appended last, and 00-override.hcl will be read first. Any valid configuration from [https://www.consul.io/docs/agent/options.html#configuration_files](https://www.consul.io/docs/agent/options.html#configuration_files) will work. -See [example](../../../../test_example/dev/vagrant/conf/consul/99-override.hcl) +See [example](../../../../template_example/dev/vagrant/conf/consul/99-override.hcl) diff --git a/dev/vagrant/conf/nomad/README.md b/dev/vagrant/conf/nomad/README.md index c08e492..97e8b6f 100644 --- a/dev/vagrant/conf/nomad/README.md +++ b/dev/vagrant/conf/nomad/README.md @@ -16,4 +16,4 @@ client { } ``` -See [example](../../../../test_example/dev/vagrant/conf/nomad/99-override.hcl) \ No newline at end of file +See [example](../../../../template_example/dev/vagrant/conf/nomad/99-override.hcl) \ No newline at end of file diff --git a/dev/vagrant/conf/pre_bootstrap/README.md b/dev/vagrant/conf/pre_bootstrap/README.md index 4350d9a..2fa6252 100644 --- a/dev/vagrant/conf/pre_bootstrap/README.md +++ b/dev/vagrant/conf/pre_bootstrap/README.md @@ -12,4 +12,4 @@ The files e.g. 0-example.yml must only include pure ansible task syntax: msg: This would be a prestart task ``` -See [example](../../../../test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml) \ No newline at end of file +See [example](../../../../template_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml) \ No newline at end of file diff --git a/dev/vagrant/conf/vault/README.md b/dev/vagrant/conf/vault/README.md index 1819c1b..fb58a81 100644 --- a/dev/vagrant/conf/vault/README.md +++ b/dev/vagrant/conf/vault/README.md @@ -6,4 +6,4 @@ Any valid configuration added to this directory will append the configuration, i Adding a file `99-override.hcl` you will ensure it will be appended last, and 00-override.hcl will be read first. Any valid configuration from [https://www.vaultproject.io/docs/configuration](https://www.vaultproject.io/docs/configuration) will work. -See [example](../../../../test_example/dev/vagrant/conf/vault/99-override.hcl) \ No newline at end of file +See [example](../../../../template_example/dev/vagrant/conf/vault/99-override.hcl) \ No newline at end of file diff --git a/docker/README.md b/docker/README.md index 2228af7..d1ca9db 100644 --- a/docker/README.md +++ b/docker/README.md @@ -9,8 +9,8 @@ If you have docker installed on your machine, you may `cd docker; docker build - This image can be built and operated behind a corporate proxy where the base os needs to trust a custom CA. While building locally using the Makefile, you may set the environment variable CUSTOM_CA to a custom .crt file in order to import it into the docker image. See [conf/certificates](conf/certificates) -See [../test_example/docker/Dockerfile](../test_example/docker/Dockerfile) for examples on how to import and trust CA for centos/debian/alpine based docker images. +See [../template_example/docker/Dockerfile](../template_example/docker/Dockerfile) for examples on how to import and trust CA for centos/debian/alpine based docker images. ## Building and testing the docker-image within the vagrant-hashistack box -We advise you to build and test your docker image within the hashistack eco-system. Running `make test` will launch the [default playbook](../dev/ansible/playbook.yml) inside the box, and [test_example/](../test_example/) shows a simple build process for building and running the docker image using this. Refer to books in [test_example/dev/ansible](../test_example/dev/ansible) to see details. +We advise you to build and test your docker image within the hashistack eco-system. Running `make test` will launch the [default playbook](../dev/ansible/playbook.yml) inside the box, and [template_example/](../template_example/) shows a simple build process for building and running the docker image using this. Refer to books in [template_example/dev/ansible](../template_example/dev/ansible) to see details. diff --git a/example/README.md b/example/README.md index 37fe4f2..3634ac2 100644 --- a/example/README.md +++ b/example/README.md @@ -1,3 +1,5 @@ # This is a terraform module example +The current directory contains terraform related files that use the module in `../`. See [template_example](../template_example/example/). -TODO: explain \ No newline at end of file +## References +- [Creating Modules - official terraform documentation](https://www.terraform.io/docs/modules/index.html) diff --git a/test_example/.env b/template_example/.env similarity index 100% rename from test_example/.env rename to template_example/.env diff --git a/template_example/Vagrantfile b/template_example/Vagrantfile new file mode 100644 index 0000000..80d35e8 --- /dev/null +++ b/template_example/Vagrantfile @@ -0,0 +1,2 @@ +default_vagrantfile = "../Vagrantfile.default" +load default_vagrantfile if File.exists?(default_vagrantfile) \ No newline at end of file diff --git a/test_example/conf/nomad/countdash.hcl b/template_example/conf/nomad/countdash.hcl similarity index 100% rename from test_example/conf/nomad/countdash.hcl rename to template_example/conf/nomad/countdash.hcl diff --git a/test_example/dev/ansible/01_build_docker_image.yml b/template_example/dev/ansible/01_build_docker_image.yml similarity index 83% rename from test_example/dev/ansible/01_build_docker_image.yml rename to template_example/dev/ansible/01_build_docker_image.yml index db4e7cd..372ec0d 100644 --- a/test_example/dev/ansible/01_build_docker_image.yml +++ b/template_example/dev/ansible/01_build_docker_image.yml @@ -16,6 +16,8 @@ TEST_DOWNLOAD_BUILD_ARGUMENT: https://nrk.no source: build +# NB! Folder /vagrant is mounted to Minio +# Folder `dev` is going to be a bucket name - name: Create tmp if it does not exist file: path: /vagrant/dev/tmp @@ -28,5 +30,6 @@ docker_image: name: docker_image tag: local + http_timeout: 120 archive_path: /vagrant/dev/tmp/docker_image.tar - source: local \ No newline at end of file + source: local diff --git a/test_example/dev/ansible/02_run_terraform.yml b/template_example/dev/ansible/02_run_terraform.yml similarity index 100% rename from test_example/dev/ansible/02_run_terraform.yml rename to template_example/dev/ansible/02_run_terraform.yml diff --git a/test_example/dev/ansible/playbook.yml b/template_example/dev/ansible/playbook.yml similarity index 100% rename from test_example/dev/ansible/playbook.yml rename to template_example/dev/ansible/playbook.yml diff --git a/test_example/dev/vagrant/conf/consul/99-override.hcl b/template_example/dev/vagrant/conf/consul/99-override.hcl similarity index 100% rename from test_example/dev/vagrant/conf/consul/99-override.hcl rename to template_example/dev/vagrant/conf/consul/99-override.hcl diff --git a/test_example/dev/vagrant/conf/nomad/99-override.hcl b/template_example/dev/vagrant/conf/nomad/99-override.hcl similarity index 100% rename from test_example/dev/vagrant/conf/nomad/99-override.hcl rename to template_example/dev/vagrant/conf/nomad/99-override.hcl diff --git a/test_example/dev/vagrant/conf/post_ansible.sh b/template_example/dev/vagrant/conf/post_ansible.sh similarity index 100% rename from test_example/dev/vagrant/conf/post_ansible.sh rename to template_example/dev/vagrant/conf/post_ansible.sh diff --git a/test_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml b/template_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml similarity index 100% rename from test_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml rename to template_example/dev/vagrant/conf/post_bootstrap/00-poststart-example.yml diff --git a/test_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml b/template_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml similarity index 100% rename from test_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml rename to template_example/dev/vagrant/conf/post_bootstrap/01-poststart-example.yml diff --git a/test_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml b/template_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml similarity index 100% rename from test_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml rename to template_example/dev/vagrant/conf/post_bootstrap/02-poststart-example.yml diff --git a/test_example/dev/vagrant/conf/pre_ansible.sh b/template_example/dev/vagrant/conf/pre_ansible.sh similarity index 100% rename from test_example/dev/vagrant/conf/pre_ansible.sh rename to template_example/dev/vagrant/conf/pre_ansible.sh diff --git a/test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml b/template_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml similarity index 100% rename from test_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml rename to template_example/dev/vagrant/conf/pre_bootstrap/00-prestart-example.yml diff --git a/test_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml b/template_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml similarity index 100% rename from test_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml rename to template_example/dev/vagrant/conf/pre_bootstrap/01-prestart-example.yml diff --git a/test_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml b/template_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml similarity index 100% rename from test_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml rename to template_example/dev/vagrant/conf/pre_bootstrap/02-prestart-example.yml diff --git a/test_example/dev/vagrant/conf/vault/99-override.hcl b/template_example/dev/vagrant/conf/vault/99-override.hcl similarity index 100% rename from test_example/dev/vagrant/conf/vault/99-override.hcl rename to template_example/dev/vagrant/conf/vault/99-override.hcl diff --git a/test_example/docker/.dockerignore b/template_example/docker/.dockerignore similarity index 100% rename from test_example/docker/.dockerignore rename to template_example/docker/.dockerignore diff --git a/test_example/docker/Dockerfile b/template_example/docker/Dockerfile similarity index 100% rename from test_example/docker/Dockerfile rename to template_example/docker/Dockerfile diff --git a/test_example/docker/conf/certificates/.gitignore b/template_example/docker/conf/certificates/.gitignore similarity index 100% rename from test_example/docker/conf/certificates/.gitignore rename to template_example/docker/conf/certificates/.gitignore diff --git a/test_example/docker/conf/certificates/README.md b/template_example/docker/conf/certificates/README.md similarity index 100% rename from test_example/docker/conf/certificates/README.md rename to template_example/docker/conf/certificates/README.md diff --git a/test_example/example/main.tf b/template_example/example/main.tf similarity index 100% rename from test_example/example/main.tf rename to template_example/example/main.tf diff --git a/test_example/example/nomad_acl_test.tf b/template_example/example/nomad_acl_test.tf similarity index 100% rename from test_example/example/nomad_acl_test.tf rename to template_example/example/nomad_acl_test.tf diff --git a/test_example/example/variables.tf b/template_example/example/variables.tf similarity index 100% rename from test_example/example/variables.tf rename to template_example/example/variables.tf diff --git a/test_example/main.tf b/template_example/main.tf similarity index 100% rename from test_example/main.tf rename to template_example/main.tf diff --git a/test_example/outputs.tf b/template_example/outputs.tf similarity index 100% rename from test_example/outputs.tf rename to template_example/outputs.tf diff --git a/test_example/variables.tf b/template_example/variables.tf similarity index 100% rename from test_example/variables.tf rename to template_example/variables.tf diff --git a/test_example/Vagrantfile b/test_example/Vagrantfile deleted file mode 100644 index e9d6a3d..0000000 --- a/test_example/Vagrantfile +++ /dev/null @@ -1,11 +0,0 @@ -Vagrant.configure("2") do |config| - config.vm.box = "fredrikhgrelland/hashistack" - config.vm.provider "virtualbox" do |vb| - vb.linked_clone = true - vb.memory = 2048 - end - config.vm.provision "ansible_local" do |ansible| - ansible.provisioning_path = "/vagrant/dev/ansible" - ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml - end -end From 8d27435094af1afc23c0d8e4729490da1cc4b1c1 Mon Sep 17 00:00:00 2001 From: Fredrik Hoem Grelland <40291976+fredrikhgrelland@users.noreply.github.com> Date: Fri, 28 Aug 2020 17:02:10 +0200 Subject: [PATCH 10/10] Update Vagrantfile.default --- Vagrantfile.default | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Vagrantfile.default b/Vagrantfile.default index bbcaf77..2236b18 100644 --- a/Vagrantfile.default +++ b/Vagrantfile.default @@ -1,6 +1,6 @@ Vagrant.configure("2") do |config| config.vm.box = "fredrikhgrelland/hashistack" - config.vm.box_version = ">= 0.3, < 0.4" + config.vm.box_version = ">= 0.4, < 0.5" config.vm.provider "virtualbox" do |vb| vb.linked_clone = true vb.memory = 2048 @@ -9,4 +9,4 @@ Vagrant.configure("2") do |config| ansible.provisioning_path = "/vagrant/dev/ansible" ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml end -end \ No newline at end of file +end