add `ziti edge quickstart` to start an ephemeral environment #1284

dovholuknf · 2023-09-01T12:46:32Z

Goal was to make an 'even quicker quickstart'. Since it's using go, it's usable in Windows too which is a plus (I guess lol)

The command will:

emit a minimal pki
emit a controller config file
start the controller in a go routine
emit a router config file
add a router to the network based on the router config file
enroll the router
start the router in a go routine
wait for ctrl-c/other signal and remove the temp dir

NOT configurable at all, using mostly/only defaulted values. Adds "localhost" and "hostname" to the PKI so that it's available from other network machines as long as those machines can connect. It uses the same commands as the bash-based quickstart and is supposed to respect the env vars used for the existing quickstats (but i didn't test it fully yet)

closess #1298

qrkourier

[   6.381]    INFO ziti/ziti/cmd/helpers.StandardErrorMessage: Connection error: Get https://localhost:1280/.well-known/est/cacerts: dial tcp [::1]:1280: connect: connection refused
The connection to the server localhost:1280 was refused - did you specify the right host or port?

My "localhost" resolves to loopback's IPv6 address, but the controller's client API isn't listening there. Should it be?

ziti/cmd/create/create_config_router_common.go

qrkourier · 2023-09-01T13:15:26Z

After re-adding the sleep:

[   6.021]    INFO edge/controller/subcmd.NewEdgeInitializeCmd.func2: Ziti Edge initialization complete
Controller running... Configuring and starting Router...
[   6.987]   ERROR ziti/ziti/controller.run: {version=[v0.0.0] arch=[amd64] build-date=[2020-01-01 01:01:01] revision=[local] go-version=[go1.20] os=[linux] error=[unable to open controller database [/tmp/quickstart1098242889/db/ctrl.db]: timeout]} error starting ziti-controller
panic: unable to open controller database [/tmp/quickstart1098242889/db/ctrl.db]: timeout

goroutine 99 [running]:
github.com/openziti/ziti/ziti/controller.run(0xc00244c900?, {0xc0021e8550, 0x1, 0x1?})
        /home/kbingham/Sites/netfoundry/github/ziti/ziti/controller/run.go:56 +0xc6e
github.com/spf13/cobra.(*Command).execute(0xc00244c900, {0xc0021e84f0, 0x1, 0x1})
        /home/kbingham/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:944 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xc00244c900)
        /home/kbingham/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/kbingham/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:992
github.com/openziti/ziti/ziti/cmd/edge.run.func1()
        /home/kbingham/Sites/netfoundry/github/ziti/ziti/cmd/edge/quickstart.go:107 +0x146
created by github.com/openziti/ziti/ziti/cmd/edge.run
        /home/kbingham/Sites/netfoundry/github/ziti/ziti/cmd/edge/quickstart.go:102 +0x71f

qrkourier · 2023-09-01T14:37:58Z

Let's run this before the fablab-smoketest: #1285 @dovholuknf @plorenz

qrkourier · 2023-09-01T16:12:31Z

After re-adding the sleep:

@dovholuknf It's not the same localhost/IPv6 error after re-adding the sleep. ziti/controller/run.go says:

unable to open controller database

The CI job gets the same result.

…cker-quickstart

try again. it's changed to hostname. it'll be helpful to know if you still have an ipv6 issue

qrkourier · 2023-09-05T14:28:00Z

@dovholuknf After "update dep" I no longer get a panic, and have these open files.

❯ pgrep -f quickstart|xargs sudo lsof -Pnp 2>/dev/null
COMMAND     PID     USER   FD      TYPE             DEVICE  SIZE/OFF     NODE NAME
ziti    1601070 kbingham  cwd       DIR              253,1      4096 22479168 /home/kbingham/Sites/netfoundry/github/ziti
ziti    1601070 kbingham  rtd       DIR              253,1      4096        2 /
ziti    1601070 kbingham  txt       REG              253,1 116066848 22451491 /home/kbingham/Sites/netfoundry/github/ziti/build/ziti
ziti    1601070 kbingham  mem-W     REG               0,37    524288    29923 /tmp/quickstart3742302670/db/ctrl.db
ziti    1601070 kbingham  mem       REG              253,1   2216304 16685451 /usr/lib/x86_64-linux-gnu/libc.so.6
ziti    1601070 kbingham  mem       REG              253,1    240936 16684477 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
ziti    1601070 kbingham    0u      CHR              136,0       0t0        3 /dev/pts/0
ziti    1601070 kbingham    1u      CHR              136,0       0t0        3 /dev/pts/0
ziti    1601070 kbingham    2u      CHR              136,0       0t0        3 /dev/pts/0
ziti    1601070 kbingham    3uW     REG               0,37    524288    29923 /tmp/quickstart3742302670/db/ctrl.db
ziti    1601070 kbingham    4u  a_inode               0,14         0       54 [eventpoll]
ziti    1601070 kbingham    5r     FIFO               0,13       0t0  4700178 pipe
ziti    1601070 kbingham    6w     FIFO               0,13       0t0  4700178 pipe
ziti    1601070 kbingham    7u     unix 0xffff9569b43c8880       0t0  4700191 /tmp/gops-agent.1601070.sock type=STREAM
ziti    1601070 kbingham    8u     IPv6            4700195       0t0      TCP *:6262 (LISTEN)
ziti    1601070 kbingham    9u     IPv6            4693691       0t0      TCP *:1280 (LISTEN)
ziti    1601070 kbingham   10u     sock                0,8       0t0  4686826 protocol: TCP
ziti    1601070 kbingham   17u     IPv6            4693707       0t0      TCP *:10080 (LISTEN)
ziti    1601070 kbingham   18u     IPv6            4693708       0t0      TCP *:3022 (LISTEN)
ziti    1601070 kbingham   21u     IPv4            4676371       0t0      TCP 127.0.0.1:52890->127.0.1.1:6262 (ESTABLISHED)
ziti    1601070 kbingham   24u     sock                0,8       0t0  4686827 protocol: TCP
ziti    1601070 kbingham   26u     IPv6            4698077       0t0      TCP 127.0.1.1:6262->127.0.0.1:52890 (ESTABLISHED)

However, the one-shot test panics.

❯ ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=localhost ZITI_ROUTER_NAME=localhost go test -v ./quickstart/test/quickstart_test.go
=== RUN   TestSimpleWebService                                                     
time="2023-09-05T10:23:48-04:00" level=info msg="connecting user: admin to https://localhost:1280"
time="2023-09-05T10:23:48-04:00" level=info msg="generating 4096 bit RSA key"
finding hostingRouterName:  localhost
--- FAIL: TestSimpleWebService (3.17s)
panic: runtime error: index out of range [0] with length 0 [recovered]                                                                                                
        panic: runtime error: index out of range [0] with length 0                                                                                                    
                                                                                                                                                                      
goroutine 6 [running]:                                                                                                                                                
testing.tRunner.func1.2({0xe9eca0, 0xc000122c18})                                                                                                                     
        /home/kbingham/goroot/src/testing/testing.go:1526 +0x24e                                                                                                      
testing.tRunner.func1()
        /home/kbingham/goroot/src/testing/testing.go:1529 +0x39f
panic({0xe9eca0, 0xc000122c18})   
        /home/kbingham/goroot/src/runtime/panic.go:884 +0x213
command-line-arguments.getIdentityByName(0xc0001031e0, {0xc000048291, 0x9})       
        /home/kbingham/Sites/netfoundry/github/ziti/quickstart/test/quickstart_test.go:286 +0x228                        
command-line-arguments.TestSimpleWebService(0xc0004e71e0?)                     
        /home/kbingham/Sites/netfoundry/github/ziti/quickstart/test/quickstart_test.go:132 +0xafc                                   
testing.tRunner(0xc0004e71e0, 0xf7c688)                                                                                                                               
        /home/kbingham/goroot/src/testing/testing.go:1576 +0x10b                                                                                                      
created by testing.(*T).Run                                                                                                                                           
        /home/kbingham/goroot/src/testing/testing.go:1629 +0x3ea                                                                                                      
FAIL    command-line-arguments  3.183s                                                                                                                                
FAIL

dovholuknf · 2023-09-06T04:15:48Z

i've refactored the manual test and added a totally independent test as well.

Use the previous/existing test with:

go clean -testcache
ZITI_CTRL_EDGE_ADVERTISED_ADDRESS=localhost \
ZITI_ROUTER_NAME=quickstart-router \
go test -tags "quickstart manual" ./ziti/cmd/edge/...

To run the new, totally self-contained test which will start a controller, start a router, run the "automated" quickstart test:

go clean -testcache
go test -tags "quickstart automated" ./ziti/cmd/edge/...

qrkourier · 2023-09-06T16:48:31Z

The quickstart manual test now works with ziti edge quickstart, but the automated test didn't.

❯ (set -euxo pipefail;
go clean -testcache;
go test -tags "quickstart automated" ./ziti/cmd/edge/...;
)
+/usr/bin/zsh:106> go clean -testcache
+/usr/bin/zsh:107> go test -tags 'quickstart automated' ./ziti/cmd/edge/...
?       github.com/openziti/ziti/ziti/cmd/edge  [no test files]

dovholuknf · 2023-09-06T17:12:57Z

you sure you're on the right branch? works on my machine. i even ran it in a docker container just to make sure.

qrkourier · 2023-09-06T17:15:25Z

I was on the wrong branch. 😞

…cker-quickstart

…copy mounted on the golang container in the quickstart's compose project

Run the quicker quickstart before fablab

.github/workflows/main.yml

…x build

.github/workflows/main.yml

run quickstart test as step in linux-build job

dismissing to make change

run quickstart automated test immediately

qrkourier

The CI workflow looks good to me, and I'm happy to launch ziti edge quickstart as is to start getting some feedback and exploring use cases.

…rg/x/crypto-0.5.0 Bump golang.org/x/crypto from 0.4.0 to 0.5.0

dovholuknf added 4 commits September 1, 2023 07:58

initial stab at making a quicker quickstart

6025dc5

catch signals and remove temp files

fce94f3

lol, put the signal handling in the right function

ae795c2

a few more tweaks after a bit of test

d63cbd0

dovholuknf requested review from a team as code owners September 1, 2023 12:46

remove sleep that isn't needed

83bcd6c

qrkourier previously requested changes Sep 1, 2023

View reviewed changes

ziti/cmd/create/create_config_router_common.go Outdated Show resolved Hide resolved

needed the sleep for now

3a8b8f6

dovholuknf added 4 commits September 1, 2023 13:46

update edge dep

b51282c

replace sleep with better mechanism

4ed54f1

Merge remote-tracking branch 'origin/release-next' into ziti-edge-qui…

4d4fc3c

…cker-quickstart

lint fixes pointed out by github and import/alias cleanup

c7e0f74

dovholuknf added 2 commits September 1, 2023 23:02

revert go.* changes

6afdf1c

update dep

2f2056e

dovholuknf added 3 commits September 5, 2023 13:59

merge release next

f2a4911

working towards a compartmentalized test for quickstart

5f7a629

update quickstart and add/update/upgrade/reuse tests

293e303

add changelog too

ddf1a00

hopefully placate the linter

e020595

more linter updates

d6580cb

dovholuknf and others added 8 commits September 6, 2023 15:41

tidy zititest too

1d3221a

Merge remote-tracking branch 'origin/release-next' into ziti-edge-qui…

376ff53

…cker-quickstart

hopefully 'fix' the compose-test

764851b

add a ci job to run the Go quicker quickstart

7a8d351

run fablab-smoketest after quickstart

ca6d348

run the manual quickstart test with GOPATH and the ziti repo working …

3e4b4d2

…copy mounted on the golang container in the quickstart's compose project

stop commenting the .env file with testing-only vars

e3e0325

Merge pull request #1299 from openziti/test-quicker-quickstart-redux

55724ff

Run the quicker quickstart before fablab

qrkourier previously requested changes Sep 6, 2023

View reviewed changes

.github/workflows/main.yml Outdated Show resolved Hide resolved

.github/workflows/main.yml Outdated Show resolved Hide resolved

run quickstart automated test immediately instead of waiting for Linu…

e782d83

…x build

plorenz reviewed Sep 7, 2023

View reviewed changes

.github/workflows/main.yml Show resolved Hide resolved

qrkourier added 2 commits September 6, 2023 21:02

run quickstart automated as step in linux-build job

9d2fb0d

Merge pull request #1301 from openziti/quickstart-run-in-linux-build

1d6e845

run quickstart test as step in linux-build job

Merge pull request #1300 from openziti/quickstart-no-always

e984a67

run quickstart automated test immediately

dovholuknf requested a review from plorenz September 8, 2023 00:50

qrkourier approved these changes Sep 8, 2023

View reviewed changes

plorenz approved these changes Sep 8, 2023

View reviewed changes

dovholuknf merged commit f2e2738 into release-next Sep 8, 2023
17 of 18 checks passed

dovholuknf deleted the ziti-edge-quicker-quickstart branch September 8, 2023 16:19

dovholuknf mentioned this pull request Sep 27, 2023

add ephemeral quickstart to ziti cli #1298

Closed

plorenz added a commit that referenced this pull request Sep 28, 2023

Merge pull request #1284 from openziti/dependabot/go_modules/golang.o…

4529b27

…rg/x/crypto-0.5.0 Bump golang.org/x/crypto from 0.4.0 to 0.5.0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add `ziti edge quickstart` to start an ephemeral environment #1284

add `ziti edge quickstart` to start an ephemeral environment #1284

dovholuknf commented Sep 1, 2023 •

edited

Loading

qrkourier left a comment •

edited

Loading

qrkourier commented Sep 1, 2023

qrkourier commented Sep 1, 2023 •

edited

Loading

qrkourier commented Sep 1, 2023

qrkourier commented Sep 5, 2023

dovholuknf commented Sep 6, 2023

qrkourier commented Sep 6, 2023

dovholuknf commented Sep 6, 2023

qrkourier commented Sep 6, 2023

qrkourier left a comment

add ziti edge quickstart to start an ephemeral environment #1284

add ziti edge quickstart to start an ephemeral environment #1284

Conversation

dovholuknf commented Sep 1, 2023 • edited Loading

qrkourier left a comment • edited Loading

Choose a reason for hiding this comment

qrkourier commented Sep 1, 2023

qrkourier commented Sep 1, 2023 • edited Loading

qrkourier commented Sep 1, 2023

qrkourier commented Sep 5, 2023

dovholuknf commented Sep 6, 2023

qrkourier commented Sep 6, 2023

dovholuknf commented Sep 6, 2023

qrkourier commented Sep 6, 2023

qrkourier left a comment

Choose a reason for hiding this comment

add `ziti edge quickstart` to start an ephemeral environment #1284

add `ziti edge quickstart` to start an ephemeral environment #1284

dovholuknf commented Sep 1, 2023 •

edited

Loading

qrkourier left a comment •

edited

Loading

qrkourier commented Sep 1, 2023 •

edited

Loading