Pilot integration tests

[wallrj]

I want to be able to quickly test that:

• Pilot runs
• logs info and error messages
• creates Events.
• Waits for PilotResource
• Identifies the correct pilot resource
• Ignores unrelated PilotResource
• Updates PilotResource status
• Starts a sub-process database
• Logs output of its sub-process
• Detects problems in the database
• Syncs the configuration of the database with the PilotResource configuration
• responds to SIGTERM
• Drains the database before it shuts down.
• Shuts down.

I think it should be possible to do this all in a single test process.

• Start an Kubernetes API server in a goroutine or start a kubectl proxy sub process or launch the tests in a pod.
• Start the Pilot in a goroutine
• Wait for the pilot to shutdown before exiting the test.

/kind bug

[munnerz]

So I'm not sure if this should all be a single huge e2e test. It'll make it difficult to determine what is actually failing without digging through log messages. Ideally, log messages should help us 'drill down' once we know what is failing (i.e. through a particular scenario failing).

We've got some prior work on integration tests in cert-manager already that it'd be good to use as a 'template' (for consistency between projects).

For example, 'detects problems in the database' is a huge umbrella, and really needs to be broken out into multiple subtests e.g.: "Elasticsearch process correctly identifies that xyz occurred and logs an Event to the apiserver".

"Logs output of it's subprocess" can be deferred to a unit test where we mock the io.Reader and io.Writer etc.

Unless that's not what you meant by a "single test process" 😄