Author: agavra | Release Target: TBD | Status: Discussion | Discussion: TBD
tl;dr: Introduce Yet Another Testing Tool (YATT) that is written and driven primarily by SQL-based syntax to add coverage for test cases that require interleaving statements and inserts. Eventually, the goal is to replace the external facing ksql-test-runner and maybe (R)QTT tests.
Our existing testing infrastructure is powerful and can handle many of our existing use cases. It, however, lacks the flexibility to be useful when testing query upgrades: neither framework supports interleaving statements/inserts/asserts. This KLIP proposes a new, imperative, testing tool to cover that use case and describes a way to replace the existing tools with this functionality.
The list below enumerates the motivating design principles:
- the tool should support imperative tests, allowing interleaved statements/inserts/asserts
- the tool should allow tests to be written primarily in SQL, avoiding special-syntax directives when possible
- the tool should leverage as much of the ksqlDB engine as possible, keeping testing-specific code to a minimum. Whenever we require new functionality for the testing tool, we will first consider supporting it directly in ksqlDB.
- the tool should run many tests quickly, with the ability to parallelize test runs
Another minor benefit of a SQL-based testing tool is to allow inline comments describing statements and asserts.
Why introduce YATT (Yet Another Testing Tool) when we already have three existing options? There are a few factors here:
- the primary motivation is practical in nature: We need interleaved testing for the query upgrades work, and building this into QTT would require essentially an entire rewrite of both the existing tool and the historical test execution while maintaining or migrating all the historical plans. In order to deliver query upgrades (see KLIP-28) quickly, we will begin work on YATT while maintaining support for QTT tests.
- the ksql-test-runner and (R)QTT have, over time, diverged significantly from the ksqlDB engine and require lots of custom code (the ksqldb-functional-test module has 11k lines of Java). Specifically, the custom JSON format requires maintaining a parallel, testing-only framework and serde to convert inputs/outputs to their desired serialization format. Instead, we should leverage the production SQL expression functionality to produce data.
- providing a SQL-driven testing tool will mesh better with our product offering, allowing users to write tests without every leaving the "SQL mindset"
This KLIP covers:
- design of the test file format
- design of the testing-specific directives
- outline features that need to be implemented in order to replace (R)QTT/testing-tool
This KLIP does not cover:
- implementation of a migration path from (R)QTT/testing-tool, but we plan to deprecate and remove the ksql-test-runner over the next few releases. At the time when we remove it, we will build some convenience scripts to help generate YATT sql files.
- generating historical plans from YATT sql tests
The intention of this KLIP is to provide a solid foundation for YATT to eventually replace the other testing tools and to make sure we are working towards a cleaner code base instead of littering it with testing code that needs to be maintained and will eventually slow down our development velocity. That being said, it does not cover all the details required, or provide a timeline, to migrate the existing tools.
The best way to describe the testing tool is by a motivating example, such as the test file below:
---------------------------------------------------------------------------------------------------
--@test: dml - stream - add filter
---------------------------------------------------------------------------------------------------
CREATE STREAM foo (id VARCHAR KEY, col1 INT) WITH (kafka_topic='foo', value_format='JSON');
CREATE STREAM bar AS SELECT * FROM foo;
INSERT INTO foo (rowtime, id, col1) VALUES (1, '1', 1);
ASSERT DATA bar OFFSET 1 (rowtime, id, col1) VALUES (1, '1', 1);
CREATE OR REPLACE STREAM bar AS SELECT * FROM foo WHERE col1 = 123;
INSERT INTO foo (rowtime, id, col1) VALUES (2, '2', 2);
INSERT INTO foo (rowtime, id, col1) VALUES (3, '3', 123);
ASSERT DATA bar OFFSET 2 (rowtime, id, col1) VALUES (3, '3', 123);
ASSERT STREAM bar (id VARCHAR KEY, col1 INT) WITH (kafka_topic='BAR', value_format='JSON');
---------------------------------------------------------------------------------------------------
--@test: dml - stream - change column
--@expected.error: io.confluent.ksql.util.KsqlException
--@expected.message: "Cannot upgrade data source: DataSource '`BAR`' has schema ..."
---------------------------------------------------------------------------------------------------
CREATE STREAM foo (id VARCHAR KEY, col1 VARCHAR, col2 VARCHAR)
WITH (kafka_topic='foo', value_format='JSON');
-- the below operation is invalid because "col2" will be missing from the
-- schema after renaming it to col3
CREATE STREAM bar AS SELECT id, col1, col2 FROM foo;
CREATE OR REPLACE STREAM bar AS SELECT id, col1, col2 AS col3 FROM foo;YATT will accept as a parameter a directory containing testing files, and will run all the tests in the directory in a single JVM. Alternatively, it can accept a single test file to run just a limited subset of tests as well as which tests to run within the file based on regex.
Each test file can contain one or more tests, separated by the --@test directive. This will
improve on a pain point of the existing ksql-test-runner, which requires three files for a single
test. The test name is a concatenation of the test file time and the contents of the --@test
directive.
Then, the test can contain any number of interleaved statements that are supported by the testing
tool. Data will be inserted into topics using INSERT INTO and asserts will be executed using the
new ASSERT syntax (described below).
There will be additional "meta" directives that will allow for testing functionality like expected exceptions.
ASSERT is the primary way to ensure conditions in YATT, and will be parsed and implemented
using ksqlDB's AstBuilder to make sure YATT can leverage the existing expression parsing support.
At first, ASSERT statements will fail if they are sent to a production server, but we may consider
allowing ASSERT in the future in order to allow REPL-based testing/experimentation.
The ASSERT DATA statement asserts that data exists at an optionally provided offset (or otherwise
sequential based on the last ASSERT DATA) with the given values. If the ASSERT DATA does not
specify certain columns they will be ignored (allowing users to assert only subset of the columns
match what they expect).
The values will be created in the same way that INSERT INTO creates values:
ASSERT DATA source_name [OFFSET at_offset] ( { column_name } [, ...] ) VALUES ( value [, ...] );The ASSERT DATA statement will allow users to specify psuedocolumns as well, such as ROWTIME,
and when ksqlDB supports constructs such as PARTITION and/or headers, YATT will inherit these
psuedocolumns as well.
The ASSERT NO DATA statement will allow users to ensure that no more data exists in a specified
source:
ASSERT NO DATA source_name [OFFSET from_offset];The ASSERT (TABLE | STREAM) statement asserts that the given stream or table has the specified
columns and physical properties.
ASSERT (STREAM | TABLE) source_name ( { column_name data_type [[PRIMARY] KEY] } [, ...] )
WITH (property_name = expression [, ...] );The ASSERT TYPE statement ensures that a custom type has the expected type. This is especially
useful when chaining multiple CREATE TYPE statements together and asserting the types later
in the chain are correct.
ASSERT TYPE type_name AS type;It is possible to use a meta directive (see below) in order to drive asserts as well. This was considered and rejected because it would require additional parsing and more testing-specific code to convert to expressions and pass into the codegen, which diverges from the motivating principle of minimizing testing-only code.
Some functionality cannot be addressed in standard SQL syntax and will instead be supported using meta directives. Some examples of these directives:
- The
[email protected]and[email protected]directives will indicate that the below test expects an exception to be thrown with the specified message and type. - The
[email protected]directive will ensure that any topics matching the denylist will not be present at the end of the test execution.
These are a set of checks that will happen without any directive or ASSERT statement. Some
checks that we may consider including:
- Processing Log Check: this check will ensure that there are no failures in the processing log.
If the test case intends to check that the processing log contains certain entries, this check
can be disabled on a test-by-test basis and assert via
ASSERT DATAon the processing log stream.
There are some features that we need to implement in ksqlDB's SQL language in order to put YATT on par with the existing testing tools. Some of those are outlined here:
- Windowed Keys: when we have structured key support, we can
ASSERT DATAand just provide the struct as the value. Until then, windowed keys will either be unsupported or stringified. - Tombstones: we will implement
DELETE FROM table WHERE key = valuesyntax to allow tombstones to be inserted into tables, both for ksqlDB and YATT. - Null Values for Streams: insert
nullvalues into streams will not be supported for the first iteration of YATT. Eventually, if we want to support this, we can add a directive like[email protected]: topic keyto produce anullvalued record into a specific topic/source. - Other Unsupported Inserts: there are other types of inserts that ksqlDB doesn't allow for at the
moment (e.g.
enumsupport or binary formats). This is only somewhat a regression from the existing (R)QTT tests, so we may or may not decide to support it in YATT.
One of the bigger concerns with the existing ksql-test-runner is that it's helpful in letting us know that an error happened, but not so much why it happened or even at times what exactly happened. To avoid this pitfall, the errors will include:
- the assert statement that failed, with a link to the line that failed in the test file
- the actual data, and the expected data
- the ability to add
PRINTstatements to debug the issue further by printing topic contents
When data comparisons fail, we will leverage our SQL formatters to display the errors the
same way we display SELECT statements - in a tabular, easy-to-digest way. YATT will alternatively
take in a flag that allows failure messages to be output as machine-readable JSON, and this
will leverage our existing REST API JSON formats and be helpful for users reporting test failures
programmatically via some CI pipeline.
Most users that will be using this testing tool as part of a CI/CD pipeline will want to run
exactly the scripts that are being deployed. To support this, the RUN SCRIPT language feature
will be supported and can be interleaved just like any other statement in the testing tool. This
is especially helpful when testing query upgrades and schema evolution.
In this use case below, we can imagine that the user has a directory structure containing all changes to their production cluster:
+ dir
|--+ 1
| |--- 1_create_foo_bar.sql
| |--- 1_test.sql
|
|--+ 2
| |--- 2_update_bar.sql
| |--- 2_test.sql
Then they write the following files, and ask the tool to recursively look through the top-level
directory and run all tests. This test utilizes some syntax (--@depends) for demonstration of
how we could use this in a CI/CD pipeline to chain tests together as well, pairing tests with
their corresponding production sql files.
-- 1_create_foo_bar.sql
CREATE STREAM foo (id INT KEY, col1 VARCHAR) WITH (...)
CREATE STREAM bar AS SELECT * from FOO;-- 1_test.sql
--@test: 1_test
RUN SCRIPT '1_create_foo_bar.sql`;
ASSERT STREAM foo (...) WITH (...);
INSERT INTO foo ...;-- 2_update_bar.sql
CREATE OR REPLACE STREAM bar AS SELECT * FROM foo WHERE foo.id > 0;--@test: 2_test
--@depends: 1/1_test.sql
RUN_SCRIPT '2_update_bar.sql';
INSERT INTO foo ...;
ASSERT DATA bar ...;There are two ways of implementing such a testing tool:
- backed by multiple topology test drivers, one for each query
- backed by a real Kafka cluster
The initial implementation will be scoped to just the former implementation in order to ensure speedy test execution. Since YATT will help drive development, it is necessary that the tests run quickly - both in batch and as single test cases.
There are some extensions to YATT that we may want to consider. I'm listing them here because I'm currently in the mindset of thinking about them, and I don't want to forget!
- BYO Kafka: we may want to allow users to "plug in their own kafka" cluster and run these tests against a real Kafka cluster. This would allow users to produce whatever data they want outside of this tool, mitigating the limitations described above. It also would allow them to "debug" further when something doesn't go the way they want by examining the input/output topics through ksqlDB.
- YaaS: (YAAT as a Service) we may want to consider a deployment option where YAAT just watches a directory in some long-running deployment, running new tests whenever a directory changes.
This is a testing tool! It is self-testing in the sense that we will be using it to test query upgrades and assert that it works the way that it is expected to. We will also make sure to have negative tests to ensure that it should fail when it is supposed to fail.
- (S) Basic testing functionality required for testing query upgrades
- (S) Supporting chained statement execution (i.e. one CSAS as the input to another)
- (S) Supported meta-directives other than
[email protected] - (M) Extending ksqlDB to support the language gaps
- (M) Historical plans
- (L) Migrate (R)QTT use cases
Documentation for this testing tool will be another entry into the documentation suite we currently maintain. It will contain an enumeration of the special directives as well as any unsupported operations.
N/A