Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: refactor schema tests #1320

Merged
merged 1 commit into from
Dec 14, 2023
Merged

test: refactor schema tests #1320

merged 1 commit into from
Dec 14, 2023

Conversation

jonbretman
Copy link
Member

@jonbretman jonbretman commented Dec 12, 2023

Overview

This PR introduces a new way of writing tests for validation errors that uses comments in the schema file.

A simple example:

model Foo {
    fields {
        //expect-error:9:13:E009:field name has an unsupported type Bar
        name Bar
    }
}

The //expect-error comment is parsed to extract a start column (9), end column (13), error code (E009), and error message (the last bit). There is then an assertion that a validation error exists on the following line that matches this expectation.

Migration

I didn't manually change all 766 files 😆 I converted all the existing tests using a little script.

Missing expectations

If a validation error is found for which there is no //expect-error comment the test will fail with a message like this:

    --- FAIL: TestValidation/foo.keel (0.00s)
        schema_test.go:137:   Unexpected: 3:9:13:E009:field name has an unsupported type Bar

Missing errors

If no validation error is found that matches a //expect-error comment the test will fail with a message like this:

    --- FAIL: TestValidation/foo.keel (0.00s)
        schema_test.go:132:   Expected:   4:4:9:13:MyErrorCode:something went wrong

Easier to see where the error occurs

Because the start and end column is part of the expectation it's much easier to see where an error is pointing to, and then hopefully we avoid weird issues like this:

Screenshot 2023-12-12 at 17 30 37

One test, many errors

In our current setup we don't tend to put too many errors in each test case, which I think is partly because the errors.json file starts to get a bit hard to manage.

With this approach it's much easier to have lots of expected errors in one schema file which I hope will lead us to have less test cases but with each test case having many expectations.

Duplicate Errors

This approach makes it much easier to spot where we have duplicate error reporting, for example:

//expect-error:9:14:RelationshipError:Cannot use @unique on a repeated model field
//expect-error:22:29:TypeError:@unique is not permitted on has many relationships
posts Post[] @unique

Here we have two errors that basically say the same thing, which isn't very helpful.

Proto tests

I think at this stage we should move away from the proto tests e.g. the valid schema examples. Any valid use-case should be covered by our integration tests.

@jonbretman jonbretman requested a review from a team December 12, 2023 17:35
@jonbretman jonbretman merged commit 65494a1 into main Dec 14, 2023
10 checks passed
@jonbretman jonbretman deleted the schema-tests branch December 14, 2023 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant