How to automatically test your rules

[lkleeven]

Hi all,

We just started working with spectral and started on writing our own OpenAPI specification rules. So far things are going fine, but some of the rules are hard to define and need quite a few manual tests to verify. Given that we want to share these rules among a large number of teams we want to be sure that our rules work as expected, and keep working as expected.

Is there any way of setting up automatic tests that test your rules? For instance run against a spec and check that a rule is/isn't triggered and do this for a number of specs.
If there isn't are there any plans for providing such a feature?

[P0lip]

Hey!
At the very moment there's no official tool you could leverage to perform automated testing. However, you can still find inspirations in our own tests.
I'll share some ideas with you, perhaps one of them turns out to be useful.

Idea 1

Assuming you have Node.js installed, you could copy over our rule tester, which we designed to be somewhat similar to the ESLint's rule tester.
The implementation can be found over here.
The package @stoplight/spectral-test-utils is not published to NPM, thus you'll need to copy over its code either.
It's available right here - just copy the whole directory.
Then, once all files are copied, you'll need to tweak the implementation of that rule tester we previously mentioned.
Fortunately, the only "bit" you'll need to change is the path to your ruleset.

https://github.com/stoplightio/spectral/blob/develop/packages/rulesets/src/__tests__/__helpers__/tester.ts#L6#L9 should be changed to reflect the location of your actual ruleset, i.e.

import myRuleset from './myRuleset';

type Ruleset = typeof myRuleset;

and https://github.com/stoplightio/spectral/blob/develop/packages/rulesets/src/__tests__/__helpers__/tester.ts#L46#L47

The caveat here is that the myRuleset cannot be a YAML/JSON, but a JS ruleset.
If you don't want to rewrite your ruleset, you can use the ruleset migrator.

Alright, now that we have the basic setup, let's evaluate the actual tests.
Plenty of examples can be found here and here.
The syntax is relatively simple.
You provide an array of tests cases, where each test case looks as follows

  {
    name: 'a directly self-referencing document from the filesystem', // some pretty name that'll displayed by the test runner
    document: {}, // the fixture you want to use as an input, 
    errors: [], // no errors = empty array
  },

  {
    name: 'an indirectly self-referencing document from the filesystem',
    document:{},
    errors: [
      { // this is a partial of the ISpectralDiagnostic
        message: 'Potentially unused component has been detected.',
        path: ['components', 'schemas', 'Unhooked'],
        severity: DiagnosticSeverity.Warning,
      },
    ],
  },
];

@stoplight/spectral-test-utils supports fs & https mocking, but I'm not sure if that'll be particularly useful.
If you do want to mock the requests / stub the fs, you can find examples over here.

Alright, now once we have some tests in place, let's pick some test-runner.
I suggest Jest as it doesn't require very little to no setup.

Here's how the example project could look like
spectral-testing-playground.zip

Idea 2

Copy test-harness.
The readme there should be fairly comprehensive, but if you have any questions, LMK.

Idea 3

If you are unable to install Node.js, then creating an own script that spawns spectral binary can be taken into the consideration.
I'd structure the project in the following way:

- rules
--- my-rule
----- spectral.js
----- test-1-input
----- test-1-expected-output
----- test-2-input
----- test-2-expected-output

And then you could write a script that iterates over the directory, and runs Spectral.
The test-1-input would be what Spectral CLI receives, and the other file is the expected output, so basically what Spectral CLI prints to stdout/stderr.
This could be run as so

cat $input | spectral lint

and then you'd need to use something to compare the output against $input-expected-output

spectral.js is the ruleset, which would be probably looking as follows

import myRuleset from '../ruleset';
export default {
  extends: [[myRuleset, 'off']],
  rules: {
     'my-rule': true,
  }
};

---

LMK if that makes sense and if there's anything I could help you with.

[EdOlson-Morgan]

Thanks @P0lip - we're looking to do the same thing so will check out Ideas 1 and 2 above.

[jamietanna]

Sorry for bumping an old thread with a bit of self-promotion, but I've written about how I did it myself on my blog and to make things a little easier have created a test harness package off the back of it for others to use - would be happy to hear any feedback you have!

A sample repo of this in use is https://gitlab.com/jamietanna/spectral-jest and https://github.com/co-cddo/api-standards-linting

[travisgosselin]

Very helpful and much appreciated.

[jamietanna]

Also coming back to this to say that APIs You Won't Hate have built a much nicer alternative to mine from the other comment, which can be found documented here