feat: support JSON Schema as input for a Type #1159

TizzySaurus · 2024-10-02T15:42:39Z

Closes #729.

What

This PR adds @ark/jsonschema, which is a library enabling users to convert a valid JSON Schema schema into an ArkType type.

API

Below is a basic sample-use of @ark/jsonschema:

import { parseJsonSchema } from "@ark/jsonschema"

const type = parseJsonSchema({type: "string"})
// ^? 
// NB: `type` is correctly inferred as `Type<string>`, and can now be used as-per any other ArkType type

Remaining Work Before Merging

For some reason the innerParseJsonSchema const is giving a type error Type "string" is not assignable to type "never". It seems my explicit type definition is somehow wrong 🤷
For some reason parseJsonSchema({type: "array", items: {type: "string"}}) raises TypeError: this.Schema1Apply is not a function despite working against an earlier ArkType version (probably a bug in ArkType that I've not yet delved into to properly give details).
Similar to above, const t = parseJsonSchema({oneOf: [{type: "string"}, {type: "number"}]}) raises TypeError: this.CompositionKeywords1Apply is not a function:
Tests are only partially implemented.
- I wasn't entirely confident on what should/shouldn't be tested, and where things should/shouldn't be tested, so am waiting on feedback for this before finishing the tests.
  - For example, I have tests for {type: "string", "maxLength": 5} and {type: "string", "pattern": "es"}, but not {type: "string", "maxLength": 5, "pattern": "es"}. Is it worth having tests for the various "permutations"? I think this is perhaps overly excessive?
- All test files also give this linting errors stating it can't find the @ark/jsonschema package:
Haven't yet properly filled out ark/jsonschema/CHANGELOG.md.
Constraints are correctly inferred, but should ideally be displayed more "prettily" (see below image):

I'd like to get custom type-errors working again, however the current problem is that calling parseJsonSchema with an invalid JSON Schema always results in a excessively deep and possibly infinite error, making this [seemingly] impossible.

Known Limitations (things I didn't implement from JSON Schema spec)

No support for JSON Schema dependencies keyword on objects.
No support for JSON Schema if/else/then.
- Sort of not required, since you can write the same (admittedly more verbosely) by combining any/oneOf/allOf/not.
multipleOf on numbers only supports integer divisors, due to ArkType only supporting integer divisors.

…text type

…rror)

…rus/arktype into json-schema-to-arktype

ark/jsonschema/json.ts

TizzySaurus · 2024-10-02T20:53:19Z

.changeset/brave-plums-clap.md

Not sure how this change got here... I guess it can probably be safely deleted?

Yeah you can remove this for sure, I'm not even using changesets anymore

Removed in f7bd853 👍

ssalbdivad

Looks really thorough and this will definitely provide a ton of value!

I didn't review the types in a ton of detail because I'm still worried it's a lot of added complexity for very marginal benefit, although I know it's fun to implement 😄

I think the best path forward regardless of whether we eventually decide to add inference is to get a version of this merged without the inference since there would likely need to be a lot of changes to get the same validation/perf/accuracy as the core parser.

Maybe keep the logic around on your branch so we can use it as a reference if we do eventually decide to include JSON schema inference.

ssalbdivad · 2024-10-04T22:11:37Z

.changeset/brave-plums-clap.md

Yeah you can remove this for sure, I'm not even using changesets anymore

ssalbdivad · 2024-10-05T14:57:39Z

ark/jsonschema/object.ts

+				)
+			}
+		})
+		return errors.length === 0


In each of these cases, instead of keeping track of the errors with a false[], you can just check ctx.hasError() when needed.

Also, by default it's important to try and use expected + actual for error messages so that they can compose in unions. message is only for the full error with contextualized path which generally should just be handled by the default.

It can be a bit tricky to word certain errors so they fit in Must be {expected} (was {actual}), but there's usually a reasonable way. Try to restructure as many of these errors as possible to use those.

ssalbdivad · 2024-10-05T15:01:39Z

ark/jsonschema/object.ts

+	if ("properties" in jsonSchema) {
+		if ("required" in jsonSchema) {
+			if (jsonSchema.required.length !== new Set(jsonSchema.required).size) {
+				ctx.reject({


I'd write this as a standalone function called duplicateElementsOf that iterates over an array and each time it encounters an element it's seen, it adds a DuplicateData like { value: unknown, indices: number[] }.

This could be added to @ark/util in arrays.ts since it would be general, but it would also help to write a much more descriptive error message here

ssalbdivad · 2024-10-05T15:04:06Z

ark/jsonschema/object.ts

+	return {
+		optionalKeys: optionalKeys.map(key => ({
+			key,
+			value: innerParseJsonSchema.assert(jsonSchema.properties![key]).internal


In general, instead of needing to reassert this later, once you validate a property is present, I'd assign it to its own variable like properties right away.

Although in this case it's not clear from the logic in the function why properties would always be defined. If that's something external where we know either required or properties will be defined when we call this function or something, it's best to refactor it in a way where those external assumptions don't exist.

ssalbdivad · 2024-10-05T15:05:24Z

ark/jsonschema/object.ts

+const parseAdditionalProperties = (jsonSchema: JsonSchema.ObjectSchema) => {
+	if (!("additionalProperties" in jsonSchema)) return
+
+	const properties = Object.keys(jsonSchema.properties ?? {})


This would probably be clearer (and very slightly more performant) as something like:

const properties = jsonSchema.properties ? Object.keys(jsonSchema.properties) : []

ssalbdivad · 2024-10-05T15:21:44Z

ark/jsonschema/composition.ts

+				return matchedValidator !== undefined
+			}) as Type<unknown>
+		)
+			// TODO: Theoretically this shouldn't be necessary due to above `ctx.mustBe` in narrow???


Description can be useful for introspectability, but yeah generally you'd either want to use that or a custom error in narrow if you need additional context.

Since it looks like oneOfValidators is already known before the type is created, it's important that stuff like mustBe that would run on every failed validation is just evaluated when the type is created, and shouldn't duplicate the logic of the description.

ssalbdivad · 2024-10-05T15:22:35Z

ark/jsonschema/any.ts

+	}
+
+	if ("enum" in jsonSchema) {
+		return rootSchema(


Best just use type.enumerated here

Yep, this was done before that existed but migrated to it in 03ba515

ssalbdivad · 2024-10-05T15:23:00Z

ark/jsonschema/any.ts

+				"Provided JSON Schema cannot have both 'const' and 'enum' keywords."
+			)
+		}
+		return rootSchema({ unit: jsonSchema.const }) as unknown as Type


Use type.unit (no reason to drop down into @ark/schema in a case like this)

Done in 03ba515

ssalbdivad · 2024-10-05T15:23:30Z

ark/jsonschema/array.ts

+
+		arktypeArraySchema.predicate = predicates
+
+		return rootSchema(arktypeArraySchema) as unknown as Type<JsonSchema.Json[]>


If you ever are casting a return multiple times like this, best to just cast it to never and annotate the return type

Fixed this and other instances of the same thing in 01e296c

ssalbdivad · 2024-10-05T15:30:42Z

ark/jsonschema/array.ts

+						...inferJsonSchema<arraySchema["additionalItems"]>[]
+					]
+				>
+			:	// JSON Schema spec explicitly says that additionalItems MUST be ignored if items is not an array, and it's NOT an error


We don't have an obligation to support everything that's technically valid JSON schema if it seems like a clear mistake.

I'd rather throw here

If it was a "should" then I'd agree, but because it's "must" I'd prefer to keep it as-is.

If "items" is an array of schemas, validation succeeds if every instance element at a position greater than the size of "items" validates against "additionalItems".

Otherwise, "additionalItems" MUST be ignored, as the "items" schema (possibly the default value of an empty schema) is applied to all elements.

https://json-schema.org/draft-07/draft-handrews-json-schema-validation-01#rfc.section.6.4.2

…chema keywords

… value

…pe into json-schema-to-arktype

…repo

…values

ssalbdivad and others added 30 commits June 5, 2024 01:23

improve wording of type example

2c2bdf1

Merge branch 'main' into v2docs-6

a3fd9cd

runtime error messages increase font size

01583b0

rename RegexNode=>PatternNode

9e3e772

add changeset

85e5558

bump deps

886e643

fix morph with alias child

00baf27

add changelogs

2a94bcd

cyclic scope

98533ad

allow overriding aliases

ae9342e

add constraints to validation scope

a2ed10f

remove constraints type parameter constraints

170bc6e

remove broken cyclic rereferences test

9d0bf28

cleanup anonymous constraint display

8628a77

Merge branch 'main' into json-schema-to-arktype

77562e3

Merge branch 'main' into json-schema-to-arktype

b7c38a0

Merge branch 'main' into json-schema-to-arktype

0e5b2da

Merge branch 'main' into json-schema-to-arktype

cf77faa

Export from ark/schema/shared/traversal.ts so can import TraversalCon…

4c4e61f

…text type

Add parsing & types for array JSON Schema (w/ excessively deep type e…

934cd36

…rror)

Add parsing & types for number/integer JSON Schema

d3bc90d

Add parsing & types for string JSON Schema

da95a37

Add ArkType Scope representing JSON Schema schemas

4d230c1

Initialise ark/jsonschema package

48a654a

Add parsing & types for object JSON Schema

b343bfe

Add parsing for allOf + anyOf + not + oneOf JSON Schemas

98e9c03

Add parsing for const + enum JSON Schemas

6dbdbcd

Add core parseJsonSchema with parsing and type inference logic

7248b27

Add ark/jsonschema/index.ts entry level to @ark/jsonschema

79c0a79

Add @ark/jsonschema to root package.json devDepencies

28da503

TizzySaurus added 6 commits September 23, 2024 09:11

Merge branch 'main' into json-schema-to-arktype

4127422

Export type Out from ark/type/keywords/inference.ts

474a849

Fix @ark/jsonschema use of arktype constraint inference

95e1b40

Preliminary tests for @ark/jsonschema

2e21553

Merge branch 'main' into json-schema-to-arktype

2d945b3

Merge branch 'json-schema-to-arktype' of personal.github.com:TizzySau…

94cdef2

…rus/arktype into json-schema-to-arktype

TizzySaurus changed the title ~~Add JSON Schema to ArkType conversion~~ feat: Add JSON Schema to ArkType conversion Oct 2, 2024

TizzySaurus added 6 commits October 2, 2024 20:59

Add ts-ignore comment for excessively deep tuple spread

6903094

Add preliminary README.md for @ark/jsonschema

d65fa30

Linting

8e97d87

Fix example in README.md

877f36e

Remove extra double-slash in comment

d59336c

Remove old TODO

e150832

TizzySaurus commented Oct 2, 2024

View reviewed changes

ark/jsonschema/json.ts Outdated Show resolved Hide resolved

Remove old comment

f625bc9

TizzySaurus commented Oct 2, 2024

View reviewed changes

TizzySaurus added 2 commits October 5, 2024 12:13

Migrate .js imports to .ts imports

dd130e8

Remove accidentally added ark/jsonschema/del.ts file

4598d02

ssalbdivad requested changes Oct 5, 2024

View reviewed changes

ssalbdivad changed the title ~~feat: Add JSON Schema to ArkType conversion~~ feat: JSON Schema to ArkType conversion Oct 7, 2024

ssalbdivad changed the title ~~feat: JSON Schema to ArkType conversion~~ feat: support JSON Schema as input for a Type Oct 7, 2024

TizzySaurus added 9 commits October 13, 2024 11:07

Remove changeset

f7bd853

Use type.enumerated and type.unit utils for 'const' and 'enum' JSON S…

03ba515

…chema keywords

Specify return type of function rather than double casting the return…

01e296c

… value

Make variable assignment clearer & remove debug log statement

dc43e17

Merge branch 'json-schema-to-arktype' of github.com:TizzySaurus/arkty…

be4fff2

…pe into json-schema-to-arktype

Update @ark/jsonschema 'scripts' and 'exports' to match new style in …

b6e5c70

…repo

Formatting

423ba89

Use conflatenateAll util instead of manually filtering out undefined …

2fe1b7a

…values

Remove accidentally added debugging file

d0e612b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support JSON Schema as input for a Type #1159

feat: support JSON Schema as input for a Type #1159

TizzySaurus commented Oct 2, 2024 •

edited

Loading

TizzySaurus Oct 2, 2024

ssalbdivad Oct 4, 2024

TizzySaurus Oct 13, 2024

ssalbdivad left a comment

ssalbdivad Oct 4, 2024

ssalbdivad Oct 5, 2024

ssalbdivad Oct 5, 2024

ssalbdivad Oct 5, 2024

ssalbdivad Oct 5, 2024

TizzySaurus Oct 13, 2024

ssalbdivad Oct 5, 2024

ssalbdivad Oct 5, 2024

TizzySaurus Oct 13, 2024

ssalbdivad Oct 5, 2024

TizzySaurus Oct 13, 2024

ssalbdivad Oct 5, 2024

TizzySaurus Oct 13, 2024

ssalbdivad Oct 5, 2024

TizzySaurus Oct 13, 2024 •

edited

Loading


		arktypeArraySchema.predicate = predicates

		return rootSchema(arktypeArraySchema) as unknown as Type<JsonSchema.Json[]>

feat: support JSON Schema as input for a Type #1159

Are you sure you want to change the base?

feat: support JSON Schema as input for a Type #1159

Conversation

TizzySaurus commented Oct 2, 2024 • edited Loading

What

API

Remaining Work Before Merging

Known Limitations (things I didn't implement from JSON Schema spec)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssalbdivad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TizzySaurus Oct 13, 2024 • edited Loading

Choose a reason for hiding this comment

TizzySaurus commented Oct 2, 2024 •

edited

Loading

TizzySaurus Oct 13, 2024 •

edited

Loading