What is the rational behind making v0.34 Type.Ref not allow passing in a schema and inferring the type?

[stuft2]

The reason I've used Type.Ref is to shorten the generated json schemas. In some cases it removes thousands of lines of repeated code.

Why add the breaking change to Type.Ref in v0.34? Why not allow backwards compatibility? Why not support type inference of reference types?

Wrapping Type.Ref with Type.Unsafe to get type inference is at least a workaround but I'd really like to understand what made this change necessary in the first place.

Thanks!

[sinclairzx81]

@stuft2 Hi,

Why add the breaking change to Type.Ref in v0.34?

The change was made in support of the new Module feature which provides enhanced (and Safe) inference support for Ref. Unfortunately the previous Ref signature wasn't able to be repurposed in a way I felt comfortable maintaining moving forward (given conflicting designs). The updated signature on 0.34.0 is able to be made backwards compatible to 0.33.0, but the inverse wasn't possible. The call was made to drop the previous signature, but provide the fallback option via Unsafe.

Why not allow backwards compatibility?

You can replicate the old TRef using Unsafe (the following should work as before)

import { Type, TSchema, Static, TUnsafe } from '@sinclair/typebox'

// Old Ref
function Ref<T extends TSchema>(schema: T): TUnsafe<Static<T>> {
  return Type.Unsafe<Static<T>>(Type.Ref(schema.$id!))
}

// ...

const T = Type.String({ $id: 'T' })

const R = Ref(T) // { $ref: 'T' }

Why not support type inference of reference types?

Type inference is supported for Ref, but only within the context of a Module where it's possible to prove to the type system that the target type is correct.

const Module = Type.Module({

  A: Type.Number(),             // Auto assigned { $id: 'A' } from property name
  
  R: Type.Ref('A'),             // { $ref: 'A' }
})

// ... 

type R = Static<typeof R>       // type R = number

const R = Module.Import('R')

Inference is also supported for mutually and self recursive types (something that wasn't possible before under the previous TRef design)

const Module = Type.Module({
  Node: Type.Object({
    id: Type.String(),
    nodes: Type.Array(Type.Ref('Node'))
  })
})

// ... 

type Node = Static<typeof Node>       //  type Node = {
                                      //    id: string;
                                      //    nodes: ...[];
                                      // }

const Node = Module.Import('Node')

---

Future

The updates to Module and Ref are part of a larger body of work to establish better alignment with the TS type system, specifically to enable order independent type definitions to reference (to be) defined types. The updates to Ref enable this (as they just accept strings to remote targets). By consequence of this, Ref is able to be used as a parse target for the new Syntax Type feature (which also support out of order type referencing)

TypeScript Link Here

import { Static } from '@sinclair/typebox'
import { Parse } from '@sinclair/typebox/syntax'

// B references A before A is defined
const Foo = Parse(`module Foo {

  type B =
    Pick<A, 'id'> & 
    Partial<Omit<A, 'id'>>

  type A = {
    id: string
    nodes: A[]
  }

}`)

const B = Foo.Import('B')

type B = Static<typeof B>             // type B = {
                                      //     id: string;
                                      // } & {
                                      //     nodes?: {
                                      //         id: string;
                                      //         nodes: ...[];
                                      //     }[] | undefined;
                                      // }

The Module, Ref (Computed) and Syntax features will be expanded upon throughout 0.34.x.

---

I apologize if this change has caused any inconvenience. I had been hesitant to implement it for several months but ultimately decided to proceed because I believe the new Module features address more problems than the breaking change to Ref. I hope the update hasn't caused too many problems

Happy to discuss more on this thread.
Cheers
S

[sinclairzx81]

@stuft2 FYI, Have just published a revert of the Ref(Schema) signature under a @deprecated comment. This is available on 0.34.7. There had been a few users impacted by the signature change, so in response to that, I'll keep the signature around for the 0.34.x revisions, and look at dropping in the next minor revision. Hopefully this helps smooth over the changes.

Cheers
S

[liu-ronny]

@sinclairzx81 Is your recommendation to strictly use the new Module feature going forward when needing to use Ref? If so, can you offer some guidance on how to go about doing this?

I'm struggling with two problems.

Schema reuse

I have some schemas (e.g., enums) that are reused by other schemas that may not be related to each other. I can't use Ref unless I do one of the following:

1. Redefine the referenced schemas within the relevant Modules
2. Define one giant Module that holds all my schemas
3. Use Unsafe as a workaround

None of these options feel quite right. Or maybe I'm missing something obvious?

Auto-generated OpenAPI documentation

I'm using TypeBox in a Fastify application, where I use plugins to generate OpenAPI documentation for my API routes. My understanding is that a Module is a schema with a $defs property, and using Import results in an object with all those $defs and a $ref to the specified subschema. How do I pair that with fastify.addSchema to make sure my OpenAPI docs are correct?

[sinclairzx81]

@liu-ronny Hi,

Is your recommendation to strictly use the new Module feature going forward when needing to use Ref? If so, can you offer some guidance on how to go about doing this?

Ref is still intended to be used in isolation without Module (so you don't need to upgrade schematics to use Modules). However previous (and future) implementations using Ref will need to be mindful of a few subtle aspects with regards to the Ref API.

const T = Type.Object({
  x: Type.Number(),
  y: Type.Number()
}, { $id: 'T' })

// ------------------------------------------
// Before 0.34.0
// ------------------------------------------

const R = Type.Ref(T)                                     // const R = { $ref: 'T' }

type R = Static<typeof R>                                 // type R = { x: number, y: number }

// ------------------------------------------
// After 0.34.0
// ------------------------------------------

const R = Type.Ref('T')                                   // const R = { $ref: 'T' }

type R = Static<typeof R>                                 // type R = unknown

CHANGES: The 0.34.0 version of Ref requires you to pass the $ref parameter via string (where-as before the $ref from read from the $id property of the schema). Additionally, because Ref now accepts a string argument, the inferred type for Ref must now be unknown as the Ref has no information about the target type.

Ajv & Fastify

By updating Ref to take the string parameter, this is all that's required to make things work from a runtime / validation standpoint (using fastify.addSchema(...) etc). However type inference will be lost. See sections below.

Type Inference

To retain inference, the recommended approach is to wrap the Ref in an Unsafe. The Unsafe type is used to override the default inference for any given TypeBox type. In the case of Ref, we need to use Unsafe to override the default inference of unknown to be that of the target type.

// ------------------------------------------
// Before 0.34.0
// ------------------------------------------

const R = Type.Ref(T)                                     // const R = { $ref: 'T' }

type R = Static<typeof R>                                 // type R = { x: number, y: number }

// ------------------------------------------
// After 0.34.0
// ------------------------------------------

const R = Type.Unsafe<Static<typeof T>>(Type.Ref('T'))    // const R = { $ref: 'T' }

type R = Static<typeof R>                                 // type R = { x: number, y: number }

UnsafeRef

As with other TypeBox types, you can construct a generic utility type that can provide a factory for the wrapped Unsafe Ref. The following reimplements pre-0.34.0 Ref via the utility type UnsafeRef.

import { Type, TSchema, TUnsafe, Static } from '@sinclair/typebox'

// Replicates pre-0.34.0 Ref
function UnsafeRef<Type extends TSchema>(type: Type): TUnsafe<Static<Type>> {
  return Type.Unsafe<Static<Type>>(Type.Ref(type.$id!))
}

// ...

const T = Type.Object({
  x: Type.Number(),
  y: Type.Number(),
}, { $id: 'T' })

const R = UnsafeRef(T)                                    // const R = { $ref: 'T' }

type R = Static<typeof R>                                 // type R = { x: number, y: number }

Why Ref changed

The main reason Ref changed was due to the following not being possible with the previous API

const R = Type.Ref(T) // Error: Cannot use variable before initialization.

const T = Type.String({ $id: 'T' })

This is solved in 0.34.0

const R = Type.Unsafe<Static<typeof T>>(Type.Ref('T')) // this is fine.

const T = Type.String({ $id: 'T' })

... so, the main reason for the change is to enable non-order dependent types to be created with reference types. The change here paves the way for upgrades to TypeBox's Recursive type implementation (due on the next release). Some of these upgrades are available via Module where deferred dereferencing via string parameter is facilitating both singular and mutual recursive schemas & inference.

Unfortunately, the change did force Ref to infer as unknown (requiring the Unsafe wrap). This is largely due to Ref types being repurposed as a kind of "deferred" / "lazy" look up when used in the context of other structures (namely Modules). When Ref is used insolation, the Unsafe wrapping facilitates an "explicit" and "immediate" (non-lazy) dereference of the target structure.

---

Hope this helps. There was a fair amount of consideration that went into the Ref type update, but outside the signature change (passing string instead of schemas), they should largely work as before. The need for the extra explicit mechanisms (Unsafe) may be ironed out in future revisions (for example, making UnsafeRef a built in type (possibly named something else)), but for now, you will need wrap the Ref in Unsafe manually.

Happy to discuss more on this thread if you have any other questions
Cheers!
S

[liu-ronny]

Thanks for the detailed response.

Let's see if I've got this right.

Ref can still be used outside the context of a Module. However, since v0.34.0, it only takes a string argument. Outside of a Module, the result of Ref infers as type unknown.

In order to get a type-safe Ref outside of a Module, we can write a utility function that uses TUnsafe to mimic the behavior of Ref before v0.34.0. This allows an underlying $ref to be inferred as the input schema type.

The Module feature was introduced to allow for out of order references and self/recursive references. Modules also enable further alignment with TypeScript (e.g., we can parse an out of order TypeScript definition into a TypeBox Module). There's no need to put all our schemas in Modules.

Is this an accurate understanding of how everything works?

I'm still a bit confused about how Module works with fastify.addSchema. Let's say I introduce a Module into my code. Below is a contrived example.

import { Type, TypeBoxTypeProvider } from '@fastify/type-provider-typebox';
import { FastifyPluginAsync } from 'fastify';

const routes: FastifyPluginAsync = async (fastify, opts) => {
  const server = fastify.withTypeProvider<TypeBoxTypeProvider>();

  const Module = Type.Module({
    A: Type.Number(),
    R: Type.Ref('A'),
  });

  server.get(
    '/my-endpoint',
    {
      schema: {
        response: {
          '200': Module.Import('R'),
        },
      },
    },
    async function (request, reply) {
      return 1;
    },
  );
};

The response is type-safe and works as intended with TypeBoxTypeProvider. However, if I look at the openapi.json file generated by fastify-swagger, I see the below.

{
  "description": "Default Response",
  "content": {
    "application/json": {
      "schema": {
        "$defs": {
          "A": {
            "type": "number",
            "title": "A"
          },
          "R": {
            "$ref": "#/components/schemas/def-29",
            "title": "R"
          }
        },
        "$ref": "#/components/schemas/def-30"
      }
    }
  }
}

Without adding anything from the Module using addSchema, the def-29 and def-30 schemas don't exist. As a result, my docs show a null response schema for the /my-endpoint route. If I try to add the entire Module, addSchema complains that it doesn't have an $id, which I can't add because there's no options argument for Module.

Do you have a recommendation for what to do here? Is this a plugin-specific problem with fastify-swagger?

[sinclairzx81]

@liu-ronny Hiya

---

Is this an accurate understanding of how everything works?

Yes, you understand everything correctly.

---

The response is type-safe and works as intended with TypeBoxTypeProvider. However, if I look at the openapi.json file generated by fastify-swagger, I see the below.

If using Modules, do not specify the $id as it will be auto generated internal to the Module (matching the internal types property name). When specifying the $ref, it must reference one of the properties names inside of the Module.

const Module = Type.Module({
  A: Type.String(),                          // the $id of 'A' will be auto generated based on the property name.
  B: Type.Ref('A')                           // B can reference 'A' property name.
})

const B = Module.Import('B')                 // const B = {
                                             //   $defs: {
                                             //     A: { type: "string", $id: "A" },
                                             //     B: { $ref: "A", $id: "B" }
                                             //   },
                                             //   $ref: "B"  <- Import Name
                                             // }

So, the following reference would be invalid in the context of Modules.

"$ref": "#/components/schemas/def-29",

If you use Modules (which you shouldn't need to), you are opting-in to let TypeBox organize, structure, reference and de-reference schematics for you. The feature works by placing hard constraints on how $defs schematics can be structured (the same as other TB types), and from this it's able to perform auto dereferencing, deferred inferencing on your behalf.

This probably isn't going to be compatible with existing open Ref implementations unless they conform to the same structures mandated by Module.

---

Do you have a recommendation for what to do here? Is this a plugin-specific problem with fastify-swagger?

Yes, just don't use Modules :) Just keep on with Ref. Just create the following function.

// Or name this function however you like.
function LegacyRef<Type extends TSchema>(type: Type): TUnsafe<Static<Type>> {
  return Type.Unsafe<Static<Type>>(Type.Ref(type.$id!))
}

Then update all existing Type.Ref(schema) instances to be LegacyRef(schema).

const T = Type.Object({ ... }, { $id: '/components/schemas/def-29' })

const R = LegacyRef(T) // Replacing Type.Ref(T)

This will replicate the pre-0.34.0 Ref. Nothing else should need to change.

---

Keep in mind that TypeBox is NOT deprecating Ref. Open Ref implementations (as per your setup) are going to be supported forever. You should perceive the Module functionality as an opt-in extension feature that utilizes Ref. It is not the "be all and end all" way of handling references in TypeBox. The only thing that has changed is the signature for Ref, but if have a look at the LegacyRef type above, you can see there really isn't much of a change at all.

Let me know this helps. If you have a codebase where the Ref types are still causing you problems. I am happy to take a look and help you refactor the Type.Ref inline with the 0.34.0 update.

Keep me posted
S