Json Codecs Directly?

[Quafadas]

I believe this to be a very simple question, that somehow I can't quite see the answer to.

Smithy must be generating Json encoders / decoders for it's case classes.

Is it possible to access / use those decoders directly? I have a weird situation where I want to tamper with some JSON (directly (!)) before passing it on... but somehow I?m not seeing how to en/decode the smithy case classes manually...

[kubukoz]

Hi @Quafadas, Smithy4s does indeed create encoders/decoders for the shapes you define in your specs. They're not generated as Scala sources but rather derived at runtime.

You can access an instance of such a decoder/encoder, but you have to be aware of the following:

• encoding/decoding concerns are protocol-specific, and as such you'll need to pick a protocol when defining your codecs. For @simpleRestJson it's not going to be difficult
• building an encoder/decoder is computationally expensive, so you're likely to want to keep the result of doing that for as long as possible (ideally, for as long as the application runs)
• there's no standard typeclass for encoders/decoders as of now
• if you're using the provided client/server, you'll still need to end up with a "decoded" type, i.e. if you have trait FooService[F[_]] { def impl(s: String): F[ImplOutput] }, impl can't just return a Json value. I'm not sure what your usecase is, so this might not be relevant.

With all of that, here's what you could do to make instances of these encoders/decoders for SimpleRestJson:

//> using scala "2.13.8"
//> using lib "com.disneystreaming.smithy4s::smithy4s-http4s:0.14.2"
import smithy4s.schema.Schema
import smithy4s.http.PayloadError
import smithy4s.http4s.SimpleRestJsonBuilder

object codecs {

  def makeDecoder[A: Schema]: String => Either[PayloadError, A] = {
    val codecApi = SimpleRestJsonBuilder.codecs

    val codec = codecApi.compileCodec(implicitly[Schema[A]])

    s => codecApi.decodeFromByteArray(s.getBytes())
  }

  def makeEncoder[A: Schema]: A => String = {
    val codecApi = SimpleRestJsonBuilder.codecs

    val codec = codecApi.compileCodec(implicitly[Schema[A]])

    codecApi.writeToArray(codec, _)
  }
}

Then you can define and use your codecs.

object usage {

  // generated code will look something like this
  case class Foo(s: String)

  object Foo {
    // Every smithy4s-generated type has an implicit `Schema` available.
    implicit val schema: Schema[Foo] = ???
  }

  // keep in mind to keep this out of the hot loop
  val fooEncoder = codecs.makeEncoder[Foo]

  def myBusinessLogic(foo: Foo): Unit = {
    println(fooEncoder.apply(foo))
  }
}

full example

[Quafadas]

@kubukoz I don't think I said thankyou for this original post - thanks - it was very educational for me!

[Baccata]

I have a weird situation where I want to tamper with some JSON (directly (!)) before passing it on...

@Quafadas can you elaborate on what you mean about "passing it on" ?

[Quafadas]

Sure. Perhaps, the question is not the right question...

But here goes anyway. I recieve a JSON definition of a (small - cca 5-10) node (completely connected, acyclic) directed graph from an external API. Good, well formed data model. nodes / edges are identified as UUIDs . I've written out the classes in the Smithy IDL and have generated my client... successfully queried api etc.

Now I want to be able to to plot my directed graph.

I have a pet library (https://github.com/Quafadas/dedav4s) which wraps vega, i.e. takes a JSON spec and plots a force directed graph, (shameless plug alert: dedav works on the same spec on the JVM and in the browser ... wraps the entire vega API... is cool... ignore this!)

So far, so good. The catch, is that vega does not like UUIDs as it's node/edge identifiers in the plot. Vega wants ... an int.

Options I'm aware of. Either expose an API which provides the right format server side...

1. write an entire set of ghost smithy specs which are the same ... except have an int instead of UUID... then proceed with the transformation to that data model... I acknowledge this as the "right" solution, but it's really heavyweight for what I need.

** Now that you've forced me to think about this, I wonder if it may be possible to bifurcate the data model through smithys traits? Which would not be nearly as heavyweight as imagined... didn't think of this before.

3. Pollute the existing API with a bunch of optional ints and cry / figure out the serialization disconnect between option[Int], Int for the plot itself

Or... in the browser, cheat...
serialize to JSON... then string replace (cough) the UUIDs with a Map[UUID, Int] (*hides) before feeding our slightly modified spec to vega directly in browser... this actually turned out to be pretty quick... it is... effective but not pretty... with the caveat that I needed to write a shadow set of json enocders to decode the smithy types.

It was fairly quick to do this in upickle, until it occured to me I must now have two sets of json codecs :-/... hence the question...

[Baccata]

Any chance you could share the smithy model ? (or a subset of it, where the UUIDs are in use ?). It would facilitate the conversation.

I'm not quite sure, but you may be able to use unions to represent identifiers :

use smithy4s.api#untagged
use smithy4s.api#UUID

@untagged
union Identifier {
   uuid: UUID, 
   int : Integer 
} 

The untagged is a smithy4s-specific trait that indicates that the union should be encoded without any discriminating information. It's arguably the worst way of encoding unions, but sometimes it helps.

In the generated code, you'd get something like :

sealed trait Identifier 

object Identifier {
   case class UuidCase(value: UUID) extends Identifier
   case class IntCase(value: Int) extends Identifier
} 

To be clear : because it's a smithy4s-specific trait that impacts serialisation, it means that other tools will be unlikely to understand that smithy model out of the box.

You can read a bit more about how smithy4s handles unions (in particular in the context of json) here

Now, regarding your initial question, which I'll rephrase to "how the f*** does smithy4s translates case class instances to JSON ?", we've recently added some design document to try and explain how the internals work. You can have a look here. The tl;dr is that smithy4s does not use traditional implicit-driven encoding mechanisms. You cannot highjack the encoding of the generated code by providing custom implicits. This is a good thing for us, as it guarantees that the implementation does not drift away from the contract.

As @kubukoz has hinted, the way it works is that each generated type comes with something called a Schema, which can be transformed into encoders by running the schema through what is essentially a SchemaVisitor. You can implement virtually any encoding logic via this mechanism, without changing the generated code. However, the "SimpleRestJson" interpreter is responsible for running a json-specific SchemaVisitor in its implementation, and storing the codecs in private values, which means you can't highjack it.

If you wanted to define your own protocol, you could very much do it, and you could tweak the encoding however you'd like, but then, it'd mean you would benefit from less things out of the box, such as the smithy => openapi conversion, which you'd have to re-implement for your own protocol, since that conversion is protocol specific.

[Quafadas]

and storing the codecs in private values, which means you can't highjack it.

I think I understand why you would want this, and it makes a lot of sense. No complaints - I've posted up a minimal set of the smithy specs below... they are simple enough :-).

Thankyou both again, for your time and willingness to help!

[Baccata]

No worries :)

[Quafadas]

This would be the minimal subset. Both the nodes and edges are / will each become their own ADT later, but i haven't gotten around to that part (successfully) yet...

structure CfnEntity {
    @httpLabel
    @required
    id: UUID,
 // Some other stuff missing here
    metaData: DigraphMetaData,
    network: CfnDigraph
}

structure CfnDigraph {
    @required
    nodes: CfnDigraphNodes,
    @required
    connections: CfnDigraphconnections
}

list CfnDigraphNodes {    
    member: CfnDigraphNode
}

list CfnDigraphconnections {
    member: CfnDigraphconnection
}

structure CfnDigraphNode {
    @required
    id: UUID,
    @required
    name: String,
    @required
    type: String
}

structure CfnDigraphconnection {
    @required
    id: UUID,
    @required
    from: UUID,
    @required
    to: UUID,
} 

[Baccata]

Yeah, I think untagged unions might be the way to go. That way you get something that can be either an Int or a UUID, both in your code, and in the JSON payloads. Then depending on the usecase, you could pre-process your graphs to transform all Ints into UUIDs (or vice-versa) according to whatever mapping you elect to use.

In case you're not aware, a UUID is actually isomorphic to (Long, Long), so you can come with Int => UUID mappings that do not require storing anything in memory. The opposite is not true though (because a pair of long has a higher cardinality than Integers), so you could do UUID => Either[NotConvertible, Int], or come up with some UUID => Int conversion that'd imply that multiple UUIDs would convert to the same Int value (which may lead to problems down the line)

[Quafadas]

Not gonna lie here... I wasn't basically aware basically any of the things in this post here!

In case you're not aware, a UUID is actually isomorphic to (Long, Long)

! ... Google-fu engaged...

untagged unions

I'll consider experimenting with this. - it is good to know they exist and think about the use cases. My first reaction, is that in this case I'm going to keep the data model disciplined. My intuition, is that choice like this has some awkward implications on typesafety throughout the app / database / codebase.

With the JSON hack, the nastiness is contained in once place / method around the plot - which is the only place it makes sense - and can be easily unit tested and signposted. The remainder of the data model stays robust.

There is a lot to think about in the answers to this question - I'm going to take them away and meditate for a while! Thankyou again.

[Baccata]

If that suggestion doesn't float your boat, it would be possible to keep the hack localised via an Http4s middleware. By that I mean a Client[F] => Client[F] function. The resulting client would intercept the request, decode the payload Json (using http4s-circe for instance), apply a Json transformation, then re-encode the json into the payload of the request you forward to the underlying client. It's not too complicated, but it does come with performance implications that you'd have to assess (because you end up parsing the body twice, once through the middleware, once through the smithy4s interpreter). Depending on the type of application you're writing, the impact might be insignificant, but you need to be aware of it nonetheless.

[Quafadas]

If that suggestion doesn't float your boat, it would be possible to keep the hack localised via an Http4s middleware. By that I mean a Client[F] => Client[F] function. The resulting client would intercept the request, decode the payload Json (using http4s-circe for instance), apply a Json transformation, then re-encode the json into the payload of the request you forward to the underlying client. It's not too complicated, but it does come with performance implications that you'd have to assess (because you end up parsing the body twice, once through the middleware, once through the smithy4s interpreter). Depending on the type of application you're writing, the impact might be insignificant, but you need to be aware of it nonetheless.

I think would probably be the "right" thing for me given all the discussion above, thinking about it, that^s basically what I've done anyway. Thanks again.

Finally, for anything I write: performance is a non-issue. Scala is already way faster than my needs... I think I have 1 hot loop and everything else doesn't matter :-).