Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Must use alternate model because of incompatibilities between java and javalite runtimes with well-known types #194

Merged
merged 3 commits into from
Jan 10, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ BIN := .tmp/bin
CACHE := .tmp/cache
LICENSE_HEADER_YEAR_RANGE := 2022-2023
LICENSE_HEADER_VERSION := v1.28.1
CONFORMANCE_VERSION := v1.0.0-rc1
CONFORMANCE_VERSION := v1.0.0-rc2
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main change this brings in is a new method on the conformance service: IdempotentUnary, which is just like Unary, but it is declared to be side-effect-free so it can be used with Connect GET.

PROTOC_VERSION ?= 25.1
GRADLE_ARGS ?=

Expand Down Expand Up @@ -75,8 +75,7 @@ generate: $(PROTOC) buildplugin generateconformance generateexamples ## Generate
.PHONY: generateconformance
generateconformance: $(PROTOC) buildplugin ## Generate protofiles for conformance tests.
buf generate --template conformance/buf.gen.yaml -o conformance conformance/proto
buf generate --template conformance/client/buf.gen.yaml -o conformance/client buf.build/connectrpc/conformance:$(CONFORMANCE_VERSION)
buf generate --template conformance/client/buf.gen.lite.yaml -o conformance/client buf.build/connectrpc/conformance:$(CONFORMANCE_VERSION)
buf generate --template conformance/buf.gen.yaml -o conformance/client buf.build/connectrpc/conformance:$(CONFORMANCE_VERSION)

.PHONY: generateexamples
generateexamples: $(PROTOC) buildplugin ## Generate proto files for example apps.
Expand Down
20 changes: 0 additions & 20 deletions conformance/client/buf.gen.lite.yaml

This file was deleted.

17 changes: 0 additions & 17 deletions conformance/client/buf.gen.yaml

This file was deleted.

28 changes: 0 additions & 28 deletions conformance/client/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -2,37 +2,9 @@ plugins {
kotlin("jvm")
}

// This base project contains generated code for the lite runtime
// and depends on the Google Protobuf Java Lite runtime.
// The main client logic is implemented in terms of generated
// code for that lite runtime.
//
// The non-lite runtime excludes the Google Protobuf Java Lite
// runtime and instead uses the full Java runtime. It then can
// adapt from the lite-runtime-generated code by serializing to
// bytes and then de-serializing into non-lite-generated types.

sourceSets {
main {
java {
srcDir("build/generated/sources/bufgen")
}
}
}

tasks {
compileKotlin {
kotlinOptions {
// Generated Kotlin code for protobufs uses RequiresOptIn annotation
freeCompilerArgs += "-opt-in=kotlin.RequiresOptIn"
}
}
}

dependencies {
implementation(project(":okhttp"))
implementation(libs.kotlin.coroutines.core)
implementation(libs.protobuf.kotlinlite)
implementation(libs.protobuf.javalite)
implementation(libs.okio.core)
implementation(libs.okhttp.tls)
Expand Down
12 changes: 7 additions & 5 deletions conformance/client/google-java/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@ plugins {
tasks {
compileKotlin {
kotlinOptions {
// Generated Kotlin code for protobufs uses OptIn annotation
// Generated Kotlin code for protobuf uses OptIn annotation
freeCompilerArgs += "-opt-in=kotlin.RequiresOptIn"
}
}
shadowJar {
archiveBaseName.set("shadow")
archiveFileName.set("conformance-client-java.jar")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made this change to make it easier to actually use the output JAR from another target in the Makefile. Without this, the actual file has a <version>-SNAPSHOT in the filename. But I couldn't find any simple way to figure out the <version>, so that a command could reference the correct filename. (It appears to be computed dynamically by Gradle based on git tags.) So this removes the suffix so we can easily reference the correct output JAR file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option here is that you can make it use the application plugin (https://docs.gradle.org/current/userguide/application_plugin.html) which when combined with installDist will create an executable script with the right jar file name embedded. I do that in protoc-gen-connect-kotlin to make it callable.

Copy link
Member Author

@jhump jhump Jan 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤦, doh! I should have done that to begin with -- good idea! I wasn't aware of the application plugin and was mainly used to building "fat jars" when doing Java dev with Maven way back in the day. That's how I ended up finding the shadowjar plugin.

I've ripped out the shadowjar stuff and converted both of these conformance programs to use the application plugin. Thanks for the pointer!

manifest {
attributes(mapOf("Main-Class" to "com.connectrpc.conformance.client.java.MainKt"))
}
Expand All @@ -31,8 +31,6 @@ tasks {
}
}

// This project contains an alternate copy of the generated
// types, generated for the non-lite runtime.
sourceSets {
main {
java {
Expand All @@ -43,8 +41,12 @@ sourceSets {

dependencies {
implementation(project(":conformance:client")) {
// Shared module depends on javalite, just for some core
// classes that are shared across both java and javalite
// runtimes, like ByteString and MessageLite. We must
// exclude it here to avoid any classpath ambiguity since
// we pull in the full runtime for this module.
exclude(group = "com.google.protobuf", module = "protobuf-javalite")
exclude(group = "com.google.protobuf", module = "protobuf-kotlinlite")
}
implementation(project(":extensions:google-java"))
implementation(project(":okhttp"))
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
// Copyright 2022-2023 The Connect Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package com.connectrpc.conformance.client.java

import com.connectrpc.ConnectException
import com.connectrpc.Headers
import com.connectrpc.SerializationStrategy
import com.connectrpc.conformance.client.adapt.AnyMessage
import com.connectrpc.conformance.client.adapt.ClientCompatRequest
import com.connectrpc.conformance.client.adapt.ClientCompatRequest.HttpVersion
import com.connectrpc.conformance.client.adapt.ClientCompatRequest.TlsCreds
import com.connectrpc.conformance.client.adapt.ClientCompatResponse
import com.connectrpc.conformance.v1.BidiStreamResponse
import com.connectrpc.conformance.v1.ClientCompatRequest.Cancel.CancelTimingCase
import com.connectrpc.conformance.v1.ClientErrorResult
import com.connectrpc.conformance.v1.ClientResponseResult
import com.connectrpc.conformance.v1.ClientStreamResponse
import com.connectrpc.conformance.v1.Codec
import com.connectrpc.conformance.v1.Compression
import com.connectrpc.conformance.v1.ConformancePayload
import com.connectrpc.conformance.v1.Error
import com.connectrpc.conformance.v1.HTTPVersion
import com.connectrpc.conformance.v1.Header
import com.connectrpc.conformance.v1.IdempotentUnaryResponse
import com.connectrpc.conformance.v1.Protocol
import com.connectrpc.conformance.v1.ServerStreamResponse
import com.connectrpc.conformance.v1.StreamType
import com.connectrpc.conformance.v1.UnaryResponse
import com.connectrpc.conformance.v1.UnimplementedResponse
import com.connectrpc.extensions.GoogleJavaJSONStrategy
import com.connectrpc.extensions.GoogleJavaProtobufStrategy
import com.connectrpc.protocols.NetworkProtocol
import com.google.protobuf.Any
import com.google.protobuf.ByteString
import com.google.protobuf.MessageLite

class JavaHelpers {
companion object {
private const val TYPE_URL_PREFIX = "type.googleapis.com/"

fun serializationStrategy(codec: ClientCompatRequest.Codec): SerializationStrategy {
return when (codec) {
ClientCompatRequest.Codec.PROTO -> GoogleJavaProtobufStrategy()
ClientCompatRequest.Codec.JSON -> GoogleJavaJSONStrategy()
else -> throw RuntimeException("unsupported codec $codec")
}
}

fun unmarshalRequest(bytes: ByteArray): ClientCompatRequest {
val msg = com.connectrpc.conformance.v1.ClientCompatRequest.parseFrom(bytes)
return ClientCompatRequestImpl(msg)
}

fun marshalResponse(resp: ClientCompatResponse): ByteArray {
val builder = com.connectrpc.conformance.v1.ClientCompatResponse
.newBuilder()
.setTestName(resp.testName)
when (val result = resp.result) {
is ClientCompatResponse.Result.ResponseResult -> {
val respBuilder = ClientResponseResult.newBuilder()
.addAllResponseHeaders(toProtoHeaders(result.response.headers))
.addAllPayloads(toProtoPayloads(result.response.payloads))
.addAllResponseTrailers(toProtoHeaders(result.response.trailers))
.setNumUnsentRequests(result.response.numUnsentRequests)
val err = result.response.error
if (err != null) {
respBuilder.setError(toProtoError(err))
}
builder.setResponse(respBuilder)
}
is ClientCompatResponse.Result.ErrorResult -> {
builder.setError(
ClientErrorResult.newBuilder()
.setMessage(result.error),
)
}
}
return builder.build().toByteArray()
}

fun extractPayload(response: MessageLite): MessageLite {
return when (response) {
is UnaryResponse -> response.payload
is IdempotentUnaryResponse -> response.payload
is UnimplementedResponse -> ConformancePayload.getDefaultInstance()
is ClientStreamResponse -> response.payload
is ServerStreamResponse -> response.payload
is BidiStreamResponse -> response.payload
else -> throw RuntimeException("don't know how to extract payload from ${response::class.qualifiedName}")
}
}

private fun fromProtoHeaders(headers: List<Header>): Headers {
return headers.groupingBy(Header::getName).aggregate { _: String, accumulator: List<String>?, element: Header, _: Boolean ->
accumulator?.plus(element.valueList) ?: element.valueList
}
}

private fun toProtoHeaders(headers: Headers): List<Header> {
return headers.map {
Header.newBuilder()
.setName(it.key)
.addAllValue(it.value)
.build()
}
}

private fun toProtoPayloads(payloads: List<MessageLite>): List<ConformancePayload> {
return payloads.map {
if (it is ConformancePayload) {
it
} else {
ConformancePayload.parseFrom(it.toByteArray())
}
}
}

private fun toProtoError(ex: ConnectException): Error {
return Error.newBuilder()
.setCode(ex.code.value)
.setMessage(ex.message ?: ex.code.codeName)
.addAllDetails(
ex.details.map {
Any.newBuilder()
.setTypeUrl(toTypeUrl(it.type))
.setValue(ByteString.copyFrom(it.payload.toByteArray()))
.build()
},
)
.build()
}

private fun toTypeUrl(typeName: String): String {
return if (typeName.contains('/')) typeName else TYPE_URL_PREFIX + typeName
}
}

private class ClientCompatRequestImpl(
private val msg: com.connectrpc.conformance.v1.ClientCompatRequest,
) : ClientCompatRequest {
override val testName: String
get() = msg.testName
override val service: String
get() = msg.service
override val method: String
get() = msg.method
override val host: String
get() = msg.host
override val port: Int
get() = msg.port
override val serverTlsCert: ByteString
get() = msg.serverTlsCert
override val clientTlsCreds: TlsCreds?
get() = if (msg.hasClientTlsCreds()) TlsCredsImpl(msg.clientTlsCreds) else null
override val timeoutMs: Int
get() = msg.timeoutMs
override val requestDelayMs: Int
get() = msg.requestDelayMs
override val useGetHttpMethod: Boolean
get() = msg.useGetHttpMethod
override val httpVersion: HttpVersion
get() = when (msg.httpVersion) {
HTTPVersion.HTTP_VERSION_1 -> HttpVersion.HTTP_1_1
HTTPVersion.HTTP_VERSION_2 -> HttpVersion.HTTP_2
else -> throw RuntimeException("unsupported HTTP version: ${msg.httpVersion}")
}
override val protocol: NetworkProtocol
get() = when (msg.protocol) {
Protocol.PROTOCOL_CONNECT -> NetworkProtocol.CONNECT
Protocol.PROTOCOL_GRPC -> NetworkProtocol.GRPC
Protocol.PROTOCOL_GRPC_WEB -> NetworkProtocol.GRPC_WEB
else -> throw RuntimeException("unsupported protocol: ${msg.protocol}")
}
override val codec: ClientCompatRequest.Codec
get() = when (msg.codec) {
Codec.CODEC_PROTO -> ClientCompatRequest.Codec.PROTO
Codec.CODEC_JSON -> ClientCompatRequest.Codec.JSON
else -> throw RuntimeException("unsupported codec: ${msg.codec}")
}
override val compression: ClientCompatRequest.Compression
get() = when (msg.compression) {
Compression.COMPRESSION_IDENTITY, Compression.COMPRESSION_UNSPECIFIED -> ClientCompatRequest.Compression.IDENTITY
Compression.COMPRESSION_GZIP -> ClientCompatRequest.Compression.GZIP
else -> throw RuntimeException("unsupported compression: ${msg.compression}")
}
override val streamType: ClientCompatRequest.StreamType
get() = when (msg.streamType) {
StreamType.STREAM_TYPE_UNARY -> ClientCompatRequest.StreamType.UNARY
StreamType.STREAM_TYPE_CLIENT_STREAM -> ClientCompatRequest.StreamType.CLIENT_STREAM
StreamType.STREAM_TYPE_SERVER_STREAM -> ClientCompatRequest.StreamType.SERVER_STREAM
StreamType.STREAM_TYPE_HALF_DUPLEX_BIDI_STREAM -> ClientCompatRequest.StreamType.HALF_DUPLEX_BIDI_STREAM
StreamType.STREAM_TYPE_FULL_DUPLEX_BIDI_STREAM -> ClientCompatRequest.StreamType.FULL_DUPLEX_BIDI_STREAM
else -> throw RuntimeException("unsupported stream type: ${msg.streamType}")
}
override val requestHeaders: Headers
get() = fromProtoHeaders(msg.requestHeadersList)
override val requestMessages: List<AnyMessage>
get() = msg.requestMessagesList.map {
AnyMessage(it.typeUrl, it.value)
}
override val cancel: ClientCompatRequest.Cancel?
get() = when (msg.cancel.cancelTimingCase) {
CancelTimingCase.CANCELTIMING_NOT_SET, null ->
null
CancelTimingCase.BEFORE_CLOSE_SEND ->
ClientCompatRequest.Cancel.BeforeCloseSend()
CancelTimingCase.AFTER_CLOSE_SEND_MS ->
ClientCompatRequest.Cancel.AfterCloseSendMs(msg.cancel.afterCloseSendMs)
CancelTimingCase.AFTER_NUM_RESPONSES ->
ClientCompatRequest.Cancel.AfterNumResponses(msg.cancel.afterNumResponses)
}
}

private class TlsCredsImpl(
private val msg: com.connectrpc.conformance.v1.ClientCompatRequest.TLSCreds,
) : TlsCreds {
override val cert: ByteString
get() = msg.cert
override val key: ByteString
get() = msg.key
}
}
Loading