Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove protoc-gen-bigquery #12

Merged
merged 2 commits into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,3 @@ issues:
# Don't ban use of fmt.Errorf to create new errors, but the remaining
# checks from err113 are useful.
- "err113: do not define dynamic errors.*"
exclude-rules:
# We allow fmt.Println to be used.
- path: internal/cmd/bigquery-protobuf-uploader/main.go
linters:
- forbidigo
2 changes: 0 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,8 @@ test: build $(BIN)/jv ## Run unit tests

.PHONY: golden
golden: generate
rm -rf internal/testdata/bigquery
rm -rf internal/testdata/pubsub
rm -rf internal/testdata/jsonschema
go run internal/cmd/bigquery-generate-testdata/main.go internal/testdata/bigquery
go run internal/cmd/pubsub-generate-testdata/main.go internal/testdata/pubsub
go run internal/cmd/jsonschema-generate-testdata/main.go internal/testdata/jsonschema
buf build ./internal/proto --exclude-source-info -o -#format=json > ./internal/testdata/codegenrequest/input.json
Expand Down
73 changes: 0 additions & 73 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ The protoschema-plugins repository contains a collection of Protobuf plugins tha
types of schema from protobuf files. This includes:

- [PubSub](#pubsub-protobuf-schema)
- [BigQuery](#bigquery-protobuf-schema)
- [JSON Schema](#json-schema)

## PubSub Protobuf Schema
Expand All @@ -35,78 +34,6 @@ plugins:
For examples see [testdata](/internal/testdata/pubsub/) which contains the generated schema for
test case definitions found in [proto](/internal/proto/).

## BigQuery Protobuf Schema

Generates a Table schema that can be used to initialize a BigQuery table, and a self-contained
message normalized to proto2, that can be used to upload data to that BigQuery table.

Install the `protoc-gen-bigquery` directly:

```sh
go install github.com/bufbuild/protoschema-plugins/cmd/protoc-gen-bigquery@latest
```

Or reference it as a [Remote Plugin](https://buf.build/docs/generate/remote-plugins) in `buf.gen.yaml`:

```yaml
version: v1
plugins:
- plugin: buf.build/bufbuild/protoschema-bigquery
out: ./gen
```

For examples see [testdata](/internal/testdata/bigquery/) which contains the generated schema for
test case definitions found in [proto](/internal/proto/).

This supports the `gen_bq_schema` annotations. For example:

```proto
message CustomBigQuery {
option (gen_bq_schema.bigquery_opts) = {
table_name: "CustomName"
use_json_names: true
};

int32 int32_field = 1 [(gen_bq_schema.bigquery) = {
type_override: "TIMESTAMP",
name: "create_time"
}];
string string_field = 2;

NestedReference nested_reference = 3 [(gen_bq_schema.bigquery) = {ignore: true}];
}
```

Results in the Table schema:

```json
[
{
"name": "create_time",
"type": "TIMESTAMP"
},
{
"name": "stringField",
"type": "STRING"
}
]
```

and the message:

```proto
message CustomBigQuery {
optional int32 create_time = 1;
optional string stringField = 2;
}
```

### Limitations

The well-known JSON types are not support in BigQuery. This includes `google.protobuf.Struct`,
`google.protobuf.Value` and `google.protobuf.ListValue`. If these types are used in a message, the
generated schema will not include them, and the generated message will not include them.

## JSON Schema

Generates a [JSON Schema](https://json-schema.org/) for a given protobuf file. This implementation
Expand Down
1 change: 0 additions & 1 deletion buf.gen.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ managed:
default: github.com/bufbuild/protoschema-plugins/internal/gen/proto
except:
- buf.build/bufbuild/protovalidate
- buf.build/liberetech/protoc-gen-bq-schema
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bonus, we ended up publishing the module as well:

https://buf.build/googlecloudplatform/bq-schema-api

plugins:
- plugin: buf.build/protocolbuffers/go
out: internal/gen/proto
Expand Down
25 changes: 0 additions & 25 deletions cmd/protoc-gen-bigquery/main.go

This file was deleted.

35 changes: 0 additions & 35 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@ go 1.22

require (
buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.33.0-20240221180331-f05a6f4403ce.1
cloud.google.com/go/bigquery v1.59.1
github.com/GoogleCloudPlatform/protoc-gen-bq-schema v0.0.0-20240229095244-e9949450f473
github.com/bufbuild/buf v1.29.0
github.com/bufbuild/protoplugin v0.0.0-20240318153824-eeb4e72439df
github.com/jhump/protoreflect v1.15.6
Expand All @@ -14,65 +12,32 @@ require (
)

require (
cloud.google.com/go v0.112.1 // indirect
cloud.google.com/go/compute v1.25.1 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.7 // indirect
github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
github.com/apache/arrow/go/v14 v14.0.2 // indirect
github.com/bufbuild/protocompile v0.9.0 // indirect
github.com/bufbuild/protovalidate-go v0.5.0 // indirect
github.com/bufbuild/protoyaml-go v0.1.7 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gofrs/uuid/v5 v5.0.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/cel-go v0.19.0 // indirect
github.com/google/flatbuffers v24.3.7+incompatible // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.3 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/klauspost/compress v1.17.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.7 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/spf13/cobra v1.8.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stoewer/go-strcase v1.3.0 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.26.0 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/exp v0.0.0-20240112132812-db7319d0e0e3 // indirect
golang.org/x/mod v0.16.0 // indirect
golang.org/x/net v0.22.0 // indirect
golang.org/x/oauth2 v0.18.0 // indirect
golang.org/x/sync v0.6.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.19.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/api v0.170.0 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto v0.0.0-20240318140521-94a12d6c2237 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 // indirect
google.golang.org/grpc v1.62.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading
Loading