Skip to content

Commit

Permalink
Remove protoc-gen-bigquery (#12)
Browse files Browse the repository at this point in the history
  • Loading branch information
mfridman authored Apr 5, 2024
1 parent b8cd8fb commit 729df87
Show file tree
Hide file tree
Showing 38 changed files with 54 additions and 5,839 deletions.
5 changes: 0 additions & 5 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,3 @@ issues:
# Don't ban use of fmt.Errorf to create new errors, but the remaining
# checks from err113 are useful.
- "err113: do not define dynamic errors.*"
exclude-rules:
# We allow fmt.Println to be used.
- path: internal/cmd/bigquery-protobuf-uploader/main.go
linters:
- forbidigo
2 changes: 0 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,8 @@ test: build $(BIN)/jv ## Run unit tests

.PHONY: golden
golden: generate
rm -rf internal/testdata/bigquery
rm -rf internal/testdata/pubsub
rm -rf internal/testdata/jsonschema
go run internal/cmd/bigquery-generate-testdata/main.go internal/testdata/bigquery
go run internal/cmd/pubsub-generate-testdata/main.go internal/testdata/pubsub
go run internal/cmd/jsonschema-generate-testdata/main.go internal/testdata/jsonschema
buf build ./internal/proto --exclude-source-info -o -#format=json > ./internal/testdata/codegenrequest/input.json
Expand Down
73 changes: 0 additions & 73 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ The protoschema-plugins repository contains a collection of Protobuf plugins tha
types of schema from protobuf files. This includes:

- [PubSub](#pubsub-protobuf-schema)
- [BigQuery](#bigquery-protobuf-schema)
- [JSON Schema](#json-schema)

## PubSub Protobuf Schema
Expand All @@ -35,78 +34,6 @@ plugins:
For examples see [testdata](/internal/testdata/pubsub/) which contains the generated schema for
test case definitions found in [proto](/internal/proto/).
## BigQuery Protobuf Schema
Generates a Table schema that can be used to initialize a BigQuery table, and a self-contained
message normalized to proto2, that can be used to upload data to that BigQuery table.
Install the `protoc-gen-bigquery` directly:

```sh
go install github.com/bufbuild/protoschema-plugins/cmd/protoc-gen-bigquery@latest
```

Or reference it as a [Remote Plugin](https://buf.build/docs/generate/remote-plugins) in `buf.gen.yaml`:

```yaml
version: v1
plugins:
- plugin: buf.build/bufbuild/protoschema-bigquery
out: ./gen
```

For examples see [testdata](/internal/testdata/bigquery/) which contains the generated schema for
test case definitions found in [proto](/internal/proto/).

This supports the `gen_bq_schema` annotations. For example:

```proto
message CustomBigQuery {
option (gen_bq_schema.bigquery_opts) = {
table_name: "CustomName"
use_json_names: true
};
int32 int32_field = 1 [(gen_bq_schema.bigquery) = {
type_override: "TIMESTAMP",
name: "create_time"
}];
string string_field = 2;
NestedReference nested_reference = 3 [(gen_bq_schema.bigquery) = {ignore: true}];
}
```

Results in the Table schema:

```json
[
{
"name": "create_time",
"type": "TIMESTAMP"
},
{
"name": "stringField",
"type": "STRING"
}
]
```

and the message:

```proto
message CustomBigQuery {
optional int32 create_time = 1;
optional string stringField = 2;
}
```

### Limitations

The well-known JSON types are not support in BigQuery. This includes `google.protobuf.Struct`,
`google.protobuf.Value` and `google.protobuf.ListValue`. If these types are used in a message, the
generated schema will not include them, and the generated message will not include them.

## JSON Schema
Generates a [JSON Schema](https://json-schema.org/) for a given protobuf file. This implementation
Expand Down
1 change: 0 additions & 1 deletion buf.gen.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ managed:
default: github.com/bufbuild/protoschema-plugins/internal/gen/proto
except:
- buf.build/bufbuild/protovalidate
- buf.build/liberetech/protoc-gen-bq-schema
plugins:
- plugin: buf.build/protocolbuffers/go
out: internal/gen/proto
Expand Down
25 changes: 0 additions & 25 deletions cmd/protoc-gen-bigquery/main.go

This file was deleted.

35 changes: 0 additions & 35 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@ go 1.22

require (
buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go v1.33.0-20240221180331-f05a6f4403ce.1
cloud.google.com/go/bigquery v1.59.1
github.com/GoogleCloudPlatform/protoc-gen-bq-schema v0.0.0-20240229095244-e9949450f473
github.com/bufbuild/buf v1.29.0
github.com/bufbuild/protoplugin v0.0.0-20240318153824-eeb4e72439df
github.com/jhump/protoreflect v1.15.6
Expand All @@ -14,65 +12,32 @@ require (
)

require (
cloud.google.com/go v0.112.1 // indirect
cloud.google.com/go/compute v1.25.1 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.7 // indirect
github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
github.com/apache/arrow/go/v14 v14.0.2 // indirect
github.com/bufbuild/protocompile v0.9.0 // indirect
github.com/bufbuild/protovalidate-go v0.5.0 // indirect
github.com/bufbuild/protoyaml-go v0.1.7 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gofrs/uuid/v5 v5.0.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/cel-go v0.19.0 // indirect
github.com/google/flatbuffers v24.3.7+incompatible // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.3 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/klauspost/compress v1.17.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.7 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/spf13/cobra v1.8.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/stoewer/go-strcase v1.3.0 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.26.0 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/exp v0.0.0-20240112132812-db7319d0e0e3 // indirect
golang.org/x/mod v0.16.0 // indirect
golang.org/x/net v0.22.0 // indirect
golang.org/x/oauth2 v0.18.0 // indirect
golang.org/x/sync v0.6.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.19.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/api v0.170.0 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto v0.0.0-20240318140521-94a12d6c2237 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237 // indirect
google.golang.org/grpc v1.62.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading

0 comments on commit 729df87

Please sign in to comment.