Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(storage): monitored resource detection #11197

Merged
merged 23 commits into from
Dec 20, 2024
Merged

fix(storage): monitored resource detection #11197

merged 23 commits into from
Dec 20, 2024

Conversation

frankyn
Copy link
Member

@frankyn frankyn commented Nov 26, 2024

Summary of changes

  • Fixed resource attribute names detected by GCP resource detector by using _ instead of ., e.g. "host_id" instead of "host.id" resulting in storageClient always using default attribute values.
  • Prioritize project ID from ADC before using resource detector project
  • Refactored preparedResource into storageMonitoredResource to separate responsibilities
  • Explicitly name gRPC metrics enabled instead of relying on opentelemetry.DefaultMetrics()
  • Introduced integration tests for gRPC Metrics are collected through a secondary Reader
  • Used struct embeddings to reduce boiler plate on log suppression exporter (thanks @BrennaEpp)

@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Nov 26, 2024
@frankyn frankyn marked this pull request as ready for review December 2, 2024 23:39
@frankyn frankyn requested review from a team as code owners December 2, 2024 23:39
@frankyn
Copy link
Member Author

frankyn commented Dec 4, 2024

Spot checked resulting monitored resource values to double check:
image

Copy link
Contributor

@tritone tritone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few minor questions/nits, otherwise looks good.

@@ -118,6 +119,24 @@ type grpcStorageClient struct {
settings *settings
}

func enableClientMetrics(ctx context.Context, s *settings, config storageConfig) (*metricsContext, error) {
var project string
// TODO: use new auth client
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have a bug for this?

mexporter.WithMonitoredResourceDescription(monitoredResourceName, []string{"project_id", "location", "cloud_platform", "host_id", "instance_id", "api"}),
)
if err != nil {
return nil, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's wrap this with a string like "storage: creating metrics exporter: %w"

p, present := s.Value("cloud.account.id")
if present {
preparedResource.projectToUse = p.AsString()
if p, present := s.Value("cloud.account.id"); present && smr.project == "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment about what this is trying to do?

detectedAttributes []attribute.KeyValue
wantAttributes attribute.Set
}{
{
desc: "default values set when GCP attributes are not detected",
desc: "default values set when GCP attributes are not detected",
project: "project-id",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

project and api seem the same in all test cases, should they just be removed as fields here? Or did you want to test other values?

}, withTestMetricReader(mr))
}

func TestIntegration_MetricsEnablementInGCE(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we expect this test to run without skipping in Kokoro?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's correct. To test the listed metrics you need direct connectivity so it expects a GCE instance;
Is there incorrect logic?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, looks fine to me, just wanted to confirm.

"grpc.lb.rls.cache_entries": false,
"grpc.lb.rls.cache_size": false,
"grpc.lb.rls.default_target_picks": false,
// TODO: determine a way to force these metrics to be collected
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove commented values here; please file an issue if there is other stuff we need to add later.

}, withTestMetricReader(mr))
}

func TestIntegration_MetricsEnablementInGCE(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, looks fine to me, just wanted to confirm.

@tritone tritone added the automerge Merge the pull request once unit tests and other checks pass. label Dec 20, 2024
@frankyn frankyn merged commit 911bcd8 into main Dec 20, 2024
9 checks passed
@frankyn frankyn deleted the fix-mr-storage branch December 20, 2024 18:03
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants