Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/valkey]: Init new component #37005

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d8e1a23
feat: init valkey receiver
rogercoll Dec 7, 2024
df723f7
Merge branch 'main' into init_valkey_receiver
rogercoll Dec 7, 2024
df456b4
refactor receiver to 0.115.0
rogercoll Dec 7, 2024
bd9ffef
add integration tests
rogercoll Dec 24, 2024
613dd31
test: add container integration tests
rogercoll Jan 2, 2025
60963de
chore: add changelog entry
rogercoll Jan 2, 2025
7d7655d
Merge branch 'main' into init_valkey_receiver
rogercoll Jan 2, 2025
9898ae9
chore: remove raw data file
rogercoll Jan 2, 2025
480f82c
fix: typo and core version
rogercoll Jan 2, 2025
d7ceb6d
Revert "chore: remove raw data file"
rogercoll Jan 2, 2025
66856f7
fix: header licenses
rogercoll Jan 2, 2025
8ec3e78
fix: crosslinks
rogercoll Jan 2, 2025
4e81208
fix: scrape tests resource attributes
rogercoll Jan 2, 2025
f8c2c52
chore: update CODEOWNERS file
rogercoll Jan 2, 2025
a64d888
fix: reference links
rogercoll Jan 2, 2025
c48db20
chore: remove user/password configurations
rogercoll Jan 2, 2025
1644903
fix linter
rogercoll Jan 2, 2025
3c08b49
Update receiver/valkeyreceiver/client.go
rogercoll Jan 3, 2025
fc0f24b
feat: return error if error on split
rogercoll Jan 3, 2025
1a76c23
fix: use strings.Cut instead of strings.Split
rogercoll Jan 3, 2025
85454e4
Merge branch 'main' into init_valkey_receiver
rogercoll Jan 3, 2025
018a5e4
docs: link configtls and remove redundant section
rogercoll Jan 7, 2025
3269428
Merge branch 'main' into init_valkey_receiver
rogercoll Jan 7, 2025
db230b1
chore: new_component changelog
rogercoll Jan 7, 2025
7b4e7d7
Merge branch 'main' into init_valkey_receiver
rogercoll Jan 9, 2025
123aa3e
fix: linter and replace deprecated scraper
rogercoll Jan 9, 2025
8309e1c
fix: bump internal versions
rogercoll Jan 9, 2025
1c23a7c
Merge branch 'main' into init_valkey_receiver
rogercoll Jan 28, 2025
7b7df5c
fix: go modules
rogercoll Jan 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .chloggen/init_valkey_receiver.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: 'enhancement'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be new_component

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed db230b1


# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: valkeyreceiver

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Init code for Valkey receiver

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [33787]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: []
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,7 @@ receiver/systemdreceiver/ @open-telemetry/collector-cont
receiver/tcplogreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski
receiver/tlscheckreceiver/ @open-telemetry/collector-contrib-approvers @atoulme @michael-burt
receiver/udplogreceiver/ @open-telemetry/collector-contrib-approvers @djaglowski
receiver/valkeyreceiver/ @open-telemetry/collector-contrib-approvers @dmitryax @rogercoll
receiver/vcenterreceiver/ @open-telemetry/collector-contrib-approvers @schmikei @StefanKurek
receiver/wavefrontreceiver/ @open-telemetry/collector-contrib-approvers @samiura
receiver/webhookeventreceiver/ @open-telemetry/collector-contrib-approvers @atoulme @shalper2
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ body:
- receiver/tcplog
- receiver/tlscheck
- receiver/udplog
- receiver/valkey
- receiver/vcenter
- receiver/wavefront
- receiver/webhookevent
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,7 @@ body:
- receiver/tcplog
- receiver/tlscheck
- receiver/udplog
- receiver/valkey
- receiver/vcenter
- receiver/wavefront
- receiver/webhookevent
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/other.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,7 @@ body:
- receiver/tcplog
- receiver/tlscheck
- receiver/udplog
- receiver/valkey
- receiver/vcenter
- receiver/wavefront
- receiver/webhookevent
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/unmaintained.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,7 @@ body:
- receiver/tcplog
- receiver/tlscheck
- receiver/udplog
- receiver/valkey
- receiver/vcenter
- receiver/wavefront
- receiver/webhookevent
Expand Down
2 changes: 2 additions & 0 deletions cmd/otelcontribcol/builder-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ receivers:
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver v0.116.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/tcplogreceiver v0.116.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/udplogreceiver v0.116.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver v0.116.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.116.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/wavefrontreceiver v0.116.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/webhookeventreceiver v0.116.0
Expand Down Expand Up @@ -343,6 +344,7 @@ replaces:
- github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza => ../../pkg/stanza
- github.com/open-telemetry/opentelemetry-collector-contrib/receiver/fluentforwardreceiver => ../../receiver/fluentforwardreceiver
- github.com/open-telemetry/opentelemetry-collector-contrib/receiver/redisreceiver => ../../receiver/redisreceiver
- github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver => ../../receiver/valkeyreceiver
- github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension => ../../extension/basicauthextension
- github.com/open-telemetry/opentelemetry-collector-contrib/exporter/influxdbexporter => ../../exporter/influxdbexporter
- github.com/open-telemetry/opentelemetry-collector-contrib/exporter/alertmanagerexporter => ../../exporter/alertmanagerexporter
Expand Down
1 change: 1 addition & 0 deletions receiver/valkeyreceiver/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
53 changes: 53 additions & 0 deletions receiver/valkeyreceiver/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Valkey Receiver

<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Stability | [development]: metrics |
| Distributions | [contrib] |
| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Areceiver%2Fvalkey%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fvalkey) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Areceiver%2Fvalkey%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fvalkey) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@dmitryax](https://www.github.com/dmitryax), [@rogercoll](https://www.github.com/rogercoll) |

[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
<!-- end autogenerated section -->

The Valkey receiver is designed to retrieve Valkey INFO data from a single
Valkey instance, build metrics from that data, and send them to the next consumer at a
configurable interval.

## Details

The Valkey INFO command returns information and statistics about a Valkey
server (see [https://valkey.io/commands/info/](https://valkey.io/commands/info/) for
details). The Valkey receiver extracts values from the result and converts them to open
telemetry metrics. Details about the metrics produced by the Valkey receiver
can be found by browsing [the metadata file](./metadata.yaml).

## Configuration

The following settings are required:

- `endpoint` (no default): The hostname and port of the Valkey instance,
separated by a colon.

The following settings are optional:

- `collection_interval` (default = `10s`): This receiver runs on an interval.
Each time it runs, it queries Valkey, creates metrics, and sends them to the
next consumer. The `collection_interval` configuration option tells this
receiver the duration between runs. This value must be a string readable by
Golang's `ParseDuration` function (example: `1h30m`). Valid time units are
`ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit: I think maybe it's not reasonable to set up a too shot collection interval to prevent too much stress on Valkey.
Therefore, we can set up to minimum duration check in code to validate it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me, which low bound would you propose? Maybe 1 second?

Do we have any other scraper that sets a minimum collection interval value?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I'm not sure about this, but 1s looks good to me, even though receiver can fetch metric the at such granularity, but I think in most cases, the user won't store or query the metrics in that granularity. For me I often query the metric in the granularity of one minute. It's enough for me in most cases.

- `tls`: see [TLS Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings) for the full set of available options.

Example:

```yaml
receivers:
valkey:
endpoint: "localhost:6379"
collection_interval: 10s
```

The full list of settings exposed for this receiver are documented [here](./config.go).
69 changes: 69 additions & 0 deletions receiver/valkeyreceiver/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0
package valkeyreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver"

import (
"context"
"fmt"
"strings"

"github.com/valkey-io/valkey-go"
)

var attrDelimiter = "\r\n"

// Interface for a Valkey client. Implementation can be faked for testing.
type client interface {
// retrieves a string of key/value pairs of valkey metadata
retrieveInfo(context.Context) (map[string]string, error)
// close release Valkey client connection pool
close() error
}

// Wraps a real Valkey client, implements `client` interface.
type valkeyClient struct {
client valkey.Client
}

var _ client = (*valkeyClient)(nil)

// Creates a new real Valkey client from the passed-in valkey.Options.
func newValkeyClient(options valkey.ClientOption) (client, error) {
innerClient, err := valkey.NewClient(options)
if err != nil {
return nil, err
}
return &valkeyClient{innerClient}, nil
}

// Retrieve Valkey INFO. We retrieve all of the 'sections'.
func (c *valkeyClient) retrieveInfo(ctx context.Context) (map[string]string, error) {
str, err := c.client.Do(ctx, c.client.B().Info().Build()).ToString()
if err != nil {
return nil, err
}

return parseRawDataMap(str)
}

func parseRawDataMap(data string) (map[string]string, error) {
attrs := make(map[string]string)
lines := strings.Split(data, attrDelimiter)
for _, line := range lines {
if len(line) == 0 || strings.HasPrefix(line, "#") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lines with a # prefix appear to start a named section of key/value pairs. Is there a promise in the API that there will be no name conflicts across categories? If no, we may need to consider grouping the parsed pairs by category, not in one shared map.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since 1.0.0, there do not appear to have been any conflicts in the sections. See the documentation: https://valkey.io/commands/info/

Although I like the idea of grouping the data by section, I would suggest keeping this feature for a future PR if needed. The current exported client metrics are not affected by the tested version (integration test 7.2)." Wdyt?

continue
}
key, value, found := strings.Cut(line, ":")
if !found {
return nil, fmt.Errorf("could not cut line %q using \":\" as delimiter", line)
}
attrs[key] = value
}
return attrs, nil
}

// close client to release connection pool.
func (c *valkeyClient) close() error {
c.client.Close()
return nil
}
40 changes: 40 additions & 0 deletions receiver/valkeyreceiver/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package valkeyreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver"

import (
"fmt"
"net"

"go.opentelemetry.io/collector/config/confignet"
"go.opentelemetry.io/collector/config/configtls"
"go.opentelemetry.io/collector/receiver/scraperhelper"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver/internal/metadata"
)

type Config struct {
scraperhelper.ControllerConfig `mapstructure:",squash"`

// The target endpoint.
confignet.AddrConfig `mapstructure:",squash"`

TLS configtls.ClientConfig `mapstructure:"tls,omitempty"`

MetricsBuilderConfig metadata.MetricsBuilderConfig `mapstructure:",squash"`
}

// configInfo holds configuration information to be used as resource/metrics attributes.
type configInfo struct {
Address string
Port string
}

func newConfigInfo(cfg *Config) (configInfo, error) {
address, port, err := net.SplitHostPort(cfg.Endpoint)
if err != nil {
return configInfo{}, fmt.Errorf("invalid endpoint %q: %w", cfg.Endpoint, err)
}
return configInfo{Address: address, Port: port}, nil
}
6 changes: 6 additions & 0 deletions receiver/valkeyreceiver/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

//go:generate ../../.tools/mdatagen metadata.yaml

package valkeyreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver"
35 changes: 35 additions & 0 deletions receiver/valkeyreceiver/documentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
[comment]: <> (Code generated by mdatagen. DO NOT EDIT.)

# valkey

## Default Metrics

The following metrics are emitted by default. Each of them can be disabled by applying the following configuration:

```yaml
metrics:
<metric_name>:
enabled: false
```

### valkey.client.connection.count

The number of connections that are currently in state described by the state attribute

| Unit | Metric Type | Value Type |
| ---- | ----------- | ---------- |
| {connection} | Gauge | Int |

#### Attributes

| Name | Description | Values |
| ---- | ----------- | ------ |
| valkey.client.connection.state | The state of a connection in the pool | Str: ``used``, ``blocked``, ``tracking`` |

## Resource Attributes

| Name | Description | Values | Enabled |
| ---- | ----------- | ------ | ------- |
| server.address | Valkey server's address | Any Str | false |
| server.port | Valkey server's port | Any Str | false |
| valkey.version | Valkey server's version. | Any Str | true |
66 changes: 66 additions & 0 deletions receiver/valkeyreceiver/factory.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package valkeyreceiver // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver"

import (
"context"
"time"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/config/confignet"
"go.opentelemetry.io/collector/config/configtls"
"go.opentelemetry.io/collector/consumer"
"go.opentelemetry.io/collector/receiver"
"go.opentelemetry.io/collector/receiver/scraperhelper"
"go.opentelemetry.io/collector/scraper"

"github.com/open-telemetry/opentelemetry-collector-contrib/receiver/valkeyreceiver/internal/metadata"
)

// NewFactory creates a factory for Valkey receiver.
func NewFactory() receiver.Factory {
return receiver.NewFactory(
metadata.Type,
createDefaultConfig,
receiver.WithMetrics(createMetricsReceiver, metadata.MetricsStability))
}

func createDefaultConfig() component.Config {
scs := scraperhelper.NewDefaultControllerConfig()
scs.CollectionInterval = 10 * time.Second
return &Config{
AddrConfig: confignet.AddrConfig{
Transport: confignet.TransportTypeTCP,
},
TLS: configtls.ClientConfig{
Insecure: true,
},
ControllerConfig: scs,
MetricsBuilderConfig: metadata.DefaultMetricsBuilderConfig(),
}
}

func createMetricsReceiver(
_ context.Context,
set receiver.Settings,
cfg component.Config,
consumer consumer.Metrics,
) (receiver.Metrics, error) {
oCfg := cfg.(*Config)

scrp, err := newValkeyScraper(oCfg, set)
if err != nil {
return nil, err
}

scraper, err := scraper.NewMetrics(
scraper.ScrapeMetricsFunc(scrp.scrape),
scraper.WithShutdown(scrp.shutdown),
)
if err != nil {
return nil, err
}

return scraperhelper.NewScraperControllerReceiver(&oCfg.ControllerConfig, set, consumer, scraperhelper.AddScraper(metadata.Type, scraper))
}
Loading
Loading