Add UTF-8 support in metric and label names #689

fedetorres93 · 2024-09-12T15:47:55Z

As part of the initiative for adding UTF-8 support to Prometheus, this PR brings UTF-8 support for metric and label names as described in this proposal (for example: {"metric.name", "label.name"="value"}.

For UTF-8 label names defined in the URL path, an escaping syntax is used. For example, the label "label.name"="value" would be escaped like: /metrics/job/example/U__label_2e_name/value. This escaping is compatible with the base64 encoding for label values: /metrics/job/example/U__label_2e_name@base64/dmFsdWU=

UTF-8 support can be enabled by running the Pushgateway with the flag --push.enable-utf8-names=true

Addresses #623

Signed-off-by: Federico Torres <[email protected]>

beorn7 · 2024-09-18T16:23:11Z

Sorry, still fighting my backlog. I hope I get to review this tomorrow.

beorn7

Thank you for doing this.
A few ideas about how to precisely control whether to escape or not. I hope I got it right, but happy to learn where I'm wrong.

In different news, this needs documentation (which currently is just in the ever growing README.md – feel free to add it there, breaking down the documentation into "proper" documentation files would be a task for another PR).

beorn7 · 2024-09-20T13:18:49Z

handler/handler_test.go

@@ -18,6 +18,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"github.com/prometheus/common/model"


This has to go into the next block (line 29).

beorn7 · 2024-09-20T13:23:49Z

main.go

@@ -17,6 +17,7 @@ import (
 	"compress/gzip"
 	"context"
 	"fmt"
+	"github.com/prometheus/common/model"


As above, has to go to line 39 or so.

beorn7 · 2024-09-20T13:30:42Z

main.go

@@ -73,6 +74,7 @@ func main() {
 		persistenceFile     = app.Flag("persistence.file", "File to persist metrics. If empty, metrics are only kept in memory.").Default("").String()
 		persistenceInterval = app.Flag("persistence.interval", "The minimum interval at which to write out the persistence file.").Default("5m").Duration()
 		pushUnchecked       = app.Flag("push.disable-consistency-check", "Do not check consistency of pushed metrics. DANGEROUS.").Default("false").Bool()
+		pushEscaped         = app.Flag("push.enable-escaped-labels", "Allow escaped characters in label names in URLs.").Default("false").Bool()


Let's call this push.enable-utf8-names to mirror the feature flag in Prometheus, and also to correctly describe what we are doing. (Without the flag, the PGW will reject non-legacy characters in names even in the payload of a push.)
Variable name and doc string has to be adjusted accordingly.

main.go

beorn7 · 2024-09-20T13:39:51Z

handler/push.go

@@ -180,19 +180,21 @@ func splitLabels(labels string) (map[string]string, error) {
 	for i := 0; i < len(components)-1; i += 2 {
 		name, value := components[i], components[i+1]
 		trimmedName := strings.TrimSuffix(name, Base64Suffix)
+		unescapedName := model.UnescapeName(trimmedName, model.ValueEncodingEscaping)


We don't even want to try to unescape if the corresponding flag is not set. To not refer to the flag value directly here, maybe set a package-level variable of type model.EscapingScheme to either model.ValueEncodingEscaping or model.NoEscaping and then use that variable here.

beorn7 · 2024-09-20T13:40:18Z

handler/push.go

 		if !model.LabelNameRE.MatchString(trimmedName) ||
+			!model.LabelName(unescapedName).IsValid() ||


With this validity test, wo don't need the one in the previous line anymore, do wo?

Signed-off-by: Federico Torres <[email protected]>

fedetorres93 · 2024-09-23T21:52:33Z

@beorn7 Thanks for your feedback, the requested changes were addressed.

beorn7

Code looks good (modulu doc comment on push.EscapingScheme. But we also need documentation. (As suggested previously, I believe it is OK for now to add a section to the ever growing README.md, keeping in mind that eventually we would like to break it up into "proper" documentation.)

beorn7 · 2024-09-25T16:16:18Z

handler/push.go

@@ -41,6 +41,8 @@ const (
 	Base64Suffix = "@base64"
 )

+var EscapingScheme = model.NoEscaping


This could benefit from a doc comment.

Signed-off-by: Federico Torres <[email protected]>

fedetorres93 · 2024-09-25T21:36:18Z

@beorn7 Sorry about the documentation, thanks for pointing it out, now it's added. Also added EscapingScheme comment.

beorn7

Cool. I just couldn't resist and applied a lot of wordsmithing. Please see my suggestions.

beorn7 · 2024-09-26T16:53:21Z

README.md

@@ -320,6 +320,28 @@ Examples:

      /metrics/job/titan/name@base64/zqDPgc6_zrzOt864zrXPjc-C

+### UTF-8 support


Let's be a bit more specific here and call this "UTF-8 support for metric and label names".
(UTF-8 is supported for label values already.)

beorn7 · 2024-09-26T17:21:10Z

README.md

+UTF-8 characters in metric and label names are supported by enabling
+the `--push.enable-utf8-names` flag. The syntax is as follows:
+
+    {"some.metric", "foo.bar"="baz"}
+
+For UTF-8 label names defined in the URL path, [an escaping syntax](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping) is used.


I would not explain the syntax of the text exposition format here. (It also doesn't cover protobuf.)

With some wordsmithing included, I would propose the following:

Suggested change

UTF-8 characters in metric and label names are supported by enabling

the `--push.enable-utf8-names` flag. The syntax is as follows:

{"some.metric", "foo.bar"="baz"}

For UTF-8 label names defined in the URL path, [an escaping syntax](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping) is used.

Newer versions of the Prometheus exposition formats (text and protobuf)

support the full UTF-8 character set in metric and label names. The

Pushgateway only accepts special characters in names if the command line

flag `--push.enable-utf8-names` is set.

To allow special characters in label names that are part of the URL path, the flag also enables a

[specific encoding mechanism](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping). This is similar to the base64 encoding for label _values_ described above,

but works differently in detail for technical and historical reasons. As before, client libraries

should usually take care of the encoding. It works as follows:

beorn7 · 2024-09-26T17:40:36Z

README.md

+* Prefix the string with `U__`.
+* All non-valid characters (i.e. characters other than letters, numbers, and underscores) will be encoded as underscores surrounding the Unicode value, like `_1F60A_`.
+* All pre-existing underscores will become doubled: `__`.
+* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).


Let's stay consistent with regard to the form. You start with an imperative form and then use future tense.

Here my suggestion with a few things added:

Suggested change

* Prefix the string with `U__`.

* All non-valid characters (i.e. characters other than letters, numbers, and underscores) will be encoded as underscores surrounding the Unicode value, like `_1F60A_`.

* All pre-existing underscores will become doubled: `__`.

* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).

* A label name containing encoded characters starts with `U__`.

* All non-standard characters (i.e. characters other than letters, numbers, and underscores) are encoded as underscores surrounding their Unicode value, like `_1F60A_`.

* All pre-existing underscores are encoded as a double-underscore: `__`.

* If a label name starts with `U__` already, these characters have to be encoded as well, resulting in `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).

* A label name starting with `U__` in it's encoded form, but containing invalid sequences (e.g. `U__in_xxx_valid`) is left unchanged.

beorn7 · 2024-09-26T17:41:24Z

README.md

+* All pre-existing underscores will become doubled: `__`.
+* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).
+
+For example, the label `"foo.bar"="baz"` would be escaped like: 


Let's better use "encoded" rather than "escaped" to only use one word throughout this file.

beorn7 · 2024-09-26T17:41:44Z

README.md

+
+    /metrics/job/example/U__foo_2e_bar/baz
+
+This escaping is compatible with the base64 encoding for label values:


escaping → encoding, see above.

beorn7 · 2024-09-26T17:42:30Z

README.md

+This escaping is compatible with the base64 encoding for label values:
+
+    /metrics/job/example/U__foo_2e_bar@base64/YmF6
+


Let's add something about the one edge case, e.g.

Note that this method has an unlikely edge case that is not handled properly: A pusher unaware of the encoding mechanism might use a label name that is also a valid encoded version of another label name. E.g. if a pusher intends to use the label name U__foo_2e_bar, but doesn't encode it as U___55_____foo__2e__bar, the Pushgateway will decode U__foo_2e_bar to foo.bar. This is the main reason why the decoding is opt-in via the --push.enable-utf8-names flag.

Signed-off-by: Federico Torres <[email protected]>

fedetorres93 · 2024-09-27T14:47:04Z

@beorn7 Thanks for the docs suggestions, done.

beorn7

🚢

fedetorres93 added 3 commits September 4, 2024 10:56

Allow UTF-8 escaped characters in label names in URLs

81d3593

Signed-off-by: Federico Torres <[email protected]>

Add unit tests

519870f

Signed-off-by: Federico Torres <[email protected]>

Merge remote-tracking branch 'upstream/master' into ftorres/utf-8

83c45eb

fedetorres93 marked this pull request as ready for review September 12, 2024 16:01

beorn7 self-assigned this Sep 17, 2024

beorn7 requested changes Sep 20, 2024

View reviewed changes

fedetorres93 force-pushed the ftorres/utf-8 branch 2 times, most recently from ab2490d to 83c45eb Compare September 23, 2024 21:48

Change UTF-8 flag name and validations

90f4e80

Signed-off-by: Federico Torres <[email protected]>

fedetorres93 requested a review from beorn7 September 23, 2024 21:51

beorn7 requested changes Sep 25, 2024

View reviewed changes

Add documentation

1c39f43

Signed-off-by: Federico Torres <[email protected]>

fedetorres93 requested a review from beorn7 September 25, 2024 21:33

beorn7 requested changes Sep 26, 2024

View reviewed changes

Documentation changes

62d5b8a

Signed-off-by: Federico Torres <[email protected]>

fedetorres93 requested a review from beorn7 September 27, 2024 14:47

beorn7 approved these changes Oct 2, 2024

View reviewed changes

beorn7 merged commit c408b8e into prometheus:master Oct 2, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add UTF-8 support in metric and label names #689

Add UTF-8 support in metric and label names #689

fedetorres93 commented Sep 12, 2024 •

edited

Loading

beorn7 commented Sep 18, 2024

beorn7 left a comment

beorn7 Sep 20, 2024

beorn7 Sep 20, 2024

beorn7 Sep 20, 2024

beorn7 Sep 20, 2024

beorn7 Sep 20, 2024

fedetorres93 commented Sep 23, 2024

beorn7 left a comment

beorn7 Sep 25, 2024

fedetorres93 commented Sep 25, 2024

beorn7 left a comment

beorn7 Sep 26, 2024

beorn7 Sep 26, 2024

beorn7 Sep 26, 2024

beorn7 Sep 26, 2024

beorn7 Sep 26, 2024

beorn7 Sep 26, 2024

fedetorres93 commented Sep 27, 2024

beorn7 left a comment

		if !model.LabelNameRE.MatchString(trimmedName) \|\|
		!model.LabelName(unescapedName).IsValid() \|\|

		@@ -320,6 +320,28 @@ Examples:

		/metrics/job/titan/name@base64/zqDPgc6_zrzOt864zrXPjc-C

		### UTF-8 support

-UTF-8 characters in metric and label names are supported by enabling
-the `--push.enable-utf8-names` flag. The syntax is as follows:
-    {"some.metric", "foo.bar"="baz"}
-For UTF-8 label names defined in the URL path, [an escaping syntax](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping) is used.
+Newer versions of the Prometheus exposition formats (text and protobuf)
+support the full UTF-8 character set in metric and label names. The
+Pushgateway only accepts special characters in names if the command line
+flag `--push.enable-utf8-names` is set.
+To allow special characters in label names that are part of the URL path, the flag also enables a
+[specific encoding mechanism](https://github.com/prometheus/proposals/blob/main/proposals/2023-08-21-utf8.md#text-escaping). This is similar to the base64 encoding for label _values_ described above,
+but works differently in detail for technical and historical reasons. As before, client libraries
+should usually take care of the encoding. It works as follows:

-* Prefix the string with `U__`.
-* All non-valid characters (i.e. characters other than letters, numbers, and underscores) will be encoded as underscores surrounding the Unicode value, like `_1F60A_`.
-* All pre-existing underscores will become doubled: `__`.
-* If a string should start with "U__" already, it will need to be escaped: `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).
+* A label name containing encoded characters starts with `U__`.
+* All non-standard characters (i.e. characters other than letters, numbers, and underscores) are encoded as underscores surrounding their Unicode value, like `_1F60A_`.
+* All pre-existing underscores are encoded as a double-underscore: `__`.
+* If a label name starts with `U__` already, these characters have to be encoded as well, resulting in `U___55_____`. (That's `U__` + `_55_` (for `U`) + `__` + `__`).
+* A label name starting with `U__` in it's encoded form, but containing invalid sequences (e.g. `U__in_xxx_valid`) is left unchanged.


		/metrics/job/example/U__foo_2e_bar/baz

		This escaping is compatible with the base64 encoding for label values:

		This escaping is compatible with the base64 encoding for label values:

		/metrics/job/example/U__foo_2e_bar@base64/YmF6

Add UTF-8 support in metric and label names #689

Add UTF-8 support in metric and label names #689

Conversation

fedetorres93 commented Sep 12, 2024 • edited Loading

beorn7 commented Sep 18, 2024

beorn7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedetorres93 commented Sep 23, 2024

beorn7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedetorres93 commented Sep 25, 2024

beorn7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedetorres93 commented Sep 27, 2024

beorn7 left a comment

Choose a reason for hiding this comment

fedetorres93 commented Sep 12, 2024 •

edited

Loading