Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http instrumentation does not support Go < 1.19 #1771

Open
MrAlias opened this issue Feb 7, 2025 · 6 comments
Open

net/http instrumentation does not support Go < 1.19 #1771

MrAlias opened this issue Feb 7, 2025 · 6 comments
Labels
bug Something isn't working
Milestone

Comments

@MrAlias
Copy link
Contributor

MrAlias commented Feb 7, 2025

Describe the bug

Running the auto-instrumentation for a service using Go < 1.19 will fail to find offsets needed for the net/http instrumentation. This causes the auto-instrumentation to crash.

Environment

  • OS: Linux
  • Go Version: 1.16
  • Version: 785339c

To Reproduce

Update the database end-2-end test:

diff --git a/internal/test/e2e/databasesql/Dockerfile b/internal/test/e2e/databasesql/Dockerfile
index a50dfc75..48551411 100644
--- a/internal/test/e2e/databasesql/Dockerfile
+++ b/internal/test/e2e/databasesql/Dockerfile
@@ -1,4 +1,4 @@
-FROM golang:1.23.6@sha256:927112936d6b496ed95f55f362cc09da6e3e624ef868814c56d55bd7323e0959
+FROM golang:1.16

 # pre-copy/cache go.mod for pre-downloading dependencies and only redownloading
 # them in subsequent builds if they change
diff --git a/internal/test/e2e/databasesql/go.mod b/internal/test/e2e/databasesql/go.mod
index bb73bb66..ca339c81 100644
--- a/internal/test/e2e/databasesql/go.mod
+++ b/internal/test/e2e/databasesql/go.mod
@@ -1,5 +1,5 @@
 module go.opentelemetry.io/auto/internal/test/e2e/databasesql

-go 1.22.0
+go 1.16

-require github.com/mattn/go-sqlite3 v1.14.24
+require github.com/mattn/go-sqlite3 v1.14.1
diff --git a/internal/test/e2e/databasesql/go.sum b/internal/test/e2e/databasesql/go.sum
index 9dcdc9b6..49dee20a 100644
--- a/internal/test/e2e/databasesql/go.sum
+++ b/internal/test/e2e/databasesql/go.sum
@@ -1,2 +1,11 @@
-github.com/mattn/go-sqlite3 v1.14.24 h1:tpSp2G2KyMnnQu99ngJ47EIkWVmliIizyZBfPrBWDRM=
-github.com/mattn/go-sqlite3 v1.14.24/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
+github.com/PuerkitoBio/goquery v1.5.1/go.mod h1:GsLWisAFVj4WgDibEWF4pvYnkVQBpKBKeU+7zCJoLcc=
+github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
+github.com/mattn/go-sqlite3 v1.14.1 h1:AHx9Ra40wIzl+GelgX2X6AWxmT5tfxhI1PL0523HcSw=
+github.com/mattn/go-sqlite3 v1.14.1/go.mod h1:JIl7NbARA7phWnGvh0LKTyg7S9BA+6gx71ShQilpsus=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
diff --git a/internal/test/e2e/databasesql/main.go b/internal/test/e2e/databasesql/main.go
index 67d83655..29471066 100644
--- a/internal/test/e2e/databasesql/main.go
+++ b/internal/test/e2e/databasesql/main.go
@@ -7,7 +7,7 @@ import (
        "database/sql"
        "fmt"
        "io"
-       "log/slog"
+       "log"
        "net/http"
        "os"
        "time"
@@ -80,11 +80,11 @@ func (s *Server) query(w http.ResponseWriter, req *http.Request, query string) {

        rows, err := conn.QueryContext(req.Context(), query)
        if err != nil {
-               slog.Error("query failed", "query", query, "error", err)
+               log.Println("query failed", "query", query, "error", err)
                return
        }

-       slog.Info("queryDB called", "query", query)
+       log.Println("queryDB called", "query", query)
        for rows.Next() {
                var id int
                var firstName string
@@ -125,7 +125,7 @@ func (s *Server) invalid(w http.ResponseWriter, req *http.Request) {

 func main() {
        port := fmt.Sprintf(":%d", 8080)
-       slog.Info("starting http server", "port", port)
+       log.Println("starting http server", "port", port)

        s := NewServer()

@@ -156,14 +156,14 @@ func main() {
        for _, t := range tests {
                resp, err := http.Get(t.url)
                if err != nil {
-                       slog.Error("failed GET", "error", err)
+                       log.Println("failed GET", "error", err)
                }
                if resp != nil {
                        body, err := io.ReadAll(resp.Body)
                        if err != nil {
-                               slog.Error("failed to read HTTP body", "error", err)
+                               log.Println("failed to read HTTP body", "error", err)
                        }
-                       slog.Info("request successful", "body", string(body[:]))
+                       log.Println("request successful", "body", string(body[:]))
                        _ = resp.Body.Close()
                }
        }

Run the fixtures test: make fixture-databasesql

See error:

{"time":"2025-02-07T18:35:51.009499004Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Fragment"},"offset":112}
{"time":"2025-02-07T18:35:51.009505453Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"raw_fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"RawFragment"},"offset":128}
{"time":"2025-02-07T18:35:51.009511623Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"username_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"Userinfo","Field":"username"},"offset":0}
{"time":"2025-02-07T18:35:51.009521122Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"url_host_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Host"},"offset":40}
{"time":"2025-02-07T18:35:51.009527558Z","level":"ERROR","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/app/internal/pkg/instrumentation/manager.go","line":377},"msg":"error while loading probes, cleaning up","error":"failed to find offset for \"net/url.URL:OmitHost\": struct field \"OmitHost\" not found: DWARF entry not found","name":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:35:51.009538489Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:35:51.009553532Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":3,"InstrumentedPkg":"database/sql"}}
{"time":"2025-02-07T18:35:51.009558969Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":2,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:35:51.009602545Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).cleanup","file":"/app/internal/pkg/instrumentation/manager.go","line":409},"msg":"Cleaning bpffs"}
{"time":"2025-02-07T18:35:51.009637196Z","level":"ERROR","source":{"function":"main.main","file":"/app/cli/main.go","line":125},"msg":"failed to load instrumentation","error":"failed to find offset for \"net/url.URL:OmitHost\": struct field \"OmitHost\" not found: DWARF entry not found"}
{"time":"2025-02-07T18:35:31.202496796Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Fragment"},"offset":112}
{"time":"2025-02-07T18:35:31.202508072Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"raw_fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"RawFragment"},"offset":128}
{"time":"2025-02-07T18:35:31.202522594Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"username_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"Userinfo","Field":"username"},"offset":0}
{"time":"2025-02-07T18:35:31.202532152Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"url_host_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Host"},"offset":40}
{"time":"2025-02-07T18:35:31.202546567Z","level":"ERROR","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/app/internal/pkg/instrumentation/manager.go","line":377},"msg":"error while loading probes, cleaning up","error":"failed to find offset for \"net/url.URL:OmitHost\": struct field \"OmitHost\" not found: DWARF entry not found","name":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:35:31.389754514Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":2,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:35:31.390015392Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:35:31.390101089Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":3,"InstrumentedPkg":"database/sql"}}
{"time":"2025-02-07T18:35:31.390497631Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).cleanup","file":"/app/internal/pkg/instrumentation/manager.go","line":409},"msg":"Cleaning bpffs"}
{"time":"2025-02-07T18:35:31.390958013Z","level":"ERROR","source":{"function":"main.main","file":"/app/cli/main.go","line":125},"msg":"failed to load instrumentation","error":"failed to find offset for \"net/url.URL:OmitHost\": struct field \"OmitHost\" not found: DWARF entry not found"}
{"time":"2025-02-07T18:36:21.056606092Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Fragment"},"offset":112}
{"time":"2025-02-07T18:36:21.056612712Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"raw_fragment_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"RawFragment"},"offset":128}
{"time":"2025-02-07T18:36:21.056618722Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"username_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"Userinfo","Field":"username"},"offset":0}
{"time":"2025-02-07T18:36:21.056624321Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.StructFieldConst.InjectOption","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":512},"msg":"Offset found","key":"url_host_pos","id":{"ModPath":"std","PkgPath":"net/url","Struct":"URL","Field":"Host"},"offset":40}
{"time":"2025-02-07T18:36:21.05663412Z","level":"ERROR","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).loadProbes","file":"/app/internal/pkg/instrumentation/manager.go","line":377},"msg":"error while loading probes, cleaning up","error":"failed to find offset for \"net/url.URL:OmitHost\": struct field \"OmitHost\" not found: DWARF entry not found","name":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:36:21.056646048Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:36:21.056664146Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":2,"InstrumentedPkg":"net/http"}}
{"time":"2025-02-07T18:36:21.056673075Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation/probe.(*Base[...]).Close","file":"/app/internal/pkg/instrumentation/probe/probe.go","line":303},"msg":"Closed","Probe":{"SpanKind":3,"InstrumentedPkg":"database/sql"}}
{"time":"2025-02-07T18:36:21.056747103Z","level":"DEBUG","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).cleanup","file":"/app/internal/pkg/instrumentation/manager.go","line":409},"msg":"Cleaning bpffs"}
{"time":"2025-02-07T18:36:21.056778218Z","level":"ERROR","source":{"function":"main.main","file":"/app/cli/main.go","line":125},"msg":"failed to load instrumentation","error":"failed to find offset for \"net/url.URL:OmitHost\": struct field \"OmitHost\" not found: DWARF entry not found"}

Expected behavior

Compatibility docs state Go 1.12 is supported. Therefore, this shouldn't fail.

Additional context

Missing offset:

"field": "OmitHost",
"offsets": [
{
"offset": null,
"versions": [
"1.12.0",
"1.12.1",
"1.12.2",
"1.12.3",
"1.12.4",
"1.12.5",
"1.12.6",
"1.12.7",
"1.12.8",
"1.12.9",
"1.12.10",
"1.12.11",
"1.12.12",
"1.12.13",
"1.12.14",
"1.12.15",
"1.12.16",
"1.12.17",
"1.13.0",
"1.13.1",
"1.13.2",
"1.13.3",
"1.13.4",
"1.13.5",
"1.13.6",
"1.13.7",
"1.13.8",
"1.13.9",
"1.13.10",
"1.13.11",
"1.13.12",
"1.13.13",
"1.13.14",
"1.13.15",
"1.14.0",
"1.14.1",
"1.14.2",
"1.14.3",
"1.14.4",
"1.14.5",
"1.14.6",
"1.14.7",
"1.14.8",
"1.14.9",
"1.14.10",
"1.14.11",
"1.14.12",
"1.14.13",
"1.14.14",
"1.14.15",
"1.15.0",
"1.15.1",
"1.15.2",
"1.15.3",
"1.15.4",
"1.15.5",
"1.15.6",
"1.15.7",
"1.15.8",
"1.15.9",
"1.15.10",
"1.15.11",
"1.15.12",
"1.15.13",
"1.15.14",
"1.15.15",
"1.16.0",
"1.16.1",
"1.16.2",
"1.16.3",
"1.16.4",
"1.16.5",
"1.16.6",
"1.16.7",
"1.16.8",
"1.16.9",
"1.16.10",
"1.16.11",
"1.16.12",
"1.16.13",
"1.16.14",
"1.16.15",
"1.17.0",
"1.17.1",
"1.17.2",
"1.17.3",
"1.17.4",
"1.17.5",
"1.17.6",
"1.17.7",
"1.17.8",
"1.17.9",
"1.17.10",
"1.17.11",
"1.17.12",
"1.17.13",
"1.18.0",
"1.18.1",
"1.18.2",
"1.18.3",
"1.18.4",
"1.18.5",
"1.18.6",
"1.18.7",
"1.18.8",
"1.18.9",
"1.18.10"
]
},
{
"offset": 88,
"versions": [
"1.19.0",
"1.19.1",
"1.19.2",
"1.19.3",
"1.19.4",
"1.19.5",
"1.19.6",
"1.19.7",
"1.19.8",
"1.19.9",
"1.19.10",
"1.19.11",
"1.19.12",
"1.19.13",
"1.20.0",
"1.20.1",
"1.20.2",
"1.20.3",
"1.20.4",
"1.20.5",
"1.20.6",
"1.20.7",
"1.20.8",
"1.20.9",
"1.20.10",
"1.20.11",
"1.20.12",
"1.20.13",
"1.20.14",
"1.21.0",
"1.21.1",
"1.21.2",
"1.21.3",
"1.21.4",
"1.21.5",
"1.21.6",
"1.21.7",
"1.21.8",
"1.21.9",
"1.21.10",
"1.21.11",
"1.21.12",
"1.21.13",
"1.22.0",
"1.22.1",
"1.22.2",
"1.22.3",
"1.22.4",
"1.22.5",
"1.22.6",
"1.22.7",
"1.22.8",
"1.22.9",
"1.22.10",
"1.22.11",
"1.22.12",
"1.23.0",
"1.23.1",
"1.23.2",
"1.23.3",
"1.23.4",
"1.23.5",
"1.23.6"
]

Note: there are many other offsets not available for Go 1.12.

@MrAlias MrAlias added the bug Something isn't working label Feb 7, 2025
@MrAlias
Copy link
Contributor Author

MrAlias commented Feb 7, 2025

cc @damemi

@MrAlias
Copy link
Contributor Author

MrAlias commented Feb 7, 2025

The fact this hasn't been reported yet supports us dropping support for Go < 1.19.

@MrAlias
Copy link
Contributor Author

MrAlias commented Feb 7, 2025

The fact this hasn't been reported yet supports us dropping support for Go < 1.19.

This has been broken for 9 months: #790

That is 8 releases without it being reported.

@damemi
Copy link
Contributor

damemi commented Feb 7, 2025

+1 I've always been in favor of adopting a reasonable support timeline as an open source project

@MrAlias MrAlias added this to the v0.21.0 milestone Feb 7, 2025
@RonFed
Copy link
Contributor

RonFed commented Feb 8, 2025

+1 on supporting versions >= 1.17.
The difference for us to maintain 1.17-1.19 is not large IMO.

@MrAlias
Copy link
Contributor Author

MrAlias commented Feb 11, 2025

SIG meeting notes:

  • Dropping support for Go < 1.19 is reasonable for the development phase of this project
  • We can add support back in the future in non-breaking manner
  • We need to define our support policy for rolling releases like Go as we stabilize this project
  • We may decide to also only support the versions of Go that are officially supported upstream
  • @MrAlias is going to take an action item to create an issue that supersedes this issue tracking the work to drop support for Go < 1.19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants