Skip to content

Commit 7f16f49

Browse files
[v2][query] Create archive reader/writer using regular factory methods (jaegertracing#6519)
## Which problem is this PR solving? - Towards jaegertracing#6065 ## Description of the changes - This PR changes the jaegerquery extension to remove usages of `CreateArchiveSpanReader` and `CreateArchiveSpanWriter` and replace them with their primary storage counterparts from the v2 storage API `CreateTraceReader` and `CreateTraceWriter`. - 🛑 **This PR contains a breaking changes for users of jaeger-v2 that have an ElasticSearch storage configured for `traces_archive`** 🛑 - The distinction between primary and archive storage was removed which causes some breaking changes that can be remediated. Refer to Step 9 of the test report below for remediation steps. ## Test Report ### 1. Establish Ground Truth on `main` To begin, configure the storage and query settings in the `all-in-one.yaml` file as follows: ```yaml jaeger_storage: backends: some_storage: elasticsearch: indices: index_prefix: "jaeger-main" spans: date_layout: "2006-01-02" rollover_frequency: "day" shards: 5 replicas: 1 services: date_layout: "2006-01-02" rollover_frequency: "day" shards: 5 replicas: 1 dependencies: date_layout: "2006-01-02" rollover_frequency: "day" shards: 5 replicas: 1 sampling: date_layout: "2006-01-02" rollover_frequency: "day" shards: 5 replicas: 1 another_storage: elasticsearch: indices: index_prefix: "jaeger-archive" ``` ### 2. Spin Up Elasticsearch Using Docker To bring up Elasticsearch, run the following commands: ```zsh jaeger % cd docker-compose/elasticsearch/v8 v8 % docker compose up ``` ### 3. Start Jaeger Start the Jaeger service: ```zsh jaeger % go run ./cmd/jaeger ``` ### 4. Archive a Trace From the Jaeger UI, select a trace and archive it. **traceID**: `0dc3e460bd9b8e0dddfa29a2f751cfb9` ![Trace Screenshot](https://github.com/user-attachments/assets/c9ef72a6-30da-4496-9356-20b5db42663f) ### 5. Update `index_prefix` Stop Jaeger and modify the `index_prefix` in the primary configuration to `jaeger-main-1`. This ensures the query for the same trace is no longer found in the primary storage but will be found in the archive storage. ### 6. Query for the Same Trace ```zsh curl -s http://localhost:16686/api/traces/0dc3e460bd9b8e0dddfa29a2f751cfb9 | jq '.data[].spans[] | {traceID, operationName}' { "traceID": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "operationName": "/api/services" } { "traceID": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "operationName": "GetService" } ``` ```zsh curl -s http://localhost:16686/api/v3/traces/0dc3e460bd9b8e0dddfa29a2f751cfb9 | jq '.result.resourceSpans[].scopeSpans[].spans[] | {traceId, name}' { "traceId": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "name": "GetService" } { "traceId": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "name": "/api/services" } ``` ### 7. Test Changes from This PR Stop Jaeger, checkout this PR, and restart Jaeger: ```zsh gh pr checkout 6519 go run ./cmd/jaeger ``` ### 8. Query for the Same Trace After PR Changes ```zsh curl -s http://localhost:16686/api/traces/0dc3e460bd9b8e0dddfa29a2f751cfb9 | jq . { "data": null, "total": 0, "limit": 0, "offset": 0, "errors": [ { "code": 404, "msg": "trace not found" } ] } ``` ```zsh curl -s http://localhost:16686/api/v3/traces/0dc3e460bd9b8e0dddfa29a2f751cfb9 | jq . { "error": { "httpCode": 404, "message": "No traces found" } } ``` 🛑 **This is where the breaking change occurs** 🛑 ### 9. Mitigation for Users #### Set `use_aliases` for Archive Storage To mitigate the issue, set the `use_aliases` configuration for your archive storage to `true`. Update the configuration as follows: ```yaml another_storage: elasticsearch: indices: index_prefix: "jaeger-archive" use_aliases: true ``` #### Add Alias from Old Index to New Index To ensure backwards compatibility, add an alias from the old index to the new index. You can query the current set of aliases in Elasticsearch with: ```bash curl -X GET "http://localhost:9200/_aliases?pretty" ``` ```json { "jaeger-main-1-jaeger-span-2025-01-16": { "aliases": {} }, "jaeger-main-2-jaeger-span-2025-01-16": { "aliases": {} }, "jaeger-archive-jaeger-span-archive": { "aliases": {} }, "jaeger-main-1-jaeger-service-2025-01-16": { "aliases": {} } } ``` To link the new index (`jaeger-archive-jaeger-span-read`) with the old index (`jaeger-archive-jaeger-span-archive`), run the following: ```bash curl -X POST "http://localhost:9200/_aliases" -H 'Content-Type: application/json' -d' { "actions": [ { "add": { "index": "jaeger-archive-jaeger-span-archive", "alias": "jaeger-archive-jaeger-span-read" } } ] }' ``` Confirm that the alias has been added: ```bash curl -X GET "http://localhost:9200/_aliases?pretty" ``` ```json { "jaeger-main-1-jaeger-span-2025-01-16": { "aliases": {} }, "jaeger-archive-jaeger-span-archive": { "aliases": { "jaeger-archive-jaeger-span-read": {} } }, "jaeger-main-2-jaeger-span-2025-01-16": { "aliases": {} }, "jaeger-main-2-jaeger-service-2025-01-16": { "aliases": {} }, "jaeger-main-1-jaeger-service-2025-01-16": { "aliases": {} } } ``` Note that if you already had `use_aliases` set to true for your archive storage before the upgrade, then Jaeger would've been using an index name with -read suffix: `jaeger-archive-jaeger-span-archive-read`. Then for the example above, you would create an alias `jaeger-archive-jaeger-span-read` pointing to `jaeger-archive-jaeger-span-archive-read`. ### 10. Restart Jaeger and Retry the Query Finally, restart Jaeger and run the same trace queries again: ```zsh curl -s http://localhost:16686/api/traces/0dc3e460bd9b8e0dddfa29a2f751cfb9 | jq '.data[].spans[] | {traceID, operationName}' { "traceID": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "operationName": "/api/services" } { "traceID": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "operationName": "GetService" } ``` ```zsh curl -s http://localhost:16686/api/v3/traces/0dc3e460bd9b8e0dddfa29a2f751cfb9 | jq '.result.resourceSpans[].scopeSpans[].spans[] | {traceId, name}' { "traceId": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "name": "/api/services" } { "traceId": "0dc3e460bd9b8e0dddfa29a2f751cfb9", "name": "GetService" } ``` ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [x] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `npm run lint` and `npm run test` --------- Signed-off-by: Mahad Zaryab <[email protected]> Co-authored-by: Yuri Shkuro <[email protected]>
1 parent 463f5f3 commit 7f16f49

File tree

8 files changed

+117
-52
lines changed

8 files changed

+117
-52
lines changed

cmd/collector/app/span_processor.go

+1-2
Original file line numberDiff line numberDiff line change
@@ -227,8 +227,7 @@ func (sp *spanProcessor) saveSpan(span *model.Span, tenant string) {
227227
}
228228

229229
func (sp *spanProcessor) writeSpan(ctx context.Context, span *model.Span) error {
230-
spanWriter, err := v1adapter.GetV1Writer(sp.traceWriter)
231-
if err == nil {
230+
if spanWriter, ok := v1adapter.GetV1Writer(sp.traceWriter); ok {
232231
return spanWriter.WriteSpan(ctx, span)
233232
}
234233
traces := v1adapter.V1BatchesToTraces([]*model.Batch{{Spans: []*model.Span{span}}})

cmd/jaeger/internal/extension/jaegerquery/server.go

+46-12
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import (
1111
"go.opentelemetry.io/collector/component"
1212
"go.opentelemetry.io/collector/extension"
1313
"go.opentelemetry.io/collector/extension/extensioncapabilities"
14+
"go.uber.org/zap"
1415

1516
"github.com/jaegertracing/jaeger/cmd/jaeger/internal/extension/jaegerstorage"
1617
queryApp "github.com/jaegertracing/jaeger/cmd/query/app"
@@ -22,7 +23,9 @@ import (
2223
"github.com/jaegertracing/jaeger/pkg/tenancy"
2324
"github.com/jaegertracing/jaeger/plugin/metricstore/disabled"
2425
"github.com/jaegertracing/jaeger/storage/metricstore"
26+
"github.com/jaegertracing/jaeger/storage/spanstore"
2527
"github.com/jaegertracing/jaeger/storage_v2/depstore"
28+
"github.com/jaegertracing/jaeger/storage_v2/tracestore"
2629
"github.com/jaegertracing/jaeger/storage_v2/v1adapter"
2730
)
2831

@@ -94,7 +97,6 @@ func (s *server) Start(ctx context.Context, host component.Host) error {
9497

9598
var opts querysvc.QueryServiceOptions
9699
var v2opts v2querysvc.QueryServiceOptions
97-
// TODO archive storage still uses v1 factory
98100
if err := s.addArchiveStorage(&opts, &v2opts, host); err != nil {
99101
return err
100102
}
@@ -140,26 +142,58 @@ func (s *server) addArchiveStorage(
140142
return nil
141143
}
142144

143-
f, err := jaegerstorage.GetStorageFactory(s.config.Storage.TracesArchive, host)
145+
f, err := jaegerstorage.GetTraceStoreFactory(s.config.Storage.TracesArchive, host)
144146
if err != nil {
145-
return fmt.Errorf("cannot find archive storage factory: %w", err)
147+
return fmt.Errorf("cannot find traces archive storage factory: %w", err)
146148
}
147149

148-
if !opts.InitArchiveStorage(f, s.telset.Logger) {
149-
s.telset.Logger.Info("Archive storage not initialized")
150+
traceReader, traceWriter := s.initArchiveStorage(f)
151+
if traceReader == nil || traceWriter == nil {
152+
return nil
150153
}
151154

152-
ar, aw := v1adapter.InitializeArchiveStorage(f, s.telset.Logger)
153-
if ar != nil && aw != nil {
154-
v2opts.ArchiveTraceReader = ar
155-
v2opts.ArchiveTraceWriter = aw
156-
} else {
157-
s.telset.Logger.Info("Archive storage not initialized")
158-
}
155+
v2opts.ArchiveTraceReader = traceReader
156+
v2opts.ArchiveTraceWriter = traceWriter
157+
158+
spanReader, spanWriter := getV1Adapters(traceReader, traceWriter)
159+
160+
opts.ArchiveSpanReader = spanReader
161+
opts.ArchiveSpanWriter = spanWriter
159162

160163
return nil
161164
}
162165

166+
func (s *server) initArchiveStorage(f tracestore.Factory) (tracestore.Reader, tracestore.Writer) {
167+
reader, err := f.CreateTraceReader()
168+
if err != nil {
169+
s.telset.Logger.Error("Cannot init traces archive storage reader", zap.Error(err))
170+
return nil, nil
171+
}
172+
writer, err := f.CreateTraceWriter()
173+
if err != nil {
174+
s.telset.Logger.Error("Cannot init traces archive storage writer", zap.Error(err))
175+
return nil, nil
176+
}
177+
return reader, writer
178+
}
179+
180+
func getV1Adapters(
181+
reader tracestore.Reader,
182+
writer tracestore.Writer,
183+
) (spanstore.Reader, spanstore.Writer) {
184+
v1Reader, ok := v1adapter.GetV1Reader(reader)
185+
if !ok {
186+
v1Reader = v1adapter.NewSpanReader(reader)
187+
}
188+
189+
v1Writer, ok := v1adapter.GetV1Writer(writer)
190+
if !ok {
191+
v1Writer = v1adapter.NewSpanWriter(writer)
192+
}
193+
194+
return v1Reader, v1Writer
195+
}
196+
163197
func (s *server) createMetricReader(host component.Host) (metricstore.Reader, error) {
164198
if s.config.Storage.Metrics == "" {
165199
s.telset.Logger.Info("Metric storage not configured")

cmd/jaeger/internal/extension/jaegerquery/server_test.go

+54-18
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ import (
3636
metricstoremocks "github.com/jaegertracing/jaeger/storage/metricstore/mocks"
3737
"github.com/jaegertracing/jaeger/storage/spanstore"
3838
spanstoremocks "github.com/jaegertracing/jaeger/storage/spanstore/mocks"
39+
"github.com/jaegertracing/jaeger/storage_v2/tracestore"
40+
tracestoremocks "github.com/jaegertracing/jaeger/storage_v2/tracestore/mocks"
41+
"github.com/jaegertracing/jaeger/storage_v2/v1adapter"
3942
)
4043

4144
type fakeFactory struct {
@@ -63,14 +66,6 @@ func (ff fakeFactory) CreateSpanWriter() (spanstore.Writer, error) {
6366
return &spanstoremocks.Writer{}, nil
6467
}
6568

66-
func (fakeFactory) CreateArchiveSpanReader() (spanstore.Reader, error) {
67-
return &spanstoremocks.Reader{}, nil
68-
}
69-
70-
func (fakeFactory) CreateArchiveSpanWriter() (spanstore.Writer, error) {
71-
return &spanstoremocks.Writer{}, nil
72-
}
73-
7469
func (ff fakeFactory) Initialize(metrics.Factory, *zap.Logger) error {
7570
if ff.name == "need-initialize-error" {
7671
return errors.New("test-error")
@@ -104,11 +99,6 @@ var _ jaegerstorage.Extension = (*fakeStorageExt)(nil)
10499
func (fakeStorageExt) TraceStorageFactory(name string) (storage.Factory, bool) {
105100
if name == "need-factory-error" {
106101
return nil, false
107-
} else if name == "no-archive" {
108-
f := fakeFactory{name: name}
109-
return struct {
110-
storage.Factory
111-
}{f}, true
112102
}
113103

114104
return fakeFactory{name: name}, true
@@ -193,7 +183,7 @@ func TestServerStart(t *testing.T) {
193183
TracesPrimary: "jaeger_storage",
194184
},
195185
},
196-
expectedErr: "cannot find archive storage factory",
186+
expectedErr: "cannot find traces archive storage factory",
197187
},
198188
{
199189
name: "metrics storage error",
@@ -296,19 +286,32 @@ func TestServerAddArchiveStorage(t *testing.T) {
296286
qSvcOpts: &querysvc.QueryServiceOptions{},
297287
v2qSvcOpts: &v2querysvc.QueryServiceOptions{},
298288
expectedOutput: "",
299-
expectedErr: "cannot find archive storage factory: cannot find extension",
289+
expectedErr: "cannot find traces archive storage factory: cannot find extension",
300290
},
301291
{
302-
name: "Archive storage not supported",
292+
name: "Error in trace reader",
303293
config: &Config{
304294
Storage: Storage{
305-
TracesArchive: "no-archive",
295+
TracesArchive: "need-span-reader-error",
306296
},
307297
},
308298
qSvcOpts: &querysvc.QueryServiceOptions{},
309299
v2qSvcOpts: &v2querysvc.QueryServiceOptions{},
310300
extension: fakeStorageExt{},
311-
expectedOutput: "Archive storage not supported by the factory",
301+
expectedOutput: "Cannot init traces archive storage reader",
302+
expectedErr: "",
303+
},
304+
{
305+
name: "Error in trace writer",
306+
config: &Config{
307+
Storage: Storage{
308+
TracesArchive: "need-span-writer-error",
309+
},
310+
},
311+
qSvcOpts: &querysvc.QueryServiceOptions{},
312+
v2qSvcOpts: &v2querysvc.QueryServiceOptions{},
313+
extension: fakeStorageExt{},
314+
expectedOutput: "Cannot init traces archive storage writer",
312315
expectedErr: "",
313316
},
314317
{
@@ -350,6 +353,39 @@ func TestServerAddArchiveStorage(t *testing.T) {
350353
}
351354
}
352355

356+
func TestGetV1Adapters(t *testing.T) {
357+
tests := []struct {
358+
name string
359+
reader tracestore.Reader
360+
writer tracestore.Writer
361+
expectedReader spanstore.Reader
362+
expectedWriter spanstore.Writer
363+
}{
364+
{
365+
name: "native tracestore.Reader and tracestore.Writer",
366+
reader: &tracestoremocks.Reader{},
367+
writer: &tracestoremocks.Writer{},
368+
expectedReader: v1adapter.NewSpanReader(&tracestoremocks.Reader{}),
369+
expectedWriter: v1adapter.NewSpanWriter(&tracestoremocks.Writer{}),
370+
},
371+
{
372+
name: "wrapped spanstore.Reader and spanstore.Writer",
373+
reader: v1adapter.NewTraceReader(&spanstoremocks.Reader{}),
374+
writer: v1adapter.NewTraceWriter(&spanstoremocks.Writer{}),
375+
expectedReader: &spanstoremocks.Reader{},
376+
expectedWriter: &spanstoremocks.Writer{},
377+
},
378+
}
379+
380+
for _, test := range tests {
381+
t.Run(test.name, func(t *testing.T) {
382+
gotReader, gotWriter := getV1Adapters(test.reader, test.writer)
383+
require.Equal(t, test.expectedReader, gotReader)
384+
require.Equal(t, test.expectedWriter, gotWriter)
385+
})
386+
}
387+
}
388+
353389
func TestServerAddMetricsStorage(t *testing.T) {
354390
host := componenttest.NewNopHost()
355391

cmd/query/app/querysvc/query_service.go

+2-2
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ type TraceQueryParameters struct {
6161

6262
// NewQueryService returns a new QueryService.
6363
func NewQueryService(traceReader tracestore.Reader, dependencyReader depstore.Reader, options QueryServiceOptions) *QueryService {
64-
spanReader, err := v1adapter.GetV1Reader(traceReader)
65-
if err != nil {
64+
spanReader, ok := v1adapter.GetV1Reader(traceReader)
65+
if !ok {
6666
// if the spanstore.Reader is not available, downgrade the native tracestore.Reader to
6767
// a spanstore.Reader
6868
spanReader = v1adapter.NewSpanReader(traceReader)

storage_v2/v1adapter/tracereader.go

+3-5
Original file line numberDiff line numberDiff line change
@@ -18,19 +18,17 @@ import (
1818
"github.com/jaegertracing/jaeger/storage_v2/tracestore"
1919
)
2020

21-
var ErrV1ReaderNotAvailable = errors.New("spanstore.Reader is not a wrapper around v1 reader")
22-
2321
var _ tracestore.Reader = (*TraceReader)(nil)
2422

2523
type TraceReader struct {
2624
spanReader spanstore.Reader
2725
}
2826

29-
func GetV1Reader(reader tracestore.Reader) (spanstore.Reader, error) {
27+
func GetV1Reader(reader tracestore.Reader) (spanstore.Reader, bool) {
3028
if tr, ok := reader.(*TraceReader); ok {
31-
return tr.spanReader, nil
29+
return tr.spanReader, ok
3230
}
33-
return nil, ErrV1ReaderNotAvailable
31+
return nil, false
3432
}
3533

3634
func NewTraceReader(spanReader spanstore.Reader) *TraceReader {

storage_v2/v1adapter/tracereader_test.go

+4-4
Original file line numberDiff line numberDiff line change
@@ -31,15 +31,15 @@ func TestGetV1Reader_NoError(t *testing.T) {
3131
traceReader := &TraceReader{
3232
spanReader: memstore,
3333
}
34-
v1Reader, err := GetV1Reader(traceReader)
35-
require.NoError(t, err)
34+
v1Reader, ok := GetV1Reader(traceReader)
35+
require.True(t, ok)
3636
require.Equal(t, memstore, v1Reader)
3737
}
3838

3939
func TestGetV1Reader_Error(t *testing.T) {
4040
fr := new(tracestoremocks.Reader)
41-
_, err := GetV1Reader(fr)
42-
require.ErrorIs(t, err, ErrV1ReaderNotAvailable)
41+
_, ok := GetV1Reader(fr)
42+
require.False(t, ok)
4343
}
4444

4545
func TestTraceReader_GetTracesDelegatesSuccessResponse(t *testing.T) {

storage_v2/v1adapter/tracewriter.go

+3-5
Original file line numberDiff line numberDiff line change
@@ -13,17 +13,15 @@ import (
1313
"github.com/jaegertracing/jaeger/storage_v2/tracestore"
1414
)
1515

16-
var ErrV1WriterNotAvailable = errors.New("spanstore.Writer is not a wrapper around v1 writer")
17-
1816
type TraceWriter struct {
1917
spanWriter spanstore.Writer
2018
}
2119

22-
func GetV1Writer(writer tracestore.Writer) (spanstore.Writer, error) {
20+
func GetV1Writer(writer tracestore.Writer) (spanstore.Writer, bool) {
2321
if tr, ok := writer.(*TraceWriter); ok {
24-
return tr.spanWriter, nil
22+
return tr.spanWriter, ok
2523
}
26-
return nil, ErrV1WriterNotAvailable
24+
return nil, false
2725
}
2826

2927
func NewTraceWriter(spanWriter spanstore.Writer) *TraceWriter {

storage_v2/v1adapter/tracewriter_test.go

+4-4
Original file line numberDiff line numberDiff line change
@@ -62,15 +62,15 @@ func TestGetV1Writer_NoError(t *testing.T) {
6262
traceWriter := &TraceWriter{
6363
spanWriter: memstore,
6464
}
65-
v1Writer, err := GetV1Writer(traceWriter)
66-
require.NoError(t, err)
65+
v1Writer, ok := GetV1Writer(traceWriter)
66+
require.True(t, ok)
6767
require.Equal(t, memstore, v1Writer)
6868
}
6969

7070
func TestGetV1Writer_Error(t *testing.T) {
7171
w := new(tracestoremocks.Writer)
72-
_, err := GetV1Writer(w)
73-
require.ErrorIs(t, err, ErrV1WriterNotAvailable)
72+
_, ok := GetV1Writer(w)
73+
require.False(t, ok)
7474
}
7575

7676
func makeTraces() ptrace.Traces {

0 commit comments

Comments
 (0)