feat(eap): Add a GetTraces endpoint #6671

phacops · 2024-12-13T15:51:19Z

This is a possible implementation of the new FindTraces endpoint.

getsentry/sentry-protos#71

Closes https://github.com/getsentry/eap-planning/issues/122.

requirements.txt

snuba/web/rpc/v1/endpoint_get_traces.py

kylemumma

lgtm

tests/web/rpc/v1/test_endpoint_get_traces.py

snuba/web/rpc/v1/endpoint_get_traces.py

…ilter

Zylphrex

Mostly nitpicks but looks good to me overall.

snuba/web/rpc/v1/endpoint_get_traces.py

Zylphrex · 2025-01-02T22:28:50Z

snuba/web/rpc/v1/endpoint_get_traces.py

+
+_ATTRIBUTES: dict[
+    TraceAttribute.Key.ValueType,
+    tuple[str, AttributeKey.Type.ValueType],


Nit: a named tuple/data class/typed dict would be a little cleaner here so you don't have to access it by index

snuba/web/rpc/v1/endpoint_get_traces.py

codecov · 2025-01-03T00:19:04Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2735	2	2733	5

View the top 2 failed tests by shortest run time

tests.datasets.test_errors_replacer.TestReplacer::test_process_offset_twice

Stack Traces | 0.284s run time

Traceback (most recent call last):
  File ".../tests/datasets/test_errors_replacer.py", line 370, in test_process_offset_twice
    assert self.replacer.process_message(message) is None
AssertionError: assert (ReplacementMessageMetadata(partition_index=1, offset=42, consumer_group='consumer_group'), UnmergeGroupsReplacement(state_name=<ReplacerState.ERRORS: 'errors'>, timestamp=datetime.datetime(2025, 1, 3, 0, 18, 32, 461363), hashes=['aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'], all_columns=[FlattenedColumn(None, 'project_id', schemas.UInt(64, modifiers=None)), FlattenedColumn(None, 'timestamp', schemas.DateTime(modifiers=None)), FlattenedColumn(None, 'event_id', schemas.UUID(modifiers=None)), FlattenedColumn(None, 'platform', schemas.String(modifiers=None)), FlattenedColumn(None, 'environment', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'release', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'dist', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'ip_address_v4', schemas.IPv4(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'ip_address_v6', schemas.IPv6(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'user', schemas.String(modifiers=None)), FlattenedColumn(None, 'user_id', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'user_name', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'user_email', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'sdk_name', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'sdk_version', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'http_method', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'http_referer', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn('tags', 'key', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('tags', 'value', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('contexts', 'key', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('contexts', 'value', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn(None, 'transaction_name', schemas.String(modifiers=None)), FlattenedColumn(None, 'span_id', schemas.UInt(64, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'trace_id', schemas.UUID(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'partition', schemas.UInt(16, modifiers=None)), FlattenedColumn(None, 'offset', schemas.UInt(64, modifiers=None)), FlattenedColumn(None, 'message_timestamp', schemas.DateTime(modifiers=None)), FlattenedColumn(None, 'retention_days', schemas.UInt(16, modifiers=None)), FlattenedColumn(None, 'deleted', schemas.UInt(8, modifiers=None)), FlattenedColumn(None, 'group_id', schemas.UInt(64, modifiers=None)), FlattenedColumn(None, 'primary_hash', schemas.UUID(modifiers=None)), FlattenedColumn(None, 'received', schemas.DateTime(modifiers=None)), FlattenedColumn(None, 'message', schemas.String(modifiers=None)), FlattenedColumn(None, 'title', schemas.String(modifiers=None)), FlattenedColumn(None, 'culprit', schemas.String(modifiers=None)), FlattenedColumn(None, 'level', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'location', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'version', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'type', schemas.String(modifiers=None)), FlattenedColumn('exception_stacks', 'type', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_stacks', 'value', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_stacks', 'mechanism_type', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_stacks', 'mechanism_handled', schemas.Array(schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'abs_path', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'colno', schemas.Array(schemas.UInt(32, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'filename', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'function', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'lineno', schemas.Array(schemas.UInt(32, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'in_app', schemas.Array(schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'package', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'module', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'stack_level', schemas.Array(schemas.UInt(16, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn(None, 'exception_main_thread', schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'sdk_integrations', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('modules', 'name', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('modules', 'version', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn(None, 'trace_sampled', schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'num_processing_errors', schemas.UInt(64, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'replay_id', schemas.UUID(modifiers=SchemaModifiers(nullable=True, readonly=False)))], project_id=1, previous_group_id=1, new_group_id=2)) is None
 +  where (ReplacementMessageMetadata(partition_index=1, offset=42, consumer_group='consumer_group'), UnmergeGroupsReplacement(state_name=<ReplacerState.ERRORS: 'errors'>, timestamp=datetime.datetime(2025, 1, 3, 0, 18, 32, 461363), hashes=['aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'], all_columns=[FlattenedColumn(None, 'project_id', schemas.UInt(64, modifiers=None)), FlattenedColumn(None, 'timestamp', schemas.DateTime(modifiers=None)), FlattenedColumn(None, 'event_id', schemas.UUID(modifiers=None)), FlattenedColumn(None, 'platform', schemas.String(modifiers=None)), FlattenedColumn(None, 'environment', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'release', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'dist', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'ip_address_v4', schemas.IPv4(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'ip_address_v6', schemas.IPv6(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'user', schemas.String(modifiers=None)), FlattenedColumn(None, 'user_id', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'user_name', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'user_email', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'sdk_name', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'sdk_version', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'http_method', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'http_referer', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn('tags', 'key', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('tags', 'value', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('contexts', 'key', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('contexts', 'value', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn(None, 'transaction_name', schemas.String(modifiers=None)), FlattenedColumn(None, 'span_id', schemas.UInt(64, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'trace_id', schemas.UUID(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'partition', schemas.UInt(16, modifiers=None)), FlattenedColumn(None, 'offset', schemas.UInt(64, modifiers=None)), FlattenedColumn(None, 'message_timestamp', schemas.DateTime(modifiers=None)), FlattenedColumn(None, 'retention_days', schemas.UInt(16, modifiers=None)), FlattenedColumn(None, 'deleted', schemas.UInt(8, modifiers=None)), FlattenedColumn(None, 'group_id', schemas.UInt(64, modifiers=None)), FlattenedColumn(None, 'primary_hash', schemas.UUID(modifiers=None)), FlattenedColumn(None, 'received', schemas.DateTime(modifiers=None)), FlattenedColumn(None, 'message', schemas.String(modifiers=None)), FlattenedColumn(None, 'title', schemas.String(modifiers=None)), FlattenedColumn(None, 'culprit', schemas.String(modifiers=None)), FlattenedColumn(None, 'level', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'location', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'version', schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'type', schemas.String(modifiers=None)), FlattenedColumn('exception_stacks', 'type', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_stacks', 'value', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_stacks', 'mechanism_type', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_stacks', 'mechanism_handled', schemas.Array(schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'abs_path', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'colno', schemas.Array(schemas.UInt(32, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'filename', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'function', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'lineno', schemas.Array(schemas.UInt(32, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'in_app', schemas.Array(schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'package', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'module', schemas.Array(schemas.String(modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn('exception_frames', 'stack_level', schemas.Array(schemas.UInt(16, modifiers=SchemaModifiers(nullable=True, readonly=False)), modifiers=None)), FlattenedColumn(None, 'exception_main_thread', schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'sdk_integrations', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('modules', 'name', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn('modules', 'version', schemas.Array(schemas.String(modifiers=None), modifiers=None)), FlattenedColumn(None, 'trace_sampled', schemas.UInt(8, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'num_processing_errors', schemas.UInt(64, modifiers=SchemaModifiers(nullable=True, readonly=False))), FlattenedColumn(None, 'replay_id', schemas.UUID(modifiers=SchemaModifiers(nullable=True, readonly=False)))], project_id=1, previous_group_id=1, new_group_id=2)) = <bound method ReplacerWorker.process_message of <snuba.replacer.ReplacerWorker object at 0x7f00d5918310>>(Message({Partition(topic=Topic(name='replacements'), index=1): 43}))
 +    where <bound method ReplacerWorker.process_message of <snuba.replacer.ReplacerWorker object at 0x7f00d5918310>> = <snuba.replacer.ReplacerWorker object at 0x7f00d5918310>.process_message
 +      where <snuba.replacer.ReplacerWorker object at 0x7f00d5918310> = <tests.datasets.test_errors_replacer.TestReplacer object at 0x7f0102a76310>.replacer

tests.web.rpc.v1.test_endpoint_get_traces.TestGetTraces::test_with_data

Stack Traces | 0.361s run time

Traceback (most recent call last):
  File ".../snuba/clickhouse/native.py", line 200, in execute
    result_data = query_execute()
                  ^^^^^^^^^^^^^^^
  File ".../snuba/clickhouse/native.py", line 183, in query_execute
    return conn.execute(  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../local/lib/python3.11.............../site-packages/clickhouse_driver/client.py", line 373, in execute
    rv = self.process_ordinary_query(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../local/lib/python3.11.............../site-packages/clickhouse_driver/client.py", line 571, in process_ordinary_query
    return self.receive_result(with_column_types=with_column_types,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../local/lib/python3.11.../sentry_sdk/integrations/clickhouse_driver.py", line 112, in _inner_end
    res = f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^
  File ".../local/lib/python3.11.............../site-packages/clickhouse_driver/client.py", line 204, in receive_result
    return result.get_result()
           ^^^^^^^^^^^^^^^^^^^
  File ".../local/lib/python3.11.../site-packages/clickhouse_driver/result.py", line 50, in get_result
    for packet in self.packet_generator:
  File ".../local/lib/python3.11.............../site-packages/clickhouse_driver/client.py", line 220, in packet_generator
    packet = self.receive_packet()
             ^^^^^^^^^^^^^^^^^^^^^
  File ".../local/lib/python3.11.............../site-packages/clickhouse_driver/client.py", line 237, in receive_packet
    raise packet.exception
clickhouse_driver.errors.ServerException: Code: 47.
DB::Exception: Missing columns: 'trace_start_timestamp' while processing query: 'SELECT CAST(replaceAll(toString(trace_id), '-', ''), 'String') AS trace_id FROM eap_spans_2_local PREWHERE trace_id IN ['10da294c9686470088d8306f9bc306a6', 'da7b514c6cd9410883fc6330b2e1d448', 'aae5fad5a3df4ceaa14deb9143fd8bae', '90c81bcd27c14a8394274f644662e4ac', 'd5681d054d154ad7be672fb754226737', '2c74ec073b344bdf80e42566831bef24', '71ecb331fde9450393055d8dd3e94184', 'b098005c51404f6d9f521907e0fdfe8d', '420d1cdddeb2491fb2c106c5db5dd995', 'aee050ef526440649796897a9c897f89'] WHERE (project_id IN [1, 2, 3]) AND (organization_id = 1) AND (_sort_timestamp < toDateTime(1735916340)) AND (_sort_timestamp >= toDateTime(1735901400)) GROUP BY trace_id ORDER BY trace_start_timestamp DESC', required columns: 'trace_id' 'project_id' 'organization_id' '_sort_timestamp' 'trace_start_timestamp', maybe you meant: 'trace_id', 'project_id', 'organization_id', '_sort_timestamp' or 'start_timestamp'. Stack trace:

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c61ff37 in ................................................................................................................................................/usr/bin/clickhouse
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007156ef1 in ................................................................................................................................................/usr/bin/clickhouse
2. DB::TreeRewriterResult::collectUsedColumns(std::shared_ptr<DB::IAST> const&, bool, bool) @ 0x000000001221ab99 in ................................................................................................................................................/usr/bin/clickhouse
3. DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000001221f801 in ................................................................................................................................................/usr/bin/clickhouse
4. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x0000000011ed191c in ................................................................................................................................................/usr/bin/clickhouse
5. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000011ec5975 in ................................................................................................................................................/usr/bin/clickhouse
6. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000011f74948 in ................................................................................................................................................/usr/bin/clickhouse
7. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000011e7b91e in ................................................................................................................................................/usr/bin/clickhouse
8. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122bf58a in ................................................................................................................................................/usr/bin/clickhouse
9. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122bb5b5 in ................................................................................................................................................/usr/bin/clickhouse
10. DB::TCPHandler::runImpl() @ 0x0000000013137519 in ................................................................................................................................................/usr/bin/clickhouse
11. DB::TCPHandler::run() @ 0x00000000131498f9 in ................................................................................................................................................/usr/bin/clickhouse
12. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in ................................................................................................................................................/usr/bin/clickhouse
13. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in ................................................................................................................................................/usr/bin/clickhouse
14. Poco::PooledThread::run() @ 0x0000000015c7a667 in ................................................................................................................................................/usr/bin/clickhouse
15. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in ................................................................................................................................................/usr/bin/clickhouse
16. ? @ 0x00007f6b7ee8e609 in ?
17. ? @ 0x00007f6b7edb3353 in ?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../snuba/web/db_query.py", line 466, in _raw_query
    result = execute_query_with_query_id(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../utils/metrics/util.py", line 75, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../snuba/web/db_query.py", line 243, in execute_query_with_query_id
    return execute_query_with_readthrough_caching(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../utils/metrics/util.py", line 75, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../snuba/web/db_query.py", line 340, in execute_query_with_readthrough_caching
    return cache_partition.get_readthrough(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cache/redis/backend.py", line 104, in get_readthrough
    return self.__get_value_with_simple_readthrough(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../cache/redis/backend.py", line 87, in __get_value_with_simple_readthrough
    raise e
  File ".../cache/redis/backend.py", line 76, in __get_value_with_simple_readthrough
    value = function()
            ^^^^^^^^^^
  File ".../utils/metrics/util.py", line 75, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../snuba/web/db_query.py", line 179, in execute_query
    result = reader.execute(
             ^^^^^^^^^^^^^^^
  File ".../snuba/clickhouse/native.py", line 516, in execute
    execute_func(
  File ".../snuba/clickhouse/native.py", line 285, in execute
    raise ClickhouseError(e.message, code=e.code) from e
snuba.clickhouse.errors.ClickhouseError: DB::Exception: Missing columns: 'trace_start_timestamp' while processing query: 'SELECT CAST(replaceAll(toString(trace_id), '-', ''), 'String') AS trace_id FROM eap_spans_2_local PREWHERE trace_id IN ['10da294c9686470088d8306f9bc306a6', 'da7b514c6cd9410883fc6330b2e1d448', 'aae5fad5a3df4ceaa14deb9143fd8bae', '90c81bcd27c14a8394274f644662e4ac', 'd5681d054d154ad7be672fb754226737', '2c74ec073b344bdf80e42566831bef24', '71ecb331fde9450393055d8dd3e94184', 'b098005c51404f6d9f521907e0fdfe8d', '420d1cdddeb2491fb2c106c5db5dd995', 'aee050ef526440649796897a9c897f89'] WHERE (project_id IN [1, 2, 3]) AND (organization_id = 1) AND (_sort_timestamp < toDateTime(1735916340)) AND (_sort_timestamp >= toDateTime(1735901400)) GROUP BY trace_id ORDER BY trace_start_timestamp DESC', required columns: 'trace_id' 'project_id' 'organization_id' '_sort_timestamp' 'trace_start_timestamp', maybe you meant: 'trace_id', 'project_id', 'organization_id', '_sort_timestamp' or 'start_timestamp'. Stack trace:

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c61ff37 in ................................................................................................................................................/usr/bin/clickhouse
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007156ef1 in ................................................................................................................................................/usr/bin/clickhouse
2. DB::TreeRewriterResult::collectUsedColumns(std::shared_ptr<DB::IAST> const&, bool, bool) @ 0x000000001221ab99 in ................................................................................................................................................/usr/bin/clickhouse
3. DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000001221f801 in ................................................................................................................................................/usr/bin/clickhouse
4. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x0000000011ed191c in ................................................................................................................................................/usr/bin/clickhouse
5. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000011ec5975 in ................................................................................................................................................/usr/bin/clickhouse
6. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000011f74948 in ................................................................................................................................................/usr/bin/clickhouse
7. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000011e7b91e in ................................................................................................................................................/usr/bin/clickhouse
8. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122bf58a in ................................................................................................................................................/usr/bin/clickhouse
9. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122bb5b5 in ................................................................................................................................................/usr/bin/clickhouse
10. DB::TCPHandler::runImpl() @ 0x0000000013137519 in ................................................................................................................................................/usr/bin/clickhouse
11. DB::TCPHandler::run() @ 0x00000000131498f9 in ................................................................................................................................................/usr/bin/clickhouse
12. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in ................................................................................................................................................/usr/bin/clickhouse
13. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in ................................................................................................................................................/usr/bin/clickhouse
14. Poco::PooledThread::run() @ 0x0000000015c7a667 in ................................................................................................................................................/usr/bin/clickhouse
15. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in ................................................................................................................................................/usr/bin/clickhouse
16. ? @ 0x00007f6b7ee8e609 in ?
17. ? @ 0x00007f6b7edb3353 in ?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".../rpc/v1/test_endpoint_get_traces.py", line 205, in test_with_data
    response = EndpointGetTraces().execute(message)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../web/rpc/__init__.py", line 83, in execute
    return self.__after_execute(in_msg, out, error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../web/rpc/__init__.py", line 104, in __after_execute
    raise error
  File ".../web/rpc/__init__.py", line 79, in execute
    out = self._execute(in_msg)
          ^^^^^^^^^^^^^^^^^^^^^
  File ".../rpc/v1/endpoint_get_traces.py", line 272, in _execute
    traces = self._get_metadata_for_traces(request=in_msg, trace_ids=trace_ids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../rpc/v1/endpoint_get_traces.py", line 402, in _get_metadata_for_traces
    results = run_query(
              ^^^^^^^^^^
  File ".../utils/metrics/util.py", line 75, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../snuba/web/query.py", line 106, in run_query
    raise error
  File ".../snuba/web/query.py", line 82, in run_query
    result = _run_query_pipeline(
             ^^^^^^^^^^^^^^^^^^^^
  File ".../snuba/web/query.py", line 61, in _run_query_pipeline
    raise res.error
  File ".../snuba/pipeline/query_pipeline.py", line 82, in execute
    data=self._process_data(pipe_input.as_data()),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../pipeline/stages/query_execution.py", line 86, in _process_data
    return _run_and_apply_column_names(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../pipeline/stages/query_execution.py", line 139, in _run_and_apply_column_names
    result = _format_storage_query_and_run(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../pipeline/stages/query_execution.py", line 270, in _format_storage_query_and_run
    return execute()
           ^^^^^^^^^
  File ".../pipeline/stages/query_execution.py", line 252, in execute
    return db_query(
           ^^^^^^^^^
  File ".../snuba/web/db_query.py", line 731, in db_query
    raise error or Exception(
  File ".../snuba/web/db_query.py", line 663, in db_query
    result = _raw_query(
             ^^^^^^^^^^^
  File ".../snuba/web/db_query.py", line 512, in _raw_query
    raise QueryException.from_args(
snuba.web.QueryException: DB::Exception: Missing columns: 'trace_start_timestamp' while processing query: 'SELECT CAST(replaceAll(toString(trace_id), '-', ''), 'String') AS trace_id FROM eap_spans_2_local PREWHERE trace_id IN ['10da294c9686470088d8306f9bc306a6', 'da7b514c6cd9410883fc6330b2e1d448', 'aae5fad5a3df4ceaa14deb9143fd8bae', '90c81bcd27c14a8394274f644662e4ac', 'd5681d054d154ad7be672fb754226737', '2c74ec073b344bdf80e42566831bef24', '71ecb331fde9450393055d8dd3e94184', 'b098005c51404f6d9f521907e0fdfe8d', '420d1cdddeb2491fb2c106c5db5dd995', 'aee050ef526440649796897a9c897f89'] WHERE (project_id IN [1, 2, 3]) AND (organization_id = 1) AND (_sort_timestamp < toDateTime(1735916340)) AND (_sort_timestamp >= toDateTime(1735901400)) GROUP BY trace_id ORDER BY trace_start_timestamp DESC', required columns: 'trace_id' 'project_id' 'organization_id' '_sort_timestamp' 'trace_start_timestamp', maybe you meant: 'trace_id', 'project_id', 'organization_id', '_sort_timestamp' or 'start_timestamp'. Stack trace:

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c61ff37 in ................................................................................................................................................/usr/bin/clickhouse
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007156ef1 in ................................................................................................................................................/usr/bin/clickhouse
2. DB::TreeRewriterResult::collectUsedColumns(std::shared_ptr<DB::IAST> const&, bool, bool) @ 0x000000001221ab99 in ................................................................................................................................................/usr/bin/clickhouse
3. DB::TreeRewriter::analyzeSelect(std::shared_ptr<DB::IAST>&, DB::TreeRewriterResult&&, DB::SelectQueryOptions const&, std::vector<DB::TableWithColumnNamesAndTypes, std::allocator<DB::TableWithColumnNamesAndTypes>> const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::TableJoin>) const @ 0x000000001221f801 in ................................................................................................................................................/usr/bin/clickhouse
4. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>)::$_0::operator()(bool) const @ 0x0000000011ed191c in ................................................................................................................................................/usr/bin/clickhouse
5. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context> const&, std::optional<DB::Pipe>, std::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::PreparedSets>) @ 0x0000000011ec5975 in ................................................................................................................................................/usr/bin/clickhouse
6. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x0000000011f74948 in ................................................................................................................................................/usr/bin/clickhouse
7. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x0000000011e7b91e in ................................................................................................................................................/usr/bin/clickhouse
8. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122bf58a in ................................................................................................................................................/usr/bin/clickhouse
9. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122bb5b5 in ................................................................................................................................................/usr/bin/clickhouse
10. DB::TCPHandler::runImpl() @ 0x0000000013137519 in ................................................................................................................................................/usr/bin/clickhouse
11. DB::TCPHandler::run() @ 0x00000000131498f9 in ................................................................................................................................................/usr/bin/clickhouse
12. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in ................................................................................................................................................/usr/bin/clickhouse
13. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in ................................................................................................................................................/usr/bin/clickhouse
14. Poco::PooledThread::run() @ 0x0000000015c7a667 in ................................................................................................................................................/usr/bin/clickhouse
15. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in ................................................................................................................................................/usr/bin/clickhouse
16. ? @ 0x00007f6b7ee8e609 in ?
17. ? @ 0x00007f6b7edb3353 in ?

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

feat(eap): Add a FindTraces endpoint

4c362c0

phacops requested review from a team as code owners December 13, 2024 15:51

phacops changed the title ~~feat(eap): Add a FindTraces endpoint~~ feat(eap): Add a GetTraces endpoint Dec 13, 2024

phacops added 5 commits December 13, 2024 11:31

Switch name to GetTraces

3afc793

Remove unneeded print

f68fa69

Add support for some aggregated fields

f9d043b

Add support for root span name

92ff176

Use https to fetch other repo

89d89cf

Zylphrex reviewed Dec 20, 2024

View reviewed changes

requirements.txt Outdated Show resolved Hide resolved

snuba/web/rpc/v1/endpoint_get_traces.py Outdated Show resolved Hide resolved

snuba/web/rpc/v1/endpoint_get_traces.py Outdated Show resolved Hide resolved

snuba/web/rpc/v1/endpoint_get_traces.py Outdated Show resolved Hide resolved

phacops added 4 commits December 23, 2024 14:20

Split into 2 queries to gather trace data fast

f99e2d1

Upgrade sentry-protos to v0.1.41

8f71604

Fix type issues

d4c93c2

Change to guarantee a root span name

a24d1a8

phacops requested a review from Zylphrex January 2, 2025 16:28

kylemumma approved these changes Jan 2, 2025

View reviewed changes

tests/web/rpc/v1/test_endpoint_get_traces.py Outdated Show resolved Hide resolved

snuba/web/rpc/v1/endpoint_get_traces.py Outdated Show resolved Hide resolved

snuba/web/rpc/v1/endpoint_get_traces.py Show resolved Hide resolved

phacops added 2 commits January 2, 2025 17:02

Refactor request_id into a single variable

136cc0f

Add error cases when querying with more than 1 filter or not a span f…

868c71a

…ilter

Zylphrex approved these changes Jan 2, 2025

View reviewed changes

phacops added 2 commits January 2, 2025 17:40

Use _select_supported_filters everywhere appropriate

293f71a

Order by start_timestamp only

01651ca

Request start_timestamp if needed

ff05437

onkar approved these changes Jan 3, 2025

View reviewed changes

Fix typing

4b21250

phacops merged commit dbfa45b into master Jan 3, 2025
31 checks passed

phacops deleted the pierre/eap-find-traces branch January 3, 2025 21:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(eap): Add a GetTraces endpoint #6671

feat(eap): Add a GetTraces endpoint #6671

phacops commented Dec 13, 2024 •

edited

Loading

kylemumma left a comment

Zylphrex left a comment

Zylphrex Jan 2, 2025

codecov bot commented Jan 3, 2025 •

edited

Loading

feat(eap): Add a GetTraces endpoint #6671

feat(eap): Add a GetTraces endpoint #6671

Conversation

phacops commented Dec 13, 2024 • edited Loading

kylemumma left a comment

Choose a reason for hiding this comment

Zylphrex left a comment

Choose a reason for hiding this comment

Zylphrex Jan 2, 2025

Choose a reason for hiding this comment

codecov bot commented Jan 3, 2025 • edited Loading

❌ 2 Tests Failed:

phacops commented Dec 13, 2024 •

edited

Loading

codecov bot commented Jan 3, 2025 •

edited

Loading