Closed as not planned
Description
What are you really trying to do?
To connect a worker to a Temporal server with an authorization
header.
Describe the bug
Starting a worker with an incorrect token, causing the server to respond with Request unauthorized
, causes the worker to hang indefinitely.
Minimal Reproduction
import asyncio
from temporalio import activity, client, workflow, worker
@activity.defn
async def a() -> None:
pass
@workflow.defn
class Workflow:
@workflow.run
async def run(self) -> None:
pass
async def main():
c = await client.Client.connect(
"my_temporal_host:7233",
rpc_metadata={"authorization": "wrong_token"},
tls=True,
)
w = worker.Worker(
c,
task_queue="default",
activities=[a],
workflows=[Workflow],
)
await w.run()
if __name__ == "__main__":
asyncio.run(main())
Output:
2024-09-03T07:26:48.707Z [temporal-worker] 2024-09-03T07:26:48.707200Z ERROR temporal_client::retry: gRPC call poll_workflow_task_queue retried 52 times error=Status { code: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 504 Gateway Timeout", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:26:48 GMT", "content-type": "text/html", "content-length": "160", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:26:49.544Z [temporal-worker] 2024-09-03T07:26:49.544345Z ERROR temporal_client::retry: gRPC call poll_workflow_task_queue retried 52 times error=Status { code: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 504 Gateway Timeout", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:26:49 GMT", "content-type": "text/html", "content-length": "160", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:27:51.707Z [temporal-worker] 2024-09-03T07:27:51.707200Z ERROR temporal_client::retry: gRPC call poll_workflow_task_queue retried 53 times error=Status { code: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 504 Gateway Timeout", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:27:51 GMT", "content-type": "text/html", "content-length": "160", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:27:52.968Z [temporal-worker] 2024-09-03T07:27:52.967933Z ERROR temporal_client::retry: gRPC call poll_workflow_task_queue retried 52 times error=Status { code: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 504 Gateway Timeout", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:27:52 GMT", "content-type": "text/html", "content-length": "160", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:28:00.036Z [temporal-worker] 2024-09-03T07:28:00.036155Z ERROR temporal_client::retry: gRPC call poll_workflow_task_queue retried 53 times error=Status { code: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 504 Gateway Timeout", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:28:00 GMT", "content-type": "text/html", "content-length": "160", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:28:00.687Z [temporal-worker] 2024-09-03T07:28:00.687484Z ERROR temporal_client::retry: gRPC call poll_workflow_task_queue retried 53 times error=Status { code: Internal, message: "protocol error: received message with invalid compression flag: 60 (valid flags are 0 and 1) while receiving response with status: 504 Gateway Timeout", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:28:00 GMT", "content-type": "text/html", "content-length": "160", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:28:03.579Z [temporal-worker] 2024-09-03T07:28:03.578940Z WARN temporal_sdk_core::worker::workflow::wft_poller: Error while polling for workflow tasks error=Status { code: PermissionDenied, message: "Request unauthorized.", details: b"\x08\x07\x12\x15Request unauthorized.\x1aJ\nHtype.googleapis.com/temporal.api.errordetails.v1.PermissionDeniedFailure", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:28:03 GMT", "content-type": "application/grpc", "content-length": "0", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:28:03.970Z [temporal-worker] 2024-09-03T07:28:03.583738Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: PermissionDenied, message: "Request unauthorized.", details: b"\x08\x07\x12\x15Request unauthorized.\x1aJ\nHtype.googleapis.com/temporal.api.errordetails.v1.PermissionDeniedFailure", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:28:03 GMT", "content-type": "application/grpc", "content-length": "0", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None })
2024-09-03T07:28:03.970Z [temporal-worker] 2024-09-03T07:28:03.969933Z WARN temporal_sdk_core::worker::workflow::wft_poller: Error while polling for workflow tasks error=Status { code: PermissionDenied, message: "Request unauthorized.", details: b"\x08\x07\x12\x15Request unauthorized.\x1aJ\nHtype.googleapis.com/temporal.api.errordetails.v1.PermissionDeniedFailure", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:28:03 GMT", "content-type": "application/grpc", "content-length": "0", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:28:03.970Z [temporal-worker] 2024-09-03T07:28:03.969960Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: PermissionDenied, message: "Request unauthorized.", details: b"\x08\x07\x12\x15Request unauthorized.\x1aJ\nHtype.googleapis.com/temporal.api.errordetails.v1.PermissionDeniedFailure", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:28:03 GMT", "content-type": "application/grpc", "content-length": "0", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None })
2024-09-03T07:28:04.023Z [temporal-worker] Worker failed, shutting down
2024-09-03T07:28:04.023Z [temporal-worker] Traceback (most recent call last):
2024-09-03T07:28:04.023Z [temporal-worker] File "/lib/python3.10/site-packages/temporalio/worker/_workflow.py", line 143, in run
2024-09-03T07:28:04.023Z [temporal-worker] act = await self._bridge_worker().poll_workflow_activation()
2024-09-03T07:28:04.023Z [temporal-worker] File "/lib/python3.10/site-packages/temporalio/bridge/worker.py", line 141, in poll_workflow_activation
2024-09-03T07:28:04.023Z [temporal-worker] await self._ref.poll_workflow_activation()
2024-09-03T07:28:04.023Z [temporal-worker] RuntimeError: Poll failure: Unhandled grpc error when workflow polling: Status { code: PermissionDenied, message: "Request unauthorized.", details: b"\x08\x07\x12\x15Request unauthorized.\x1aJ\nHtype.googleapis.com/temporal.api.errordetails.v1.PermissionDeniedFailure", metadata: MetadataMap { headers: {"date": "Tue, 03 Sep 2024 07:28:03 GMT", "content-type": "application/grpc", "content-length": "0", "strict-transport-security": "max-age=15724800; includeSubDomains"} }, source: None }
2024-09-03T07:28:04.023Z [temporal-worker]
2024-09-03T07:28:04.023Z [temporal-worker] The above exception was the direct cause of the following exception:
2024-09-03T07:28:04.023Z [temporal-worker]
2024-09-03T07:28:04.023Z [temporal-worker] Traceback (most recent call last):
2024-09-03T07:28:04.023Z [temporal-worker] File "/lib/python3.10/site-packages/temporalio/worker/_workflow.py", line 153, in run
2024-09-03T07:28:04.023Z [temporal-worker] raise RuntimeError("Workflow worker failed") from err
2024-09-03T07:28:04.023Z [temporal-worker] RuntimeError: Workflow worker failed
Environment/Versions
- Temporal Version: Server v1.23.1, SDK v1.7.0
Additional context
Related issue: #459