S3-compatible storage cannot be used on Teleport.

[ZeroClover]

I tried to configure S3 API-compatible storage for Teleport, but it doesn't work.

I can see in S3 file manager that Teleport has created the directory and uploaded the file in .tar format, but Teleport throws errors in the log.

Apr 16 19:06:49 hostname teleport[19753]: INFO [S3]        Upload created in 88.677244ms. s3sessions/s3stream.go:42
Apr 16 19:06:49 hostname teleport[19753]: INFO [S3]        UploadPart(A54DC4C8CD75435299374EAECAD8AEA1) part(1) uploaded in 13.520818ms. s3sessions/s3stream.go:63
Apr 16 19:06:49 hostname teleport[19753]: INFO [S3]        UploadPart(A54DC4C8CD75435299374EAECAD8AEA1) completed in 20.614992ms. s3sessions/s3stream.go:89
Apr 16 19:06:49 hostname teleport[19753]: INFO [AUDIT]     session.upload cluster_name:Teleport-HKG-Master code:T2005I ei:2.147483647e+09 event:session.upload sid:4366a95d-973c-4356-a393-3c5db0506ab9 time:2021-04-16T11:06:49.424Z url:s3://zerotp/4366a95d-973c-4356-a393-
Apr 16 19:06:49 hostname teleport[19753]: WARN [AUTH:GRPC] Failed to flush close the stream. error:[
Apr 16 19:06:49 hostname teleport[19753]: ERROR REPORT:
Apr 16 19:06:49 hostname teleport[19753]: Original Error: *trace.ConnectionProblemError emitter has been closed
Apr 16 19:06:49 hostname teleport[19753]: Stack Trace:
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/events/stream.go:422 github.com/gravitational/teleport/lib/events.(*ProtoStream).Close
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/events/emitter.go:453 github.com/gravitational/teleport/lib/events.(*CheckingStream).Close
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/auth_with_roles.go:1913 github.com/gravitational/teleport/lib/auth.(*streamWithRoles).Close
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/grpcserver.go:139 github.com/gravitational/teleport/lib/auth.(*GRPCServer).CreateAuditStream.func2
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/grpcserver.go:214 github.com/gravitational/teleport/lib/auth.(*GRPCServer).CreateAuditStream
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/teleport/api/client/proto/authservice.pb.go:9114 github.com/gravitational/teleport/api/client/proto._AuthService_CreateAuditStream_Handler
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/middleware.go:386 github.com/gravitational/teleport/lib/auth.(*Middleware).StreamInterceptor
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/google.golang.org/grpc/server.go:1336 google.golang.org/grpc.(*Server).processStreamingRPC
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/google.golang.org/grpc/server.go:1409 google.golang.org/grpc.(*Server).handleStream
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/google.golang.org/grpc/server.go:746 google.golang.org/grpc.(*Server).serveStreams.func1.1
Apr 16 19:06:49 hostname teleport[19753]:         /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
Apr 16 19:06:49 hostname teleport[19753]: User Message: emitter has been closed] auth/grpcserver.go:140

At the same time, I was unable to play back Session recordings in Teleport WebGUI, prompting "Recording for this session is not available."

I opened the browser console and found that when playing back the session, there was a 404 that went to https://<teleport url>/v1/webapi/sites/<cluster name>/sessions/4366a95d-973c-4356- a393-3c5db0506ab9/events .

[benarent]

Which S3-Compatible storage provider/project are you using?

[ZeroClover]

I am using AlibabaCloud OSS.

According to their documentation, they provide interoperability with the AWS S3 API.

https://www.alibabacloud.com/help/doc-detail/64919.html

[webvictim]

@ZeroClover What version of Teleport are you running?

[ZeroClover]

Sorry for the delay in replying, but I did not receive an email notification of this discussion.

I always update to the latest Teleport version, for now, this is version 6.2.0.

[webvictim]

Just to confirm, this is still broken on 6.2.0?

[ZeroClover]

I just tested it again on 6.2.0 and it still has the same problem.

The Teleport log shows that the Session has been uploaded and that the Teleport uploaded files can be viewed in Ali OSS, but Teleport cannot playback the Session.

[webvictim]

Issue #6726 is tracking this.

[oanqa]

Hi @ZeroClover have you resolve this issue? i also having same issue, using teleport web and tsh recordings export {session} result to session not found