increased memory usage after grpc ResourceExhausted error occurred

[dwkang]

(pprof) top 50
Showing nodes accounting for 804.67MB, 97.75% of 823.21MB total
Dropped 197 nodes (cum <= 4.12MB)
Showing top 50 nodes out of 51
      flat  flat%   sum%        cum   cum%
  368.97MB 44.82% 44.82%   368.97MB 44.82%  google.golang.org/grpc/mem.(*simpleBufferPool).Get
  347.47MB 42.21% 87.03%   347.47MB 42.21%  google.golang.org/grpc/internal/transport.newBufWriter (inline)
      32MB  3.89% 90.92%       32MB  3.89%  github.com/dgraph-io/ristretto.newCmRow (inline)
      16MB  1.94% 92.86%       16MB  1.94%  github.com/dgraph-io/ristretto/z.(*Bloom).Size (inline)
   13.93MB  1.69% 94.55%    13.93MB  1.69%  bufio.NewReaderSize (inline)
   10.12MB  1.23% 95.78%    10.12MB  1.23%  golang.org/x/net/webdav.(*memFile).Write
    9.67MB  1.17% 96.96%     9.67MB  1.17%  golang.org/x/net/http2.(*Framer).startWriteDataPadded
    6.01MB  0.73% 97.69%     6.01MB  0.73%  bufio.NewWriterSize (inline)
    0.50MB 0.061% 97.75%   361.90MB 43.96%  google.golang.org/grpc/internal/transport.newHTTP2Client
         0     0% 97.75%    48.50MB  5.89%  github.com/dgraph-io/ristretto.NewCache
         0     0% 97.75%       32MB  3.89%  github.com/dgraph-io/ristretto.newCmSketch
         0     0% 97.75%       48MB  5.83%  github.com/dgraph-io/ristretto.newDefaultPolicy
         0     0% 97.75%       48MB  5.83%  github.com/dgraph-io/ristretto.newPolicy (inline)
         0     0% 97.75%       48MB  5.83%  github.com/dgraph-io/ristretto.newTinyLFU
         0     0% 97.75%       16MB  1.94%  github.com/dgraph-io/ristretto/z.NewBloomFilter
         0     0% 97.75%   371.97MB 45.19%  github.com/pinpoint-apm/pinpoint-go-agent.(*agent).sendMetaWorker
         0     0% 97.75%   369.97MB 44.94%  github.com/pinpoint-apm/pinpoint-go-agent.(*agentGrpc).sendExceptionMetadata
         0     0% 97.75%   371.97MB 45.19%  github.com/pinpoint-apm/pinpoint-go-agent.(*agentGrpc).sendExceptionMetadataWithRetry
         0     0% 97.75%   369.97MB 44.94%  github.com/pinpoint-apm/pinpoint-go-agent.(*metaGrpcClient).RequestExceptionMetaData
         0     0% 97.75%   369.97MB 44.94%  github.com/pinpoint-apm/pinpoint-go-agent/protobuf.(*metadataClient).RequestExceptionMetaData
         0     0% 97.75%     6.01MB  0.73%  go.uber.org/zap.(*Logger).Info
         0     0% 97.75%     6.01MB  0.73%  go.uber.org/zap/zapcore.(*BufferedWriteSyncer).Write
         0     0% 97.75%     6.01MB  0.73%  go.uber.org/zap/zapcore.(*BufferedWriteSyncer).initialize
         0     0% 97.75%     6.01MB  0.73%  go.uber.org/zap/zapcore.(*CheckedEntry).Write
         0     0% 97.75%     6.01MB  0.73%  go.uber.org/zap/zapcore.(*ioCore).Write
         0     0% 97.75%     9.67MB  1.17%  golang.org/x/net/http2.(*Framer).WriteData (inline)
         0     0% 97.75%     9.67MB  1.17%  golang.org/x/net/http2.(*Framer).WriteDataPadded
         0     0% 97.75%   370.47MB 45.00%  google.golang.org/grpc.(*ClientConn).Invoke
         0     0% 97.75%   361.90MB 43.96%  google.golang.org/grpc.(*addrConn).connect
         0     0% 97.75%   361.90MB 43.96%  google.golang.org/grpc.(*addrConn).createTransport
         0     0% 97.75%   361.90MB 43.96%  google.golang.org/grpc.(*addrConn).resetTransportAndUnlock
         0     0% 97.75%   361.90MB 43.96%  google.golang.org/grpc.(*addrConn).tryAllAddrs
         0     0% 97.75%   370.97MB 45.06%  google.golang.org/grpc.(*clientStream).SendMsg
         0     0% 97.75%   370.97MB 45.06%  google.golang.org/grpc.encode
         0     0% 97.75%   370.47MB 45.00%  google.golang.org/grpc.invoke
         0     0% 97.75%   370.97MB 45.06%  google.golang.org/grpc.prepareMsg
         0     0% 97.75%   370.97MB 45.06%  google.golang.org/grpc/encoding/proto.(*codecV2).Marshal
         0     0% 97.75%     9.67MB  1.17%  google.golang.org/grpc/internal/transport.(*loopyWriter).processData
         0     0% 97.75%    12.18MB  1.48%  google.golang.org/grpc/internal/transport.(*loopyWriter).run
         0     0% 97.75%   361.90MB 43.96%  google.golang.org/grpc/internal/transport.NewClientTransport (inline)
         0     0% 97.75%   361.40MB 43.90%  google.golang.org/grpc/internal/transport.newFramer
         0     0% 97.75%    12.18MB  1.48%  google.golang.org/grpc/internal/transport.newHTTP2Client.func6
         0     0% 97.75%   368.97MB 44.82%  google.golang.org/grpc/mem.(*tieredBufferPool).Get
         0     0% 97.75%    52.51MB  6.38%  main.main