The error of forwarding TSO should be handled

[rleungx]

Bug Report

pd/server/grpc_service.go

Lines 2626 to 2653 in ab8bf7b

	for i := 0; i < maxRetryTimesRequestTSOServer; i++ {
	forwardedHost, ok := s.GetServicePrimaryAddr(ctx, utils.TSOServiceName)
	if !ok || forwardedHost == "" {
	return pdpb.Timestamp{}, ErrNotFoundTSOAddr
	}
	forwardStream, err = s.getTSOForwardStream(forwardedHost)
	if err != nil {
	return pdpb.Timestamp{}, err
	}
	forwardStream.Send(request)
	ts, err = forwardStream.Recv()
	if err != nil {
	if strings.Contains(err.Error(), errs.NotLeaderErr) {
	s.tsoPrimaryWatcher.ForceLoad()
	time.Sleep(retryIntervalRequestTSOServer)
	continue
	}
	if strings.Contains(err.Error(), codes.Unavailable.String()) {
	s.tsoClientPool.Lock()
	delete(s.tsoClientPool.clients, forwardedHost)
	s.tsoClientPool.Unlock()
	continue
	}
	log.Error("get global tso from tso service primary addr failed", zap.Error(err), zap.String("tso-addr", forwardedHost))
	return pdpb.Timestamp{}, err
	}
	return *ts.GetTimestamp(), nil
	}

pd/server/grpc_service.go

Line 2635 in ab8bf7b

forwardStream.Send(request)

The error of sending the request is not properly handled. We should do the error handling in case there is something wrong when forwarding the request to get TSO.

[rleungx]

/cc @lhy1024