Kuberesolver got state endpoints

[rye-sw]

Description

We noticed that Endpoints sometimes has stale data when the deployment scaled out and in frequently, but we won't have that problem if we use "service-name.namespace.svc.cluster_name:8080" instead of "kubernetes:///service-name.namespace:8080"

After digging into that, we noticed that there are difference between the Endpoints and Endpointslices of the same service, where the endpointslices contains the correct IPs but the endpoints has some stale ones. I found couple stale endpoints that belonged to pods that got killed a day ago.

Enpoints:

devbox%  kubectl get endpoints service-name -n namespace 
NAME              ENDPOINTS                                                                               AGE
service-name   10.192.1.23:8080,10.192.1.26:8080,10.192.1.27:8080 + 997 more...   27h

Endpointslices:

devbox%  kubectl get endpointslices -n namespace   | grep service-name
NAME                     ADDRESSTYPE   PORTS     ENDPOINTS                                                                     AGE
service-name-f9cm4            IPv4                      8080        10.192.118.170,10.192.1.18,10.192.17.44 + 3 more...     5h43m

As kuberesolver uses the Endpoints API to get the endpoint, it explains why we got stale endpoints when we use kuberesolver to resolve the url "kubernetes:///service-name.namespace:8080" .

kuberesolver/kubernetes.go

Lines 155 to 198 in b382846

	func getEndpoints(client K8sClient, namespace, targetName string) (Endpoints, error) {
	u, err := url.Parse(fmt.Sprintf("%s/api/v1/namespaces/%s/endpoints/%s",
	client.Host(), namespace, targetName))
	if err != nil {
	return Endpoints{}, err
	}
	req, err := client.GetRequest(u.String())
	if err != nil {
	return Endpoints{}, err
	}
	resp, err := client.Do(req)
	if err != nil {
	return Endpoints{}, err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
	return Endpoints{}, fmt.Errorf("invalid response code %d for service %s in namespace %s", resp.StatusCode, targetName, namespace)
	}
	result := Endpoints{}
	err = json.NewDecoder(resp.Body).Decode(&result)
	return result, err
	}
	func watchEndpoints(ctx context.Context, client K8sClient, namespace, targetName string) (watchInterface, error) {
	u, err := url.Parse(fmt.Sprintf("%s/api/v1/watch/namespaces/%s/endpoints/%s",
	client.Host(), namespace, targetName))
	if err != nil {
	return nil, err
	}
	req, err := client.GetRequest(u.String())
	if err != nil {
	return nil, err
	}
	req = req.WithContext(ctx)
	resp, err := client.Do(req)
	if err != nil {
	return nil, err
	}
	if resp.StatusCode != http.StatusOK {
	defer resp.Body.Close()
	return nil, fmt.Errorf("invalid response code %d for service %s in namespace %s", resp.StatusCode, targetName, namespace)
	}
	return newStreamWatcher(resp.Body), nil
	}

Version

We are using github.com/sercand/kuberesolver/v5 v5.1.1

Suggestion

I think this issue could be fixed if we could use endpointslices API instead of endpointsAPI to get the endpoint. This should be a very easy fix, I could do that if no objection.

[sercand]

@rye-sw I would be happy to accept a PR that changes the usage. The only downside would be deciding how we handle the ready and serving conditions.