Enhance/Replace k8s python client.

[BalaBalaYi]

Background

Currently, DLRover uses the official Kubernetes Python client to interact with the Kubernetes API Server. This part of implementations are quite important because it involves managing the lifecycle of training workers. However, the Python client has inherent limitations and lags behind the Go (and Java) clients (e.g., lacks an informer implementation), leading to occasional unexpected usage issues in certain scenarios. Therefore, we intend to:

[Option 1] Replace the current Python client with the Go client(need to use CFFI).
[Option 2] Enhance k8s client implements in python.

Requirement

The enhancement/replacement should ensure compatibility with all existing Kubernetes-related calls while also adapting the usage in dlrover/python/scheduler/kubernetes.py.

1. Ensure compatibility with all related features (no regression).
2. Reimplement the watch mechanism using the 'informer'. (dlrover/python/master/watcher/k8s_watcher.py)

For replacement scheme requires evaluation of:

1. Limitations on different system platforms.
2. Issues with cross-language object transfer.
3. Potential performance overhead.
4. Additional costs for building and deployment.

For enhancement scheme requires evaluation of:

1. Feasibility and complexity of implementation.

[Mukku27]

hello @BalaBalaYi
Please assign me the issue
I will be enhancing the Python Kubernetes client in dlrover/python/scheduler/kubernetes.py to address #1291 by adding an informer-like watch mechanism, improved error handling with retries, and resource caching to reduce API calls. This approach will optimize performance, ensure compatibility, and avoid moving to a Go client.