Operator Should Create Headless Services

[jeff-hegerhorst-deepsee]

Overview
Operator Image Version: milvusdb/milvus-operator:v0.7.13

Currently the operator doesn't create headless services for the various Milvus coordinators and nodes. This causes issues when running Milvus in an Istio service mesh with strict mtls set since the pods are trying to talk directly to each other. The Istio sidecars show the following debug logs:

2023-06-27T15:11:08.925313Z     debug   envoy filter external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:66  original_dst: set destination to 10.244.0.141:53100       thread=35
2023-06-27T15:11:08.925332Z     debug   envoy conn_handler external/envoy/source/extensions/listener_managers/listener_manager/active_stream_listener_base.cc:34  closing connection from 10.244.0.144:35662: no matching filter chain found      thread=35

Solution
Have the operator create headless services for the Milvus coordinator and node pods

[haorenfsa]

Hi @jeff-hegerhorst-deepsee, I'm not sure adding headless services alone can solve this, since all milvus pods register themselves in the etcd using its IP, and they talking to each other by the IP they acquired from etcd.

[haorenfsa]

Seems that we also need to add some options in milvus code to enable it to register using a customed endpoint (like the headless service address).

[haorenfsa]

Anyway the headless services are needed, would you like to add a patch for this?
I'll see to the Mivus' part of it.

[jeff-hegerhorst-deepsee]

@haorenfsa, I got some time to look at implementing it and I made this PR for it