Can Milvus use Ozone instead of Minio? #35976

a28124791 · 2024-09-04T09:52:56Z

a28124791
Sep 4, 2024

Can Milvus use Ozone instead of Minio?

yhmo · 2024-09-04T10:24:37Z

yhmo
Sep 4, 2024
Collaborator

The Ozone document says it is compatible with S3. Then it is possible to be used as milvus storage. Fully testing is required before you apply it to milvus.

6 replies

yhmo Sep 4, 2024
Collaborator

milvus.yaml里有关于minio的参数，实际上可以来连接s3兼容的存储系统。
关键设置address/port/accessKeyID/secretAccessKey/bucketName/rootPath 这几个。先看能否连得上，启动之后看看日志里有没有带有“chunk manager”字样的报错。如果没报错，就试着建表，插入数据，调用collection.flush()强制数据落盘，再看看Ozone里有没有数据产生。如果Ozone没数据，再去milvus的日志里找有没有“chunk manager”字样的报错。

minio:
  # IP address of MinIO or S3 service.
  # Environment variable: MINIO_ADDRESS
  # minio.address and minio.port together generate the valid access to MinIO or S3 service.
  # MinIO preferentially acquires the valid IP address from the environment variable MINIO_ADDRESS when Milvus is started.
  # Default value applies when MinIO or S3 is running on the same network with Milvus.
  address: localhost
  port: 9000 # Port of MinIO or S3 service.
  # Access key ID that MinIO or S3 issues to user for authorized access.
  # Environment variable: MINIO_ACCESS_KEY_ID or minio.accessKeyID
  # minio.accessKeyID and minio.secretAccessKey together are used for identity authentication to access the MinIO or S3 service.
  # This configuration must be set identical to the environment variable MINIO_ACCESS_KEY_ID, which is necessary for starting MinIO or S3.
  # The default value applies to MinIO or S3 service that started with the default docker-compose.yml file.
  accessKeyID: minioadmin
  # Secret key used to encrypt the signature string and verify the signature string on server. It must be kept strictly confidential and accessible only to the MinIO or S3 server and users.
  # Environment variable: MINIO_SECRET_ACCESS_KEY or minio.secretAccessKey
  # minio.accessKeyID and minio.secretAccessKey together are used for identity authentication to access the MinIO or S3 service.
  # This configuration must be set identical to the environment variable MINIO_SECRET_ACCESS_KEY, which is necessary for starting MinIO or S3.
  # The default value applies to MinIO or S3 service that started with the default docker-compose.yml file.
  secretAccessKey: minioadmin
  useSSL: false # Switch value to control if to access the MinIO or S3 service through SSL.
  ssl:
    tlsCACert: /path/to/public.crt # path to your CACert file
  # Name of the bucket where Milvus stores data in MinIO or S3.
  # Milvus 2.0.0 does not support storing data in multiple buckets.
  # Bucket with this name will be created if it does not exist. If the bucket already exists and is accessible, it will be used directly. Otherwise, there will be an error.
  # To share an MinIO instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs.
  # The data will be stored in the local Docker if Docker is used to start the MinIO service locally. Ensure that there is sufficient storage space.
  # A bucket name is globally unique in one MinIO or S3 instance.
  bucketName: a-bucket
  # Root prefix of the key to where Milvus stores data in MinIO or S3.
  # It is recommended to change this parameter before starting Milvus for the first time.
  # To share an MinIO instance among multiple Milvus instances, consider changing this to a different value for each Milvus instance before you start them. For details, see Operation FAQs.
  # Set an easy-to-identify root key prefix for Milvus if etcd service already exists.
  # Changing this for an already running Milvus instance may result in failures to read legacy data.
  rootPath: files
  # Whether to useIAM role to access S3/GCS instead of access/secret keys
  # For more information, refer to
  # aws: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html
  # gcp: https://cloud.google.com/storage/docs/access-control/iam
  # aliyun (ack): https://www.alibabacloud.com/help/en/container-service-for-kubernetes/latest/use-rrsa-to-enforce-access-control
  # aliyun (ecs): https://www.alibabacloud.com/help/en/elastic-compute-service/latest/attach-an-instance-ram-role
  useIAM: false
  # Cloud Provider of S3. Supports: "aws", "gcp", "aliyun".
  # You can use "aws" for other cloud provider supports S3 API with signature v4, e.g.: minio
  # You can use "gcp" for other cloud provider supports S3 API with signature v2
  # You can use "aliyun" for other cloud provider uses virtual host style bucket
  # When useIAM enabled, only "aws", "gcp", "aliyun" is supported for now
  cloudProvider: aws
  # Custom endpoint for fetch IAM role credentials. when useIAM is true & cloudProvider is "aws".
  # Leave it empty if you want to use AWS default endpoint
  iamEndpoint: 
  logLevel: fatal # Log level for aws sdk log. Supported level:  off, fatal, error, warn, info, debug, trace
  region:  # Specify minio storage system location region
  useVirtualHost: false # Whether use virtual host mode for bucket
  requestTimeoutMs: 10000 # minio timeout for request time in milliseconds
  # The maximum number of objects requested per batch in minio ListObjects rpc, 
  # 0 means using oss client by default, decrease these configration if ListObjects timeout
  listObjectsMaxKeys: 0

a28124791 Oct 9, 2024
Author

milvus的日志里没有找到“chunk manager”字样的报错关键字，用aws命令可以连ozone的s3接口
indexnode提示这样的日志:
[2024/10/09 07:02:51.970 +00:00] [DEBUG] [minio_kv.go:73] ["MinioKV NewMinioKV"] ["Check bucket"="bucket not exist"]
[2024/10/09 07:02:51.970 +00:00] [DEBUG] [minio_kv.go:75] ["MinioKV NewMinioKV create bucket."]
[2024/10/09 07:02:52.182 +00:00] [DEBUG] [minio_kv.go:73] ["MinioKV NewMinioKV"] ["Check bucket"="bucket not exist"]
[2024/10/09 07:02:52.182 +00:00] [DEBUG] [minio_kv.go:75] ["MinioKV NewMinioKV create bucket."]

yhmo Oct 10, 2024
Collaborator

你的milvus是什么版本？

a28124791 Oct 10, 2024
Author

docker pull milvusdb/milvus:v2.4.5
---使用的这个版本

a28124791 Oct 10, 2024
Author

您看看我的参数对吗？其中参数一在Milvus文档（https://milvus.io/docs/zh/configure_minio.md）里面找的
参数一：

indexnode

docker run -d -it
--name milvus_indexnode-up2
-e "ETCD_ENDPOINTS=192.168.95.200:12379"
-e "ETCD_ROOTPATH=gbotby"
-e "ETCD_METASUBPATH=meta"
-e "ETCD_KVSUBPATH=kv"
-e "MINIO_ADDRESS=192.168.95.200"
-e "MINIO_PORT=9878"
-e "MINIO_ACCESS_KEY_ID=werwerwerwerwrwrw"
-e "MINIO_SECRET_ACCESS_KEY=ererererererererr"
-e "PULSAR_ADDRESS=pulsar://192.168.95.200:6650"
-e "INDEX_COORD_ADDRESS=192.168.95.200:31000"
milvusdb/milvus milvus run indexnode

参数二：
docker run -d -it
--name milvus_datacoord-up1
-p 13333:13333
-e "ETCD_ENDPOINTS=192.168.95.200:12379"
-e "ETCD_ROOTPATH=gbotby"
-e "ETCD_METASUBPATH=meta"
-e "ETCD_KVSUBPATH=kv"
-e "AWS_ACCESS_KEY_ID=werwerwerwerwrwrw"
-e "AWS_SECRET_ACCESS_KEY=ererererererererr"
-e "S3_BUCKET_NAME=wordcount"
-e "S3_ENDPOINT_URL=http://192.168.95.200:9878"
-e "PULSAR_ADDRESS=pulsar://192.168.95.200:6650"
-e "DATA_COORD_ADDRESS=192.168.95.200:13333"
-e "DATA_COORD_ENABLE_ACTIVE_STANDBY=true"
milvusdb/milvus milvus run datacoord

a28124791 · 2024-10-09T10:00:29Z

a28124791
Oct 9, 2024
Author

-e "MINIO_ADDRESS=192.168.95.200"
-e "MINIO_PORT=9878"
-e "MINIO_ACCESS_KEY_ID=werwerwerwerwrwrw"
-e "MINIO_SECRET_ACCESS_KEY=ererererererererr" \

和
-e "AWS_ACCESS_KEY_ID=werwerwerwerwrwrw"
-e "AWS_SECRET_ACCESS_KEY=ererererererererr"
-e "S3_BUCKET_NAME=wordcount"
-e "S3_ENDPOINT_URL=http://192.168.95.200:9878" \

都不行。

0 replies

xiaofan-luan · 2024-10-09T12:48:34Z

xiaofan-luan
Oct 9, 2024
Maintainer

你有日志么？需要看一下报的什么异常信息

10 replies

a28124791 Oct 10, 2024
Author

我没有修改过Milvus源码，我用的docker镜像就是 docker pull milvusdb/milvus:v2.4.5

a28124791 Oct 10, 2024
Author

Milvus-config.zip
这个是配置文件。我找打了是这样的参数项： rootCoordDml: "rootcoord-dml"

a28124791 Oct 10, 2024
Author

rootCoordDml: 这个配置项要改成dmlChannelNum: 256 ？

yhmo Oct 10, 2024
Collaborator

不改，默认应该是16才对，但为何你日志里产生了256个，不太对

a28124791 Oct 10, 2024
Author

我的启动参数对吗？看日志Milvus连上s3了么？

a28124791 · 2024-10-10T07:20:41Z

a28124791
Oct 10, 2024
Author

Milvus-config.zip
这个是配置文件。我找打了是这样的参数项： rootCoordDml: "rootcoord-dml"

我没有修改过Milvus源码，我用的docker镜像就是 docker pull milvusdb/milvus:v2.4.5

0 replies

a28124791 · 2024-10-11T02:43:34Z

a28124791
Oct 11, 2024
Author

我的启动参数对吗？看日志Milvus连上s3了么？
我这个下一步需要做什么尝试，可以调试通吗？
Are my startup parameters correct? Did Milvus connect to S3 according to the log?
What do I need to try next and can I debug it successfully?

4 replies

yhmo Oct 11, 2024
Collaborator

除了datacoord跟pulsar通讯有问题，以及256个dml channel不符合预期之外，我没看到其他的问题。
[2024/10/09 07:15:01.558 +00:00] [WARN] [util.go:101] ["we haven't received tt for 3.000000 minutes"]
不过这个datacoord接收pulsar的报错估计服务就没法用了。不知什么原因接收不到。

这种日志是info，应该不会造成问题，s3看起来是连通的。
[2024/10/09 07:02:51.970 +00:00] [DEBUG] [minio_kv.go:73] ["MinioKV NewMinioKV"] ["Check bucket"="bucket not exist"]
[2024/10/09 07:02:51.970 +00:00] [DEBUG] [minio_kv.go:75] ["MinioKV NewMinioKV create bucket."]

a28124791 Oct 12, 2024
Author

我重新安装了pulsar，并用下面的命令测试了消费组和生产组测试并成功，重新启动了所有的Milvus服务，dotacoord任务还会有 ["we haven't received tt for 3.000000 minutes"]，同时attu无法确实登录Milvus，你在看看帮忙看看pulsar的测试和datacoord的日志信息。
pulsar消费者和生产者测试.txt
datacoord20241012.log

yhmo Oct 14, 2024
Collaborator

你的milvus似乎版本很老啊，看这个Built日期是2021年的。太陈旧的版本很难查问题原因的。

   /  |/  /  _/ /| | / / / / / __/    
  / /|_/ // // /_| |/ / /_/ /\ \    
 /_/  /_/___/____/___/\____/___/     

Welcome to use Milvus!
Version:   d1f4106
Built:     Thu Nov  4 13:15:21 UTC 2021

a28124791 Oct 22, 2024
Author

sorry，发现之前的确实是老的，用2.4.11重新启动了，也没有之前的pulsar报错了，下面是完整的最新的日志，帮忙看看，谢谢
mllvus-2.4.11-logs.tgz

xiaofan-luan · 2024-10-12T07:11:00Z

xiaofan-luan
Oct 12, 2024
Maintainer

pulsar:
address: localhost
port: 6650
maxMessageSize: 5242880 # 5 * 1024 * 1024 Bytes, Maximum size of each message in pulsar.

你pulsar的host是localost么？

2 replies

a28124791 Oct 12, 2024
Author

不是localhost吧，你看下面的端口监听，
(base) [root@localhost ~]# netstat -antulp |grep 6650
tcp 0 0 0.0.0.0:6650 0.0.0.0:* LISTEN 28140/docker-proxy
tcp 0 0 192.168.95.200:6650 172.17.0.3:59348 ESTABLISHED 28140/docker-proxy
tcp 0 0 192.168.95.200:6650 172.17.0.10:40798 ESTABLISHED 28140/docker-proxy
tcp 0 0 172.17.0.1:38030 172.17.0.2:6650 ESTABLISHED 28140/docker-proxy
tcp 0 0 192.168.95.200:6650 172.17.0.10:40802 ESTABLISHED 28140/docker-proxy
tcp 0 0 172.17.0.1:18832 172.17.0.2:6650 ESTABLISHED 28140/docker-proxy
tcp 0 0 172.17.0.1:38026 172.17.0.2:6650 ESTABLISHED 28140/docker-proxy
tcp 0 0 192.168.95.200:6650 172.17.0.3:59352 ESTABLISHED 28140/docker-proxy
tcp 0 0 172.17.0.1:18836 172.17.0.2:6650 ESTABLISHED 28140/docker-proxy
tcp6 0 0 :::6650 :::* LISTEN 28147/docker-proxy
另外Ozone，etcd，Milvus，pulsar都在一台机器上

xiaofan-luan Oct 12, 2024
Maintainer

配置上，你的pulsar配置的是localhost

a28124791 · 2024-10-22T08:57:03Z

a28124791
Oct 22, 2024
Author

发现之前的确实是老的，用2.4.11重新启动了，也没有之前的pulsar报错了，下面是完整的最新的日志，帮忙看看，谢谢
mllvus-2.4.11-logs.tgz

0 replies

Reidddddd · 2024-10-29T06:48:11Z

Reidddddd
Oct 29, 2024

The answer is yes, we are using ozone which needs provide s3 gateway

1 reply

a28124791 Oct 29, 2024
Author

Yeah, I connected to ozone, too.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can Milvus use Ozone instead of Minio? #35976

{{title}}

Replies: 8 comments 23 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Can Milvus use Ozone instead of Minio? #35976

a28124791 Sep 4, 2024

Replies: 8 comments · 23 replies

yhmo Sep 4, 2024 Collaborator

yhmo Sep 4, 2024 Collaborator

a28124791 Oct 9, 2024 Author

yhmo Oct 10, 2024 Collaborator

a28124791 Oct 10, 2024 Author

a28124791 Oct 10, 2024 Author

indexnode

a28124791 Oct 9, 2024 Author

xiaofan-luan Oct 9, 2024 Maintainer

a28124791 Oct 10, 2024 Author

a28124791 Oct 10, 2024 Author

a28124791 Oct 10, 2024 Author

yhmo Oct 10, 2024 Collaborator

a28124791 Oct 10, 2024 Author

a28124791 Oct 10, 2024 Author

Milvus-config.zip 这个是配置文件。我找打了是这样的参数项 ： rootCoordDml: "rootcoord-dml"

a28124791 Oct 11, 2024 Author

yhmo Oct 11, 2024 Collaborator

a28124791 Oct 12, 2024 Author

yhmo Oct 14, 2024 Collaborator

a28124791 Oct 22, 2024 Author

xiaofan-luan Oct 12, 2024 Maintainer

a28124791 Oct 12, 2024 Author

xiaofan-luan Oct 12, 2024 Maintainer

a28124791 Oct 22, 2024 Author

Reidddddd Oct 29, 2024

a28124791 Oct 29, 2024 Author

a28124791
Sep 4, 2024

Replies: 8 comments 23 replies

yhmo
Sep 4, 2024
Collaborator

yhmo Sep 4, 2024
Collaborator

a28124791 Oct 9, 2024
Author

yhmo Oct 10, 2024
Collaborator

a28124791 Oct 10, 2024
Author

a28124791 Oct 10, 2024
Author

a28124791
Oct 9, 2024
Author

xiaofan-luan
Oct 9, 2024
Maintainer

a28124791 Oct 10, 2024
Author

a28124791 Oct 10, 2024
Author

a28124791 Oct 10, 2024
Author

yhmo Oct 10, 2024
Collaborator

a28124791 Oct 10, 2024
Author

a28124791
Oct 10, 2024
Author

Milvus-config.zip
这个是配置文件。我找打了是这样的参数项： rootCoordDml: "rootcoord-dml"

a28124791
Oct 11, 2024
Author

yhmo Oct 11, 2024
Collaborator

a28124791 Oct 12, 2024
Author

yhmo Oct 14, 2024
Collaborator

a28124791 Oct 22, 2024
Author

xiaofan-luan
Oct 12, 2024
Maintainer

a28124791 Oct 12, 2024
Author

xiaofan-luan Oct 12, 2024
Maintainer

a28124791
Oct 22, 2024
Author

Reidddddd
Oct 29, 2024

a28124791 Oct 29, 2024
Author