This repository has been archived by the owner on Nov 13, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* milvus2x->2x * support mutil type vertor field & fix issue * milvus2x migration support no field param default all fields * connect milvus add timeout * milvus2x->2x readme * upgrade milvus-go-sdk
- Loading branch information
1 parent
acd5ee2
commit 421edfe
Showing
39 changed files
with
1,408 additions
and
209 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Milvus Migration: Milvus2.x to Milvus2.x | ||
|
|
||
| ## Limitation | ||
|
|
||
| ### Soft version | ||
|
|
||
| - Source Milvus support version: 2.3.0+ | ||
| - Target Milvus support version: 2.2+ | ||
|
|
||
|
|
||
| ## Milvus2.x to Milvus2.x migration.yaml example | ||
|
|
||
| ```yaml | ||
| dumper: | ||
| worker: | ||
| workMode: milvus2x # work mode：milvus2x->milvus2x | ||
| reader: | ||
| bufferSize: 500 # Read source data rows in each time read from Source Milvus. | ||
|
|
||
| meta: # meta part | ||
| mode: config # 'config' mode means will get meta config from this config file itself. | ||
| version: 2.3.0 # Source Milvus version | ||
| collection: src_coll_name # migrate data from this source collection | ||
|
|
||
| source: # source milvus connection info | ||
| milvus2x: | ||
| endpoint: {milvus2x_domain}:{milvus2x_port} | ||
| username: xxxx | ||
| password: xxxxx | ||
|
|
||
| target: # target milvus collection info | ||
| milvus2x: | ||
| endpoint: {milvus2x_domain}:{milvus2x_port} | ||
| username: xxxx | ||
| password: xxxxx | ||
| ``` | ||
| you can place the migration.yaml to configs/ directory, then tool will auto read config from the configs/migration.yaml | ||
| when you execute cmd: | ||
| ```shell | ||
| ./milvus-migration start | ||
| ``` | ||
|
|
||
| or you can place the migration.yaml to any directory, then will read config from `--config` param path when execute cmd | ||
| like below: | ||
|
|
||
| ```shell | ||
| ./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml | ||
| ``` | ||
| migration success when you see the print log like below: | ||
| ```html | ||
| [INFO] [migration/milvus2x_starter.go:79] ["=================>JobProcess!"] [Percent=100] | ||
| [INFO] [migration/milvus2x_starter.go:27] ["[Starter] migration Milvus2X to Milvus2X finish!!!"] [Cost=94.877717375] | ||
| [INFO] [starter/starter.go:109] ["[Starter] Migration Success!"] [Cost=94.878243583] | ||
| [INFO] [cleaner/none_cleaner.go:17] ["[None Cleaner] not need clean files"] [mode=] | ||
| [INFO] [cmd/start.go:32] ["[Cleaner] clean file success!"] | ||
| ``` | ||
| if you want to verify the migration data result, you can use Attu see source collection already in your target Milvus. [Attu](https://github.com/zilliztech/attu) | ||
|
|
||
| ## Other introduce | ||
| - if you don't migration all the source collection fields to the target Milvus, you can add fields config in meta part to specify the need migration fields. | ||
| - btw, you at least need migration the PrimaryKey and Vector type field to target Milvus. | ||
| ```yaml | ||
| ... | ||
| meta: | ||
| #...... | ||
| fields: # optional configuration, only migration below source collection fields to target milvus: | ||
| - name: id | ||
| - name: title_vector | ||
| - name: reading_time | ||
| #...... | ||
| ... | ||
| ``` | ||
| - if you want to customize target collection properties, you can add below config in your meta part | ||
| ```yaml | ||
| ... | ||
| meta: | ||
| #...... | ||
| milvus: #below info are target collection optional configuration: | ||
| collection: target_coll_name # If not, the source collection name will be used. | ||
| closeDynamicField: false # If not, the source collection DynamicField prop will be used. | ||
| shardNum: 2 # If not, the source collection ShardNum prop will be used. | ||
| consistencyLevel: Customized # If not, the source collection consistencyLevel prop will be used. | ||
| #...... | ||
| ... | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| package check | ||
|
|
||
| import ( | ||
| "errors" | ||
| "github.com/zilliztech/milvus-migration/core/common" | ||
| "strconv" | ||
| "strings" | ||
| ) | ||
|
|
||
| func verifyCollNameIsOk(collection string) bool { | ||
| if strings.Contains(ArabicNumer, collection[:1]) { | ||
| return false | ||
| } | ||
| for i, _ := range collection { | ||
| s := collection[i : i+1] | ||
| if !strings.Contains(LowerAlphabet, s) && !strings.Contains(UpperAlphabet, s) && | ||
| Underline != s && !strings.Contains(ArabicNumer, s) { | ||
| return false | ||
| } | ||
| } | ||
| return true | ||
| } | ||
|
|
||
| func verifyShardNum(shardNum int) error { | ||
| if shardNum > common.MAX_SHARD_NUM { | ||
| return errors.New("[Verify Meta file] Milvus shardNum can not > " + strconv.Itoa(common.MAX_SHARD_NUM)) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| func verifyMilvusCollName(collectionName string) error { | ||
| if !verifyCollNameIsOk(collectionName) { | ||
| return errors.New("[Verify Meta file] collection name not match [A-Z|a-z|0-9|_] format cannot as Milvus collection name, " + | ||
| "you can set meta.milvus.collection property to replace it, invalid collection name is：" + collectionName) | ||
| } | ||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| package check | ||
|
|
||
| import ( | ||
| "errors" | ||
| convert "github.com/zilliztech/milvus-migration/core/transform/common" | ||
| "github.com/zilliztech/milvus-migration/core/type/milvus2xtype" | ||
| "github.com/zilliztech/milvus-migration/core/type/milvustype" | ||
| ) | ||
|
|
||
| func VerifyMilvus2xMetaCfg(metaJson *milvus2xtype.MetaJSON) error { | ||
|
|
||
| for _, coll := range metaJson.CollCfgs { | ||
|
|
||
| //if len(coll.Fields) <= 0 { | ||
| // return errors.New("[Verify Milvus2x Meta file] Index migration Field is empty, Collection:" + coll.Collection) | ||
| //} | ||
|
|
||
| if coll.MilvusCfg == nil { | ||
| coll.MilvusCfg = &milvustype.MilvusCfg{ShardNum: 0} //当没定义时，会用source collection shardNum | ||
| } | ||
|
|
||
| err := verifyShardNum(coll.MilvusCfg.ShardNum) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| //如果自定义了milvus collection name, 则用它作为collection name | ||
| if len(coll.MilvusCfg.Collection) > 0 { | ||
| err := verifyMilvusCollName(coll.MilvusCfg.Collection) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| } else { | ||
| //否则使用 source Milvus2x collection name 作为collection name | ||
| err := verifyMilvusCollName(coll.Collection) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| } | ||
|
|
||
| if len(coll.MilvusCfg.ConsistencyLevel) > 0 { | ||
| //如果存在ConsistencyLevel配置： | ||
| if _, ok := convert.ConsistencyLevelMap[coll.MilvusCfg.ConsistencyLevel]; !ok { | ||
| return errors.New("[Verify Milvus2x Meta file] ConsistencyLevel value invalid :" + coll.MilvusCfg.ConsistencyLevel) | ||
| } | ||
| } | ||
| } | ||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| package cleaner | ||
|
|
||
| import ( | ||
| "github.com/zilliztech/milvus-migration/internal/log" | ||
| "go.uber.org/zap" | ||
| ) | ||
|
|
||
| type NoneCleaner struct { | ||
| Mode string | ||
| } | ||
|
|
||
| func newNoneCleaner(mode string) *NoneCleaner { | ||
| return &NoneCleaner{Mode: mode} | ||
| } | ||
|
|
||
| func (this *NoneCleaner) CleanFiles() error { | ||
| log.Info("[None Cleaner] not need clean files", zap.String("mode", this.Mode)) | ||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.