Support integrating TOS as the UnderFileSystem #18621
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jackson-Wang-7
requested changes
Jun 7, 2024
| .build(); | ||
| public static final PropertyKey TOS_REGION = | ||
| stringBuilder(Name.TOS_REGION) | ||
| .setDescription("The region name of TOS bucket.") |
There was a problem hiding this comment.
please align to the previous line.
underfs/tos/pom.xml
Outdated
| <dependency> | ||
| <groupId>com.volcengine</groupId> | ||
| <artifactId>ve-tos-java-sdk</artifactId> | ||
| <version>2.7.1</version> |
There was a problem hiding this comment.
define the version in the root pom.xml, not here
| import javax.annotation.concurrent.NotThreadSafe; | ||
|
|
||
| /** | ||
| * A stream for reading a file from COS. This input stream returns 0 when calling read with an empty |
There was a problem hiding this comment.
Suggested change
| * A stream for reading a file from COS. This input stream returns 0 when calling read with an empty | |
| * A stream for reading a file from TOS. This input stream returns 0 when calling read with an empty |
| import javax.annotation.concurrent.NotThreadSafe; | ||
|
|
||
| /** | ||
| * A stream for writing a file into COS. The data will be persisted to a temporary directory on the |
There was a problem hiding this comment.
Suggested change
| * A stream for writing a file into COS. The data will be persisted to a temporary directory on the | |
| * A stream for writing a file into TOS. The data will be persisted to a temporary directory on the |
| return new ObjectStatus(key, output.getEtag(), output.getContentLength(), | ||
| lastModifiedTime); | ||
| } catch (TosException e) { | ||
| return null; |
There was a problem hiding this comment.
please check the exception, if it's 404 not found, then return null, otherwise throw a exception
| } | ||
| } | ||
|
|
||
| String err = "TOS Credentials not available, cannot create TOS Under File System."; |
There was a problem hiding this comment.
Suggested change
| String err = "TOS Credentials not available, cannot create TOS Under File System."; | |
| String err = "TOS Credentials or configurations not available, cannot create TOS Under File System."; |
Jackson-Wang-7
added
area-ufs
|
Automated checks report:
Some checks failed. Please fix the reported issues and reply |
|
Automated checks report:
All checks passed! |
Jackson-Wang-7
changed the title
Jackson-Wang-7
approved these changes
Jun 7, 2024
| @@ -47,7 +48,8 @@ public UnderFileSystem create(String path, UnderFileSystemConfiguration conf) { | |||
| } | |||
| } | |||
|
|
|||
| String err = "OSS Credentials not available, cannot create OSS Under File System."; | |||
| String err = | |||
There was a problem hiding this comment.
@thu-david this is the OSS UnderFileSystem Factory, please remove this line
|
alluxio-bot, merge this please. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Created an interface for TOS, enabling Alluxio to operate and cache data specifically for TOS.
Why are the changes needed?
Tested the creation, deletion, metadata retrieval, and directory information retrieval for files of sizes 1MB, 64MB, 1GB, and 10GB. This can support the most basic TOS query requests.
Does this PR introduce any user facing changes?
This implementation continues to follow Alluxio's API. To use it, you only need to provide the TOS information in the configuration file.