v0.21.0

Upgrade to v0.21

v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our Accessor APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously.

Let's start with our errors.

Error Handling

As described in RFC-0977: Refactor Error, we refactor opendal error by a new error
called opendal::Error.

This change will affect all APIs that are used to return io::Error.

To migrate this, please replace std::io::Error with opendal::Error:

- use std::io::Result;
+ use opendal::Result;

And the following error kinds should be updated:

• std::io::ErrorKind::NotFound => opendal::ErrorKind::ObjectNotFound
• std::io::ErrorKind::PermissionDenied => opendal::ErrorKind::ObjectPermissionDenied

And since v0.21, we will return errors ObjectIsADirectory and ObjectNotADirectory instead of anyhow::Error.

Accessor API

In v0.21, we refactor the whole Accessor's API:

- async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<u64>
+ async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<RpWrite>

Since v0.21, we will return a reply struct for different operations called RpWrite instead of an exact type. We can split OpenDAL's public API and raw API with this change.

ObjectList and ObjectPage

Since v0.21, Accessor will return ObjectPager for List:

- async fn list(&self, path: &str, args: OpList) -> Result<ObjectStreamer>
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, ObjectPager)>

And Object will return an ObjectLister which is built upon ObjectPage:

pub async fn list(&self) -> Result<ObjectLister> { ... }

ObjectLister can be used as an object stream as before. It also provides the function next_page to get the underlying pages directly:

impl ObjectLister {
    pub async fn next_page(&mut self) -> Result<Option<Vec<Object>>>;
}

Code Layout

Since v0.21, we have categorized all APIs into public and raw.

Public APIs are exposed under opendal::Xxx; they are user-face APIs that are easy to use and understand.

Raw APIs are exposed under opendal::raw::Xxx; they are implementation details for underlying services and layers.

Please replace all usage of opendal::io_util::* and opendal::http_util::* to opendal::raw::* instead.

With this change, new users of OpenDAL maybe be it easier to get started.

Summary

Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0.

What's Changed

• docs: Add greptimedb and mars into projects by @Xuanwo in #975
• RFC-0977: Refactor Error by @Xuanwo in #977
• refactor: Use seperate Error instead of std::io::Error to avoid confusing by @Xuanwo in #976
• fix: RetryAccessor is too verbose by @Xuanwo in #980
• refactor: Return ReplyCreate for create operation by @Xuanwo in #981
• refactor: Add ReplyRead for read operation by @Xuanwo in #982
• refactor: Add RpWrite for write operation by @Xuanwo in #983
• refactor: Add RpStat for stat operation by @Xuanwo in #984
• refactor: Add RpDelete for delete operations by @Xuanwo in #985
• refactor: Add RpPresign for presign operation by @Xuanwo in #986
• refactor: Add reply for all multipart operations by @Xuanwo in #988
• refactor: Add Reply for all blocking operations by @Xuanwo in #989
• feat: impl atomic write for fs service by @killme2008 in #991
• refactor: Avoid accessor in object entry by @Xuanwo in #992
• refactor: Move accessor into raw apis by @Xuanwo in #994
• refactor: Move io to raw by @Xuanwo in #996
• feat: Add OperatorMetadata to avoid expose AccessorMetadata by @Xuanwo in #997
• refactor: Move {path,wrapper,http_util,io_util} into raw modules by @Xuanwo in #998
• refactor: Move ObjectEntry and ObjectPage into raw by @Xuanwo in #999
• refactor: Accept Operator intead of Arc by @Xuanwo in #1001
• feat: Improve display for error by @Xuanwo in #1002
• Bump to version 0.21 by @Xuanwo in #1003

Full Changelog: v0.20.1...v0.21.0

v0.20.1

What's Changed

• fix: Use std Duration as args instead by @Xuanwo in #966
• build: Make opendal buildable on 1.60 by @Xuanwo in #968
• feat: Implement blocking operations for cache services by @Xuanwo in #970
• fix: Avoid cache missing after write by @Xuanwo in #971
• Bump to version 0.20.1 by @Xuanwo in #972

Full Changelog: v0.20.0...v0.20.1

v0.20.0

Upgrade to v0.20

v0.20 is a big release that we introduce a lot of performance related changes.

To make the best of information from read operation, we propose and implemented RFC-0926: Object Reader. By this RFC, we can fetch content length from ObjectReader now!

pub struct ObjectReader {
    inner: BytesReader
    meta: ObjectMetadata,
}

impl ObjectReader {
    pub fn content_length(&self) -> u64 {}
    pub fn last_modified(&self) -> Option<OffsetDateTime> {}
    pub fn etag(&self) -> Option<String> {}
}

To make this happen, we changed our Accessor API:

- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
+ async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}

All layers should be updated to meet this change. Also, it's required to return content_length while building ObjectReader. Please make sure the returning ObjectMetadata is used correctly.

What's Changed

• RFC-0926: Object Reader by @Xuanwo in #926
• feat: Implement Object Reader by @Xuanwo in #928
• refactor: Return ObjectReader in Accessor::read by @Xuanwo in #929
• feat(services/s3): Return Object Meta for Read operation by @Xuanwo in #932
• feat: Implement Bytes Content Range by @Xuanwo in #933
• feat: Add Content Range support in ObjectMetadata by @Xuanwo in #935
• feat(layers/content_cache): Implement WholeCacheReader by @Xuanwo in #936
• feat: CompressAlgorithm derive serde. by @youngsofun in #939
• fix(ops): Fix suffix range behavior of bytes range by @Xuanwo in #942
• refactor(oay,oli): drop unnecessary patch.crates-io from Cargo.toml by @messense in #944
• feat: Allow using opendal without tls support by @messense in #945
• refactor: Refactor OpRead with BytesRange by @Xuanwo in #946
• feat: Allow using opendal with native tls support by @messense in #949
• docs: add docs for tls dependencies features by @messense in #951
• refactor: Polish bytes range by @Xuanwo in #950
• feat: Make ObjectReader content_length returned for all services by @Xuanwo in #954
• feat(layers): Implement fixed content cache by @Xuanwo in #953
• fix: Fix cache path not used correctly by @Xuanwo in #958
• refactor: Use simplifed kv adapter instead by @Xuanwo in #959
• feat: Enable default_ttl support for redis by @Xuanwo in #960
• Bump to version 0.20.0 by @Xuanwo in #961

New Contributors

• @messense made their first contribution in #944

Full Changelog: v0.19.8...v0.20.0

v0.19.8

What's Changed

• ci: Disable ipfs's compress integration tests by @Xuanwo in #899
• deps(oay,oli): Update deps by @Xuanwo in #906
• refactor: Reduce backend builder log level to debug by @Xuanwo in #907
• ci: Make stable rust clippy happy by @Xuanwo in #911
• feat(services/moka): Use entry's bytes as capacity weigher by @Xuanwo in #914
• feat: Implement rocksdb service by @wfxr in #913
• docs: add docs for rocksdb service by @wfxr in #915
• fix(http): Check already read size before returning by @Xuanwo in #919
• refactor: Remove deprecated features by @Xuanwo in #920
• refactor: use moka::sync::SegmentedCache by @PsiACE in #921
• Bump to version 0.19.8 by @Xuanwo in #923

New Contributors

• @wfxr made their first contribution in #913

Full Changelog: v0.19.7...v0.19.8

v0.19.7

What's Changed

• feat: Implement content type support for stat by @Xuanwo in #891
• refactor(layers/metrics): Holding all metrics handlers to avoid lock by @Xuanwo in #894
• refactor(layers/metrics): Only update metrics while dropping readers by @Xuanwo in #896
• Bump to version 0.19.7 by @Xuanwo in #897

Full Changelog: v0.19.6...v0.19.7

v0.19.6

What's Changed

• fix: Metrics blocking reader doesn't handle operation correctly by @Xuanwo in #887
• Bump to version 0.19.6 by @Xuanwo in #888

Full Changelog: v0.19.5...v0.19.6

v0.19.5

What's Changed

• chore(deps): update redis requirement from 0.21 to 0.22 by @dependabot in #876
• feat: add a feature named trust-dns by @PsiACE in #879
• feat: implement write_with by @ClSlaid in #880
• feat: content-type configuration by @ClSlaid in #878
• chore(deps): update quick-xml requirement from 0.25 to 0.26 by @dependabot in #882
• fix: Allow forward layers' acesser operations to inner by @Xuanwo in #884
• Bump to version 0.19.5 by @Xuanwo in #885

Full Changelog: v0.19.4...v0.19.5

v0.19.4

What's Changed

• refactor: replace md5 with md-5 by @PsiACE in #862
• feat: Improve into_stream by reduce zero byte fill by @Xuanwo in #864
• debug: Add log for sync http client by @Xuanwo in #865
• refactor: replace the hard code to X_AMZ_BUCKET_REGION constant by @WenyXu in #866
• feat: Add debug log for finishing read by @Xuanwo in #867
• feat: Try to use trust-dns-resolver by @Xuanwo in #869
• feat: Add log for dropping reader and streamer by @Xuanwo in #870
• Bump to version 0.19.4 by @Xuanwo in #871

New Contributors

• @WenyXu made their first contribution in #866

Full Changelog: v0.19.3...v0.19.4

v0.19.3

What's Changed

• fix: Retry for wirte is not implemented correctly by @Xuanwo in #860
• Bump to version 0.19.3 by @Xuanwo in #861

Full Changelog: v0.19.2...v0.19.3

v0.19.2

What's Changed

• feat(experiment): Allow user to config http connection pool by @Xuanwo in #843
• feat: Add concurrent limit layer by @Xuanwo in #848
• feat: Allow kv services implemented without list support by @Xuanwo in #850
• feat(experiment): add a basic content data cache layer by @PsiACE in #849
• feat: Implement service for moka by @Xuanwo in #852
• docs: Add docs for moka service and concurrent limit layer by @Xuanwo in #857
• Bump to version 0.19.2 by @Xuanwo in #858

Full Changelog: v0.19.1...v0.19.2