ci: Aliyun Drive Test failed

[Xuanwo]

https://github.com/apache/opendal/actions/runs/9566562882/job/26372216975

2024-06-18T14:14:17.2955702Z test behavior::test_delete_stream                                      ... ok
2024-06-18T14:14:17.2956375Z 
2024-06-18T14:14:17.2956529Z failures:
2024-06-18T14:14:17.2956761Z 
2024-06-18T14:14:17.2957113Z ---- behavior::test_list_prefix ----
2024-06-18T14:14:17.2957794Z test panicked: assertion `left == right` failed
2024-06-18T14:14:17.2958781Z   left: 0
2024-06-18T14:14:17.2959188Z  right: 1
2024-06-18T14:14:17.2959426Z 
2024-06-18T14:14:17.2959764Z ---- behavior::test_list_empty_dir ----
2024-06-18T14:14:17.2960766Z test panicked: assertion `left == right` failed: only return the dir itself, but found: {}
2024-06-18T14:14:17.2961705Z   left: 0
2024-06-18T14:14:17.2962073Z  right: 1
2024-06-18T14:14:17.2962302Z 
2024-06-18T14:14:17.2962596Z ---- behavior::test_list_sub_dir ----
2024-06-18T14:14:17.2964939Z test panicked: dir should be found in list, but only got: [Entry { path: "6717aa09-c611-4047-89b7-7036c6dffa0c/", metadata: Metadata { metakey: FlagSet(Complete | Mode | LastModified), mode: DIR, cache_control: None, content_disposition: None, content_length: None, content_md5: None, content_range: None, content_type: None, etag: None, last_modified: Some(2024-06-18T14:09:45.798Z), version: None } }]
2024-06-18T14:14:17.2966735Z 
2024-06-18T14:14:17.2966978Z ---- behavior::test_list_dir_with_file_path ----
2024-06-18T14:14:17.2967433Z test panicked: assertion `left == right` failed
2024-06-18T14:14:17.2967819Z   left: 0
2024-06-18T14:14:17.2968052Z  right: 1
2024-06-18T14:14:17.2968183Z 
2024-06-18T14:14:17.2968187Z 
2024-06-18T14:14:17.2968283Z failures:
2024-06-18T14:14:17.2968530Z     behavior::test_list_prefix
2024-06-18T14:14:17.2968862Z     behavior::test_list_empty_dir
2024-06-18T14:14:17.2969201Z     behavior::test_list_sub_dir
2024-06-18T14:14:17.2969560Z     behavior::test_list_dir_with_file_path
2024-06-18T14:14:17.2969823Z 
2024-06-18T14:14:17.2970207Z test result: FAILED. 110 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 274.82s

[Xuanwo]

Reading data at 4K doesn't make sense:

  2024-06-18T15:04:32.588102Z TRACE opendal::services: service=aliyun_drive operation=Reader::read path=07367cad-8f97-4e24-a75c-19c507944a55 read=154902 -> read returns 4096B
    at src/layers/logging.rs:986

  2024-06-18T15:04:32.591711Z TRACE opendal::services: service=aliyun_drive operation=Reader::read path=07367cad-8f97-4e24-a75c-19c507944a55 read=158998 -> read returns 4096B
    at src/layers/logging.rs:986

Maybe aliyun drive is too slow to test from us-east-1. This read is about 1.1KiB/s.

[Xuanwo]

We will also hit errors like:

>                   operator.read(f"{parent}/{path}")
E                   opendal.UnexpectedError: RateLimited (persistent) at stat, context: { service: aliyun_drive, path: random_dir_86b5bcca-098d-4190-9511-2b92cd912490/x/x/y } => 请求过快，请等待 x-retry-after 毫秒后再请求

[Xuanwo]

Hello @yuchanns and @suyanhanx, thank you for your efforts in bringing the Aliyun Drive integration test online. However, it consistently fails on our main branch. The primary issue seems to be that our CI runs in us-east-1, while Aliyun Drive is located in mainland China.

I propose we disable this test until we can resolve the issue. What are your thoughts?

[Xuanwo]

It looks good on #4767 tests. While AliyunDrive severely restricts concurrency access per account. Core-Test, Binding-Python, and Binding-Java consist of three parallel on one account leads to many retries.

How about disabling aliyun_drive test on bindings? cc @suyanhanx

[suyanhanx]

It looks good on #4767 tests. While AliyunDrive severely restricts concurrency access per account. Core-Test, Binding-Python, and Binding-Java consist of three parallel on one account leads to many retries.

How about disabling aliyun_drive test on bindings? cc @suyanhanx

SGTM

[Xuanwo]

SGTM

We can remove aliyun drive from the binding's planner. Would you like to create a PR for this?

[yuchanns]

Maybe aliyun drive is too slow to test from us-east-1. This read is about 1.1KiB/s.

Aliyun has had a limitation of speed on non-VIP accounts recently.

[suyanhanx]

SGTM

We can remove aliyun drive from the binding's planner. Would you like to create a PR for this?

I'll do this.

[yuchanns]

I propose optional splittable credentials for binding tests since the limit is per account. Besides, there may be other services encountering such limitations in the future.

[Xuanwo]

I propose optional splittable credentials for binding tests since the limit is per account. Besides, there may be other services encountering such limitations in the future.

Well, it's much harder...

All our services use the same setup and configuration. Allowing splittable credentials for services adds significant complexity to our current CI system.

[Xuanwo]

All our services use the same setup and configuration. Allowing splittable credentials for services adds significant complexity to our current CI system.

A quick glance:

• https://github.com/apache/opendal/blob/main/.github/scripts/test_behavior/plan.py
• https://github.com/apache/opendal/blob/main/.github/workflows/test_behavior_bin_ofs.yml
• https://github.com/apache/opendal/blob/main/.github/actions/test_behavior_core/action.yaml

[Xuanwo]

Good news, the aliyun drive test seems much stable now (passed in previous twice without retry!)

[yuchanns]

Yes. Previously before #4766, there was a logical error I misunderstood about list behavior. It is hard to understand why it passed tests, especially on one thread.

[Xuanwo]

Yes. Previously before #4766, there was a logical error I misunderstood about list behavior. It is hard to understand why it passed tests, especially on one thread.

Cool, we can watch this test for sometime. We can keep the binding tests if it's stable for a while.

[suyanhanx]

Opened a PR #4770 that disabled the test aliyun_drive on bindings temporarily.

[Xuanwo]

Thank you all, now aliyun tests work well!