Who is using Apache Celeborn?

[SteNicholas]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Who's using Apache Celeborn?

The Apache Celeborn community grows with the combined efforts of every user and developer and appreciates every user and developer who uses and cares about Celeborn. We are committed to listening to each one of you to make the community more prosperous.

Therefore, the purpose of this topic is to encourage you, as a user or developer, to leave your real-world scenarios so that we can subsequently explore new and better solutions, schedule the releases, and solve real-world problems based on these scenarios.

We look forward to your comments in the comments space, including but not limited to the following areas.

• Organization: e.g. Alibaba Corp.
• Region: e.g. Hangzhou, China
• Contact: e.g. [email protected]
• Usage Scenario: e.g., we use Apache Celeborn to improve the efficiency and elasticity of different map-reduce engines and provides an elastic, high-efficient management service for intermediate data including shuffle data, spilled data, result data, etc.

Regards,

Apache Celeborn Community

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[gaochao0509]

Organization: New Baatur
Region: Shen zhen, China
Contact information: [email protected]
Usage scenario: Use Apache Celeborn to improve the efficiency and resilience of the Spark/MapReduce engine, providing elastic and efficient management services for intermediate data such as Shuffle data, overflow data, and result data.
Location: Currently, 10+locations have been deployed, with significant stability improvement and performance improvement of over 30%.

[cxzl25]

• Organization: Trip.com Group
• Region: Shanghai, China
• Contact: [email protected]
• Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

[big-loser]

Organization: inmobi Group
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

[RuiQin7]

Organization: zto.com Group
Region: Shanghai, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle

[BsoBird]

Organization: shuyun Inc
Region: hangzhou, China
Contact: [email protected]
Usage Scenario: By using apache celeborn, We realised the separation of storage and compute in HIVE-MR3/SPARK-KYUUBI, which effectively reduces the cost of the server room.

[SteNicholas]

• Organization: Bilibili Inc
• Region: Shanghai, China
• Contact: [email protected]
• Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark and Flink(Batch) engine shuffle.
  

[jiaoqingbo]

• Organization: AsiaInfo
• Region: Beijing, China
• Contact: [email protected]
• Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.
  

[lifulong]

Organization: Zhihu Inc
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark engine shuffle.

[TongWei1105]

Organization: Xiaohongshu Inc
Region: Shanghai, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark and Flink(Batch) engine shuffle.

[onebox-li]

• Organization: BIGO
• Region: Guangzhou, China
• Contact: [email protected]
• Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.
  

[AngersZhuuuu]

Organization: SHOPEE
Region: Shenzhen, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

[lianneli]

Organization: DMALL
Region: Beijing, China
Contact: [email protected], [email protected], [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle.

[camper42]

Organization: Douban
Region: Beijing, China
Contact: [email protected]
Usage Scenario: we use Apache Celeborn to support better Spark dynamic resource allocation in Kubernetes.

[wfxxh]

Organization: WanFang Data
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Apache Celeborn to improve the stability and efficiency of Spark shuffle on k8s

[wxplovecc]

Organization: Tongcheng Travel（同程旅行）.
Region: Jiangsu, China
Contact: www.ly.com
Usage Scenario: we use Apache Celeborn to improve the stability and efficiency of spark shuffle.

[CodingCat]

Organization: e.g. Pinterest
Region: US
Contact: @CodingCat
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

[akpatnam25]

Organization: LinkedIn
Region: US
Contact: @akpatnam25
Usage Scenario: we use Apache Celeborn to improve the stability and efficiency of spark shuffle.

[liugs0213]

Organization: BOSS Zhipin
Region: Beijing, China
Contact: @liugs0213
Usage Scenario: we use Apache Celeborn to improve the efficiency and elasticity of Spark@K8S

[lohitvijayarenu]

Organization: Stripe
Region: US
Contact: @lohitvijayarenu
Usage Scenario: Apache Celeborn helps Spark workloads with efficient and elastic shuffle. Celeborn will also help in Storage and compute disaggregation for efficient use of resources.

[wangshengjie123]

Organization: Xiaomi
Region: Beijing, China
Contact: [email protected]
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

[ASiegeLion]

Organization: YY
Region: GuangZhou, China
Contact: https://www.yy.com/web/jinhong/#/connect
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.

[jinyuchen724]

Organization: WuXianShengHuo
Region: HangZhou, China
Contact: https://www.weidian.com
Usage Scenario: We use Celeborn to improve the stability and efficiency of shuffle.