Description
Application contact emails
jicheng.shi@datenlord.com,pu.wang@datenlord.com, zheng.pan@datenlord.com, tracy.zhu@datenlord.com
Project Summary
A geo-distributed KV store for metadata management
Project Description
Xline is a distributed KV storage for metadata management. Xline makes it possible to manage metadata, such as indexes, permissions, and configurations across multiple clusters. Xline can achieve high-performance data access and strong consistency in cross data center scenarios. Currently, Xline is compatible with the ETCD interface, so that existing ETCD users can seamlessly switch to Xline and get high performance metadata management across multiple clusters.
The Aim of the Project
Data isolation and data fragmentation resulting from cloud barrier become impediments to business growth. The biggest challenge of multi-data center architecture is how to maintain strong data consistency and ensure high performance in the scenarios of multi-data center competition and modification. Traditional single data center solutions cannot meet the availability, performance and consistency requirements of multi-data center scenarios. DatenLord launched the open source distributed KV storage called Xline, which is targeted to multi-data center scenarios. It aims to realize a high-performance metadata management solution across multiple clusters, which is critical for businesses with geo-distributed, and multi-active deployment requirements.
Characteristics of the project
- Accelerate data access across clouds by caching hot data in memory, and provide unified data management to automate data migration and backup.
- It is the first geo distributed consistency management service based on the WAN consensus protocol. It addresses the challenges of convergence and consistency across clouds.
- Unified data management for multi-cluster scenarios, making mutual access, discovery, and modification simple and convenient.
- High stability and excellent storage performance; Provide KV interface, Multi-Version Concurrency Control and is compatible with ETCD.
Org repo URL
Project repo URL
https://github.com/datenlord/xline
Additional repos
No response
Website URL
No response
Roadmap
https://github.com/datenlord/Xline#roadmap
Roadmap context
No response
Contributing Guide
https://github.com/datenlord/Xline/blob/master/CONTRIBUTING.md
Code of Conduct (CoC)
https://github.com/datenlord/Xline/blob/master/CODE_OF_CONDUCT.md
Adopters
No response
Contributing or Sponsoring Org
No response
Maintainers file
https://github.com/datenlord/Xline/blob/master/MAINTAINERS
IP Policy
- If the project is accepted, I agree the project will follow the CNCF IP Policy
Trademark and accounts
- If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF
Why CNCF?
The geo-distributed deployment has become increasingly popular nowadays, in which situation Xline is of help in terms of unified management of metadata with high performance and security. We believe that Xline is a missing piece of the CNCF landscape. The project is motivated by challenges observed in cloud-native environments, and will benefit users and organizations that store data in multi-clusters and across clouds. Xline is built naturally for cloud native, compatible with ETCD interface. The success of this project depends on maintaining its vendor-neutrality, as well as keeping collaboration with project adjacent in the ecosystem. Given its mission and relationship with the CNCF projects, we truly believe CNCF, as an open source platform, is a perfect place to host Xline.
We hope to acquire support (e.g., legal assistance in aspects of potential IP and trademark issues) from CNCF, and collaborate with other projects in CNCF to make geo-distributed deployment convenient. We believe the inclusion of Xline in the CNCF ecosystem could help increase the awareness of the project. With the guidance of CNCF, we can follow its procedures and standards, and set attainable goals to refine our open source tools, promote community engagement, and integrate Xline with other cloud native projects.
Benefit to the Landscape
Xline is focused on the infrastructure of cloud native storage, which is vital to the development of cloud native technology. The joining of Xline will help CNCF accelerate the evolution of cloud native storage system in terms of data access in geo-distributed scenarios, in turn benefiting the entire cloud native ecosystem.
First, Xline is fully aligned with the cloud native computing ecosystem because it is compatible with ETCD interface. ETCD enables K8S management in a single cluster, while Xline expands cloud native scenarios to multi clusters and allows users to experience task orchestration across multiple clusters by K8S. Moreover, Xline achieves metadata management among multi-clusters with high performance and security.
Cloud Native 'Fit'
No response
Cloud Native 'Integration'
No response
Cloud Native Overlap
No response
Similar projects
Product or Service to Project separation
N/A
Project presentations
No response
Project champions
No response
Additional information
Here's the product intro slides
Activity
dims commentedon Jan 6, 2023
@rogercloud looks like some notes on CURP vs RAFT would be a good thing, can you please tell us how they compare? thanks!
rogercloud commentedon Feb 27, 2023
@dims For the requests don't conflict with each other, CURP takes half time to get the requests committed compared to Raft, because CURP requries only one RTT while Raft requires two. In the multi-cluster or multi-datecenter casese, CURP out-performs Raft. Even in the worst case that every request conflicts with others, CURP's performance is the same as Raft's. You get the benchmark result in our project page.
cathyhongzhang commentedon May 8, 2023
It is mentioned that the geo distributed consistency management service is based on the WAN consensus protocol. Which protocol is it? How does this protocol compare with raft and paxos? Could you list the Pros and cons?
cathyhongzhang commentedon May 8, 2023
I do not see the maintainer's name. Did I miss something?
justincormack commentedon May 9, 2023
Have you done any correctness testing on the project? eg Jepsen or equivalent?
rogercloud commentedon May 9, 2023
@cathyhongzhang
The protocol is called CURP, which is published in NSDI'19.
Compared to Raft or Paxos, it takes only 1 RRT to commit a request in the normal case while Raft takes 2. Even for the worst case, every request is changing the same key's value, CUPR's performance is the same as Raft. For more protocol comparison, please refer to the blog. For more performance result, you can refer to the README of the project.
The Pros of CURP:
The Cons of CURP:
23 remaining items
rogercloud commentedon Jun 17, 2023
Dear TOC members,
We would like to take this opportunity to highlight some key features and innovations of Xline. Xline is an open source distributed KV storage, which is used to manage metadata, and ensure high performance and strong data consistency in cross-cloud and cross-data center scenarios. Its unique features are as follows:
Distributed Multi-Cluster KV Storage Engine: Xline enables seamless data management in multi-cluster environments, providing a simple and convenient approach for mutual access, discovery, and modification of data.
Unified Metadata Management System: By caching hot data in memory, Xline facilitates cross-cloud data access and automates data migration and backup, thus establishing a unified platform for efficient and effective data management.
High-performance Multi-Data Center Consensus Protocol: Xline pioneers a geographically distributed consistency management server based on a wide area network consensus protocol. This breakthrough innovation resolves the challenges associated with cross-cloud integration and ensures data consistency across multiple data centers.
Compatible with ETCD Interface: Xline offers compatibility with the widely adopted ETCD interface, providing users with familiar KV interface and multi-version concurrency control. This compatibility enables smooth utilization and migration within Kubernetes (K8S) environments.
As for Xline’s roadmap, we plan to
If you have any questions or comment, please feel free to let us know. Thank you!
rogercloud commentedon Jun 18, 2023
/check-vote
git-vote commentedon Jun 18, 2023
Vote status
So far
72.73%
of the users with binding vote are in favor (passing threshold:66%
).Summary
Binding votes (8)
Non-binding votes (8)
git-vote commentedon Jun 20, 2023
Vote closed
The vote passed! 🎉
81.82%
of the users with binding vote were in favor (passing threshold:66%
).Summary
Binding votes (9)
Non-binding votes (8)
amye commentedon Jun 21, 2023
Closing with approved, new onboarding issue: #156