Description
Application contact emails
jicheng.shi@datenlord.com,pu.wang@datenlord.com, zheng.pan@datenlord.com, tracy.zhu@datenlord.com
Project Summary
A geo-distributed KV store for metadata management
Project Description
Xline is a distributed KV storage for metadata management. Xline makes it possible to manage metadata, such as indexes, permissions, and configurations across multiple clusters. Xline can achieve high-performance data access and strong consistency in cross data center scenarios. Currently, Xline is compatible with the ETCD interface, so that existing ETCD users can seamlessly switch to Xline and get high performance metadata management across multiple clusters.
The Aim of the Project
Data isolation and data fragmentation resulting from cloud barrier become impediments to business growth. The biggest challenge of multi-data center architecture is how to maintain strong data consistency and ensure high performance in the scenarios of multi-data center competition and modification. Traditional single data center solutions cannot meet the availability, performance and consistency requirements of multi-data center scenarios. DatenLord launched the open source distributed KV storage called Xline, which is targeted to multi-data center scenarios. It aims to realize a high-performance metadata management solution across multiple clusters, which is critical for businesses with geo-distributed, and multi-active deployment requirements.
Characteristics of the project
- Accelerate data access across clouds by caching hot data in memory, and provide unified data management to automate data migration and backup.
- It is the first geo distributed consistency management service based on the WAN consensus protocol. It addresses the challenges of convergence and consistency across clouds.
- Unified data management for multi-cluster scenarios, making mutual access, discovery, and modification simple and convenient.
- High stability and excellent storage performance; Provide KV interface, Multi-Version Concurrency Control and is compatible with ETCD.
Org repo URL
Project repo URL
https://github.com/datenlord/xline
Additional repos
No response
Website URL
No response
Roadmap
https://github.com/datenlord/Xline#roadmap
Roadmap context
No response
Contributing Guide
https://github.com/datenlord/Xline/blob/master/CONTRIBUTING.md
Code of Conduct (CoC)
https://github.com/datenlord/Xline/blob/master/CODE_OF_CONDUCT.md
Adopters
No response
Contributing or Sponsoring Org
No response
Maintainers file
https://github.com/datenlord/Xline/blob/master/MAINTAINERS
IP Policy
- If the project is accepted, I agree the project will follow the CNCF IP Policy
Trademark and accounts
- If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF
Why CNCF?
The geo-distributed deployment has become increasingly popular nowadays, in which situation Xline is of help in terms of unified management of metadata with high performance and security. We believe that Xline is a missing piece of the CNCF landscape. The project is motivated by challenges observed in cloud-native environments, and will benefit users and organizations that store data in multi-clusters and across clouds. Xline is built naturally for cloud native, compatible with ETCD interface. The success of this project depends on maintaining its vendor-neutrality, as well as keeping collaboration with project adjacent in the ecosystem. Given its mission and relationship with the CNCF projects, we truly believe CNCF, as an open source platform, is a perfect place to host Xline.
We hope to acquire support (e.g., legal assistance in aspects of potential IP and trademark issues) from CNCF, and collaborate with other projects in CNCF to make geo-distributed deployment convenient. We believe the inclusion of Xline in the CNCF ecosystem could help increase the awareness of the project. With the guidance of CNCF, we can follow its procedures and standards, and set attainable goals to refine our open source tools, promote community engagement, and integrate Xline with other cloud native projects.
Benefit to the Landscape
Xline is focused on the infrastructure of cloud native storage, which is vital to the development of cloud native technology. The joining of Xline will help CNCF accelerate the evolution of cloud native storage system in terms of data access in geo-distributed scenarios, in turn benefiting the entire cloud native ecosystem.
First, Xline is fully aligned with the cloud native computing ecosystem because it is compatible with ETCD interface. ETCD enables K8S management in a single cluster, while Xline expands cloud native scenarios to multi clusters and allows users to experience task orchestration across multiple clusters by K8S. Moreover, Xline achieves metadata management among multi-clusters with high performance and security.
Cloud Native 'Fit'
No response
Cloud Native 'Integration'
No response
Cloud Native Overlap
No response
Similar projects
Product or Service to Project separation
N/A
Project presentations
No response
Project champions
No response
Additional information
Here's the product intro slides
Activity
dims commentedon Jan 6, 2023
@rogercloud looks like some notes on CURP vs RAFT would be a good thing, can you please tell us how they compare? thanks!
rogercloud commentedon Feb 27, 2023
@dims For the requests don't conflict with each other, CURP takes half time to get the requests committed compared to Raft, because CURP requries only one RTT while Raft requires two. In the multi-cluster or multi-datecenter casese, CURP out-performs Raft. Even in the worst case that every request conflicts with others, CURP's performance is the same as Raft's. You get the benchmark result in our project page.
cathyhongzhang commentedon May 8, 2023
It is mentioned that the geo distributed consistency management service is based on the WAN consensus protocol. Which protocol is it? How does this protocol compare with raft and paxos? Could you list the Pros and cons?
cathyhongzhang commentedon May 8, 2023
I do not see the maintainer's name. Did I miss something?
justincormack commentedon May 9, 2023
Have you done any correctness testing on the project? eg Jepsen or equivalent?
rogercloud commentedon May 9, 2023
@cathyhongzhang
The protocol is called CURP, which is published in NSDI'19.
Compared to Raft or Paxos, it takes only 1 RRT to commit a request in the normal case while Raft takes 2. Even for the worst case, every request is changing the same key's value, CUPR's performance is the same as Raft. For more protocol comparison, please refer to the blog. For more performance result, you can refer to the README of the project.
The Pros of CURP:
The Cons of CURP:
38 remaining items