Skip to content

feat(cache): add generic cache trait for different object reader #9436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Jan 1, 2023
Merged

feat(cache): add generic cache trait for different object reader #9436

merged 11 commits into from
Jan 1, 2023
+710 −0

Conversation

BohuTANG
Copy link
Member

@BohuTANG BohuTANG commented Dec 31, 2022 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Goal

The goal is for each object cache provider to be more flexible and able to define its cache policy:

CachedObjectAccessor -> Cache Provider(ByPassCache/MemoryBytesCache/MemoryItemsCache/FileCache) --> OpenDAL object operations

Usage

Usage(MemoryItemsCache):

    let mut builder: Builder = fs::Builder::default();
    builder.root("/tmp");
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();
    let object = op.object(&path);

    // Cache provider.
    let settings = CacheSettings::default();
    let cache = Arc::new(MemoryItemsCache::create(&settings));

    // Cached Object Accessor.
    let accessor = CachedObjectAccessor::create(cache.clone());
    accessor.write(&object, expect.clone()).await?;

    let actual: Arc<TestMeta> = accessor.read(&object, 0, 16).await?;

Usage(ByPassCache = No cache, same as object operations via CachedObjectAccessor):

    let mut builder: Builder = fs::Builder::default();
    builder.root("/tmp");
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();
    let object = op.object(&path);

    // Cache provider.
    let settings = CacheSettings::default();
    let cache = Arc::new(ByPassCache::create(&settings));

    // Cached Object Accessor.
    let accessor = CachedObjectAccessor::create(cache.clone());
    accessor.write(&object, expect.clone()).await?;

    let actual: Arc<TestMeta> = accessor.read(&object, 0, 16).await?;

Notes

This is the first part. We implement three cache backends:

ByPassCache. -- DONE
MemoryBytesCache -- DONE
MemoryItemsCache -- DONE
FileCache -- TODO , for block data

Next PR will try to replace the following:
SegmentInfoCache/TableSnapshotCache/TableSnapshotStatisticCache/BloomIndexCache/BloomIndexMetaCache/FileMetaDataCache to the new.

Closes #issue

@vercel
Copy link

vercel bot commented Dec 31, 2022 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated
databend ⬜️ Ignored (Inspect) Dec 31, 2022 at 3:02PM (UTC)

@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Dec 31, 2022
@BohuTANG BohuTANG requested a review from dantengsky December 31, 2022 15:12
@BohuTANG BohuTANG marked this pull request as ready for review December 31, 2022 15:12
@BohuTANG
Copy link
Member Author

BohuTANG commented Jan 1, 2023

Let's merge and continue the work.

@BohuTANG BohuTANG merged commit ed69629 into databendlabs:main Jan 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PsiACE PsiACE

@dantengsky dantengsky

Assignees
No one assigned
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@BohuTANG @PsiACE