Skip to content

feat(storage): read_parquet page index. #9563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jan 16, 2023
Merged

feat(storage): read_parquet page index. #9563

merged 8 commits into from
Jan 16, 2023
+1,139 −299

Conversation

RinChanNOWWW
Copy link
Contributor

@RinChanNOWWW RinChanNOWWW commented Jan 10, 2023 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

  • Filter (Prune) pages by column page stats, and get a row selection.
  • Add unit tests of pruning row groups and pages.
  • Fix bug: page reader should use nums_row but not length.

Others

Closes #issue

@vercel
Copy link

vercel bot commented Jan 10, 2023 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated
databend ⬜️ Ignored (Inspect) Jan 16, 2023 at 6:49AM (UTC)

@RinChanNOWWW RinChanNOWWW changed the title feat(query): read_parquet page index. (WIP) feat(storage): read_parquet page index. (WIP) Jan 10, 2023
@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Jan 10, 2023
@RinChanNOWWW RinChanNOWWW force-pushed the page-index branch 4 times, most recently from 436076c to b58d786 Compare January 13, 2023 09:26
@RinChanNOWWW RinChanNOWWW requested a review from sundy-li January 13, 2023 09:26
@RinChanNOWWW RinChanNOWWW marked this pull request as ready for review January 13, 2023 09:26
@RinChanNOWWW RinChanNOWWW changed the title feat(storage): read_parquet page index. (WIP) feat(storage): read_parquet page index. Jan 13, 2023
@RinChanNOWWW RinChanNOWWW mentioned this pull request Jan 16, 2023
Closed
@RinChanNOWWW
Make pruning a single mod and bump version.
4e8d98c 
And add `ReadOptions`.
@RinChanNOWWW
Prune pages.
8105366
@RinChanNOWWW
Add unit tests for pruning.
bd13ca3 
Other modification:

When creating `RangeFilter`, pass `FunctionContext`.
@RinChanNOWWW
Enrich ReadOptions.

Verified

This commit was signed with the committer’s verified signature.
DmitriyMV Dmitriy Matrenichev
GPG key ID: 94B473337258BFD5
Verified
Learn about vigilant mode
2a1aea8
@RinChanNOWWW
Apply page index filter.

Verified

This commit was signed with the committer’s verified signature.
DmitriyMV Dmitriy Matrenichev
GPG key ID: 94B473337258BFD5
Verified
Learn about vigilant mode
653d50f
@RinChanNOWWW
Fix num_rows of page reader.

Verified

This commit was signed with the committer’s verified signature.
DmitriyMV Dmitriy Matrenichev
GPG key ID: 94B473337258BFD5
Verified
Learn about vigilant mode
c5edd81
@RinChanNOWWW
Fold prewhere expr.
6283505
@BohuTANG
Copy link
Member

@mergify update

@mergify
Copy link
Contributor

mergify bot commented Jan 16, 2023

update

✅ Branch has been successfully updated

@BohuTANG BohuTANG merged commit b68855d into databendlabs:main Jan 16, 2023
@RinChanNOWWW RinChanNOWWW deleted the page-index branch January 16, 2023 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sundy-li sundy-li

Assignees
No one assigned
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@RinChanNOWWW @BohuTANG @sundy-li