Skip to content

feature: support concurrent multipart object upload with s3 backend? #3837

New issue
New issue
Open
Open
feature: support concurrent multipart object upload with s3 backend?#3837
Labels
area/storage/s3enhancementfeaturehelp wanted
@zhucan

Description

@zhucan
zhucan
opened on Feb 7, 2023

rgw logs:

2023-01-30T09:06:29.411+0000 7f4a8d2be700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:06:28.269 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=1&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.142009735s

2023-01-30T09:06:37.914+0000 7f49e1166700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:06:36.585 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=2&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.329011321s

2023-01-30T09:06:46.531+0000 7f4a5c25c700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:06:45.225 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=3&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.306011081s

2023-01-30T09:06:54.822+0000 7f4a8a2b8700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:06:53.265 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=4&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.557013273s

2023-01-30T09:07:02.879+0000 7f49cd13e700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:07:01.416 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=5&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.463012457s

2023-01-30T09:07:10.677+0000 7f4a181d4700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:07:09.415 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=6&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.262010694s

2023-01-30T09:07:18.327+0000 7f49bf122700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:07:17.081 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=7&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.245010734s

2023-01-30T09:07:26.367+0000 7f4a2c1fc700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx [30/Jan/2023:09:07:24.893 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=8&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.473012567s

2023-01-30T09:07:33.966+0000 7f49e596f700  1 beast: 0x7f499a681630: [x.x.x.x](http://x.x.x.x/) - xx[30/Jan/2023:09:07:32.752 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=9&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 104857600 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.214010358s

2023-01-30T09:07:37.338+0000 7f4a35a0f700  1 beast: 0x7f499a681630: [x.x.x.x](http:/x.x.x.x/) - xx[30/Jan/2023:09:07:35.505 +0000] "PUT /test/docker/registry/v2/repositories/test/xhu3/_uploads/4bb7e46b-af19-4d9b-9221-5291202eefe5/data?partNumber=10&uploadId=2~8Pl6k46tGEWRCEhmFXCSzDqtRQnup7p HTTP/1.1" 200 130351486 - "aws-sdk-go/1.15.11 (go1.19.4; linux; amd64)" - latency=1.832015634s

from ceph rgw logs: we know the distribution push the object with multiparts one by one and not in concurrency.

Activity

davidspek
added
enhancement
feature
area/storage/s3
on Aug 16, 2023
davidspek
changed the title [-]features: need to support multipart upload object in concurrency with s3 backend?[/-] [+]feature: support concurrent multipart object upload with s3 backend?[/+] on Aug 16, 2023
davidspek
added
help wanted
on Aug 16, 2023
shuaichang

shuaichang commented on Dec 1, 2023

@shuaichang
shuaichang
on Dec 1, 2023

+1, seems to be a great improvement for accelerating artifact pushes, e.g. a very large model stored as an OCI artifact.

milosgajdos

milosgajdos commented on Dec 1, 2023

@milosgajdos
milosgajdos
on Dec 1, 2023
Member

Feel free to open a PR, but I have a feeling this might require some serious refactoring 🤔

biello

biello commented on May 7, 2024

@biello
biello
on May 7, 2024

is multipart upload still serial now?

zhucan

zhucan commented on May 7, 2024

@zhucan
zhucan
on May 7, 2024
Author

is multipart upload still serial now?

yes

ujjwal

ujjwal commented on Apr 18, 2025

@ujjwal
ujjwal
on Apr 18, 2025

Any progress on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/storage/s3enhancementfeaturehelp wanted

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @ujjwal@milosgajdos@biello@zhucan@davidspek

        Issue actions
          feature: support concurrent multipart object upload with s3 backend? · Issue #3837 · distribution/distribution