Can you provide the specific error via the debug flags:

s3fs $BUCKET $MOUNTPOINT -d -d -f -o f2 -o curldbg

as well as share the S3 implementation you use, e.g., Amazon, Ceph? Errors at the 5 GB boundary imply some misconfiguration around multi-part uploads.

Hi Andrew,

Thanks for your help!

Could you please explain a bit more the step in order to retreive the log please?

• So correct me if I'm wrong but I need to umount my bucket first then mount it with :

s3fs $BUCKET $MOUNTPOINT -d -d -f -o f2 -o curldbg (I also need to add -o use_cache and -o allow_other)

Regards.
Jonathan

Yes please invoke s3fs with those options then reproduce the symptoms with your application. When you encounter the error, please attach the relevant symptoms here or in a gist.

@Phantom-Studio
s3fs has file size limit which is dependent on "multipart_size" option.
S3 has a limit count for multi part uploading, it is 10000 parts, and s3fs is specified each part size by this option.
So that, the file size limitation is (10000 * multipart_size).
(If disk space is low it is also another restriction, but the standard will in the above size.)

https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/fdcache.cpp#L1353

Please try to set multipart_size option.
Thanks in advance for your help.

We kept this issue open for a long time.
I will close this, but if the problem persists, please reopen or post a new issue.

Hi ggtakec,
I have problem uploading 230GB file to ceph s3 server,
I saw it was caching locally,
but when copy was donw, the file size in s3 became 0
and got below error:
fail to close: operation not supported

below is how I mount my ceph s3:
s3fs testing /mnt/testing -o passwd_file=/testing/passwd-s3fs -o url=http://192.168.0.100:7480 -o use_path_request_style -o dbglevel=dbg -f -o use_cache="/nova/tmp" -o curldbg

May I know if there is any configuration mistakes?

I have no problem when file size is small, says 20GB.

@alphainets I am seeing the same issue with a 100GB, and 432GB files. Did you resolve this issue?
I am using 1.8.7, on a Ubuntu 18.04 server.

s3fs 1.87 and earlier require temporary space equal to the object size. Please test with the latest master which includes a large file optimization that reduces temporary space usage. If this symptom persists, please run with -f -d and open a new issue.