Skip to content

Instantly share code, notes, and snippets.

@chiller
Last active June 11, 2023 11:04
Show Gist options
  • Save chiller/dec373004894e9c9bb38ac647c7ccfa8 to your computer and use it in GitHub Desktop.
Save chiller/dec373004894e9c9bb38ac647c7ccfa8 to your computer and use it in GitHub Desktop.
ab benchmark file upload

Benchmarking file uploads

Based on this question with some adjustments https://stackoverflow.com/questions/20220270/posting-multipart-form-data-with-apache-bench-ab

Constructing the post payload

You'll need a text file post_data.txt with the following contents:

--1234567890
Content-Disposition: form-data; name="file"; filename="ab1_pod.jpg"
Content-Type: application/jpeg
Content-Transfer-Encoding: base64

[base64 encoding of the file]
--1234567890--

I have added the line specifying a base64 encoding that was necessary.

To base64 encode your file you could do:

base64 yourfile.png >> post_data.txt
echo "--1234567890--" >> post_data.txt

Some tutorials mention switching to a windows style line ending: unix2dos post_data.txt I have not noticed that being necessary. You can doublecheck with cat -e post_data.txt the file should end in ^M$

The ab call

Suppose you need authentication for your file upload, then you'll need to add that as a header in our case:

ab -v 4 -n 1 -c 1 -H 'Authorization: Bearer e...Y' -p post_data.txt -T "multipart/form-data; boundary=1234567890" http://localhost:9091/v1/images

Parameters

-v 4  # this will give you a lot of useful info like the response from the server, useful for debugging
-n 1  # number of requests in total
-c 1  # number of concurrent requests -n 7 -c 5 will do 7 requests in total: first 5 then 2

You probably want to start with -n 1 -c 1 and then increase the numbers. It's also a good idea to try a GET first.

@liuchengjie01
Copy link

I use this method but I met an error, " malformed MIME header line". I have not found the reason and did not know how to solve it.

@liuchengjie01
Copy link

please note that after “Content-Transfer-Encoding: base64” should be an empty line in post_data.txt. I miss this and make an error.

@Digital2Slave
Copy link

@chiller @liuchengjie01 how to post image file without base64 encode?

@qshuai
Copy link

qshuai commented Mar 17, 2023

I try serveral times and success:
generate a gzip file.

dd if=/dev/urandom of=/tmp/1m.txt bs=1M count=1
gzip 1m.txt

prepare postdata.txt for ab tool

vi postdata.txt
# add the following content. Notice: add a blank line after `Content-Type: xxx `.
--1234567890
Content-Disposition: form-data; name="logfile"; filename="1m.txt.gz"
Content-Type: application/octet-stream

write file content and boundary ending

cat 1m.txt.gz >> postdata.txt
echo -e "\n--1234567890" >> postdata.txt

Ok. let us try:

ab -n 10000 -c 100 -p postdata.txt -T "multipart/form-data; boundary=1234567890" http://x.x.x.x:8080/api/upload/syslog

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment