Bug 27780 - create multipart/form-data HTTP request without uploading file
create multipart/form-data HTTP request without uploading file
Status: RESOLVED FIXED
Product: JMeter
Classification: Unclassified
Component: HTTP
1.9.1
Other other
: P3 enhancement with 3 votes (vote)
: ---
Assigned To: JMeter issues mailing list
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2004-03-18 16:16 UTC by Thomas Fischer
Modified: 2007-06-28 13:42 UTC (History)
0 users



Attachments
Suggested patch (45.96 KB, patch)
2007-03-24 11:56 UTC, Alf Hogemark
Details | Diff
Patch for unit test for svn version of PostWriter (36.03 KB, patch)
2007-03-27 06:29 UTC, Alf Hogemark
Details | Diff
Unit test for the suggested updated PostWriter / HTTPSampler in patch above (37.36 KB, patch)
2007-03-27 06:36 UTC, Alf Hogemark
Details | Diff
Patch for unit test for current svn version of PostWriter (59.38 KB, patch)
2007-03-28 05:55 UTC, Alf Hogemark
Details | Diff
Updated patch for unit testing for new suggested PostWriter/HTTPSampler/2 (60.54 KB, patch)
2007-03-28 05:58 UTC, Alf Hogemark
Details | Diff
Updated patch for unit testing of current svn version of PostWriter/HTTPSampler/2 (65.97 KB, patch)
2007-03-30 04:21 UTC, Alf Hogemark
Details | Diff
Updated patch for unit testing for new suggested PostWriter/HTTPSampler/2 (67.15 KB, patch)
2007-03-30 04:24 UTC, Alf Hogemark
Details | Diff
Updated patch for unit testing of current svn version of PostWriter/HTTPSampler/2 (67.65 KB, patch)
2007-04-01 06:48 UTC, Alf Hogemark
Details | Diff
Fix bug in PostWriter2 for content transfer encoding, and add unit tests (28.96 KB, patch)
2007-04-02 12:36 UTC, Alf Hogemark
Details | Diff
Suggested patch to allow user to choose to use multipart/form-data for post, as well as HTTP Proxy server (7.71 KB, patch)
2007-04-12 06:15 UTC, Alf Hogemark
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Fischer 2004-03-18 16:16:26 UTC
Hi,

Sometimes a web based application expects to receive a request with content 
type multipart/form-data and does not accept a simple post request. A browser 
creates a multipart/form-data request even if no file is uploaded, if the user 
leaves the upload-field empty. However, with jmeter it is not possible to 
create a multipart/form-data request without uploading a file, jmeter just 
sends a simple post form. It would be nice if there was a checkbox in the GUI 
to create a multipart/form-data request even without uploading a file.

Also, while recording a workflow with a multipart/form-data request without 
uploaded file with the jmeter proxy, the proxy resends it as a simple request. 
It would be nice if the content-type of the original request would be preserved 
in the call by the proxy. Even more problematic is that the HTTP header still 
proclaims the sent request to have the content type multipart/form-data, even 
if the body does not contain the multipart boundary.

Then, if a file is choosen which is not present in the directory where jmeter 
was started, the Jmeter proxy displays a FileNotFoundException in the Browser, 
although the proxy is located on the same host as the browser and the full path 
was given when selecting the file. To prevent this error, I had to put a file 
with the same name in the directory where jmeter was started from. If possible, 
it would be nice if the jmeter proxy would use the full path when re-sending 
the request to the server. (I don't know enough about proxies: probably the 
image is already contained in the request sent by the browser, so it is not 
necessary to access the file system again ???)

Cheers, Thomas
Comment 1 SHIMIZU Takuya 2004-10-31 13:10:14 UTC
I notice a problem in org.apache.jmeter.protocol.http.sampler.PostWriter at line 
137 in setHeaders.(line 98 at version 2.0.1)

The filename should be acquired from getFilename().

The filename should look like :
String filename = sampler.getFilename();
and not :
String filename = sampler.getFileField();

Thanks,

Takuya
Comment 2 Alf Hogemark 2007-03-24 11:56:33 UTC
Created attachment 19792 [details]
Suggested patch

This is a sugggested patch for improving the HTTP POST request functionality.

This patch adds support for using multipart/form-data even if no file is being
uploaded. I have not currently added a property in the jmeter.properties file
to control that you want multipart/form-data. In the start, I want to let the
Proxy Server create "HTTP Samplers" that are using multipart/form-data.

More headers are set both on the request, and in the multiparts, for both
HTTPSampler and HTTPSampler2. That means that both content-type, and
content-length are now set.

Also the encoding used are set for the multiparts.
I think this means that different encodings are now supported for POSTs. You
have to set the "content encoding" in the "HTTP Request" gui, to have the POST
sent with the correct encoding. This applies for both x-www-form-urlencoded and
multipart/form-data. The "HTTPSamplerBase.getQyeryString" could not be used,
since that always encodes to UTF-8, also the parameter values.

Both the HTTPSampler and HTTPSampler2 is changed, so that the "querystring"
which is set on the created sample, more fully represents what is being sent to
the web server. For multipart/form-data, that means that the multiparts are
displayed, except the actual file content.

It changes PostWriter, by changing methods to non-static, i.e. requiring that
PostWriter is instantiated. This is needed to keep state set in "setHeaders",
and reuse that state when doing "sendPostHeaders".

What I currently do not like about the patch :
HTTPSampler and HTTPSampler2 are using different "default encoding" if none is
specified in the "HTTP Request" GUI. HTTPSampler uses iso-8859-1, while
HTTPSampler2 uses us-ascii. I think both should use the same default encoding.
I think the default should be iso-8859-1, I think that is what most browsers
use by default.

The HTTPSampler currently provides the full path to the file in the header of
the file multipart. I think it is correct to only include the filename. That is
also what the HTTPSampler2 does.

I am not 100% confident that I am handling the "httpargument.isalwaysencoded"
correctly. As of now, I ignore it, and always encode the parameter name and
value. I think the "isalwaysencoded" only applies to parameters that should be
used in a HTTP GET request. But I might very well be wrong.

Conclusion :
I think this patch has potential for solving a number of issues related to HTTP
POST handling. But please have a review of the code, and ask and comment about
the code. I'll also look into making some automated tests for HTTP POST
handling.
Comment 3 Alf Hogemark 2007-03-27 06:29:07 UTC
Created attachment 19814 [details]
Patch for unit test for svn version of PostWriter

This adds unit tests for the PostWriter that is currently in SVN.
The code lines with // at the very start of the line, will fail in this
version, if uncommented.
The whole "testSendFormData_Multipart" will also fail, since there is no way to
tell HTTPSampler to write a multipart/form-data request, if you are not
uploading files.

The "testSendPostData_FileAsBody" will also fail for the last test, because the
current PostWriter writes the file content to the body, if both a filename and
some form data are present in the HTTPSampler.
Comment 4 Alf Hogemark 2007-03-27 06:36:40 UTC
Created attachment 19815 [details]
Unit test for the suggested updated PostWriter / HTTPSampler in patch above

This is unit tests for the suggested updated PostWriter / HTTPSampler, the
patch for that is in this same bugzilla issue.

This unit tests passes all tests.
So it shows that the content length is set correctly for all requests.
And it shows that multipart/form-data works even if you have no files to
upload.

And it shows that posting of UTF-8 values as part of multipart/form-data works,
which is not working in the current PostWriter.
The lines 113 and 114 in the unit test for the current PostWriter looks like
this (i.e. look in the patch for unit test for current PostWriter)
//	  checkContentLength(connection, expectedFormBody.length);	  
//	  checkArraysHaveSameContent(expectedFormBody,
connection.getOutputStreamContent());
because it fails for current PostWriter.

And it shows that if a HTTPSampler contains both form parameters, and just a
filename, then only the form parameters will be sent, and not the file as post
body.
Comment 5 Alf Hogemark 2007-03-28 05:55:26 UTC
Created attachment 19826 [details]
Patch for unit test for current svn version of PostWriter

Patch for unit test for svn version of PostWriter.
This adds on the patch for current svn version of PostWriter, by adding test
cases for adding HTTPArguments with values which have been urlencoded by hand
adding it to the HTTPSampler.

This patch also contains a new class TestHTTPSamplersAgainstHttpMirrorServer,
which executes real samples against the HttpMirrorServer. The test does not
care about what reply one gets from the HttpMirrorServer, but it allows to test
on the request headers and content created by the HTTPSampler and HTTPSampler2.


As before, the lines starting with // at the very beginning of the lines, are
either tests which fail with current svn versions of the HTTPSampler/2 code, or
that depends on new code being in place.
Comment 6 Alf Hogemark 2007-03-28 05:58:43 UTC
Created attachment 19827 [details]
Updated patch for unit testing for new suggested PostWriter/HTTPSampler/2

This patch contains updated PostWriterTest.java for the suggested new
PostWriter and HTTPSampler and HTTPSampler2, i.e. the "main" suggested patch of
this bugzilla entry.

It also contains the TestHTTPSamplersAgainstHttpMirrorServer, where all tests
passes when the suggested patch for HTTPSampler, PostWriter and HTTPSampler2 is
in place.
Comment 7 Alf Hogemark 2007-03-30 04:21:31 UTC
Created attachment 19846 [details]
Updated patch for unit testing of current svn version of PostWriter/HTTPSampler/2

Updated, and I think final, version of unit testing of PostWriter, HTTPSampler
and HTTPSampler2.

Compared to the previous suggested patch for unit testing og current svn
versions of PostWriter, HTTPSampler and HTTPSampler2, this patch adds checking
against the reply sent from the mirror server. The HTTP Mirror server is
sending back the request headers and body we sent to the mirror server.
I have also added a test on using JMeterVariables, to replace values in form
data., i.e. use variables as part of a form parameter value.
Comment 8 Alf Hogemark 2007-03-30 04:24:03 UTC
Created attachment 19847 [details]
Updated patch for unit testing for new suggested PostWriter/HTTPSampler/2

Updated, and I think final, version of unit testing of PostWriter, HTTPSampler
and HTTPSampler2.

Compared to the previous suggested patch for unit testing of the suggested new
versions of PostWriter, HTTPSampler and HTTPSampler2, this patch adds checking
against the reply sent from the mirror server. The HTTP Mirror server is
sending back the request headers and body we sent to the mirror server.
I have also added a test on using JMeterVariables, to replace values in form
data., i.e. use variables as part of a form parameter value.
Comment 9 Sebb 2007-03-31 09:42:48 UTC
(In reply to comment #7)
> Created an attachment (id=19846) [edit]
> Updated patch for unit testing of current svn version of
> PostWriter/HTTPSampler/2
> Updated, and I think final, version of unit testing of PostWriter, 
HTTPSampler
> and HTTPSampler2.
> Compared to the previous suggested patch for unit testing og current svn
> versions of PostWriter, HTTPSampler and HTTPSampler2, this patch adds 
checking
> against the reply sent from the mirror server. The HTTP Mirror server is
> sending back the request headers and body we sent to the mirror server.
> I have also added a test on using JMeterVariables, to replace values in form
> data., i.e. use variables as part of a form parameter value.

When the above patch is applied, I get 4 test failures:
checkContentTypeUrlEncoded(PostWriterTest.java:821)
checkContentTypeMultipart(PostWriterTest.java:817)
checkContentLength(PostWriterTest.java:825)
checkPostRequestFormMultipart(TestHTTPSamplersAgainstHttpMirrorServer.java:295)
Are these expected? The code does not indicate so.
Comment 10 Alf Hogemark 2007-04-01 06:48:01 UTC
Created attachment 19859 [details]
Updated patch for unit testing of current svn version of PostWriter/HTTPSampler/2

In this updated patch for unit testing of current svn version of
PostWriter/HTTPSampler/2, I have commented out all tests that were failing
expectedly.

Since PostWriter/HTTPSampler/2 does not produce multipart/form-data requests,
if only form parameters are present, the whole method for testing that has been
commented out.

I have also commented out tests in PostWriter for testing sending parameter
which has been url decoded beforehand, and when the request is made using a
content type different from UTF-8, because that is not working in the current
PostWriter version.

So if you now apply this patch, all the unit tests should pass.
Comment 11 Sebb 2007-04-01 09:17:08 UTC
Applied patch 19859; SVN r524624.
Comment 12 Sebb 2007-04-01 15:10:00 UTC
Patch 19792 applied to SVN (r524696).
This required some fixes to the test cases for the additional headers.
Comment 13 Alf Hogemark 2007-04-02 12:36:53 UTC
Created attachment 19892 [details]
Fix bug in PostWriter2 for content transfer encoding, and add unit tests

Attached is patch for PostWriter, to fix bug where "Content-Transfer-Encoding:
8bit" was sent instead of the correct "Content-Transfer-Encoding: binary" for
the file multipart. This makes PostWriter work like HTTPSampler2.

I have also enabled more of the unit tests in
TestHTTPSamplersAgainstHttpMirrorServer, which were commented out. And I have
also added unit tests for file upload in that class, although I am not doing as
much testing on the post body sent as I wish I had.

I have also changed the ENCODING of PostWriter to "ISO-8859-1", i.e. uppercase.
I do not think it is case sensitive, but I think it is most common to use
uppercase.
See for example :
http://www.iana.org/assignments/character-sets which says "Alias: ISO-8859-1
(preferred MIME name)"

I have also made a constant HTTP_ENCODING for both PostWriterTest and
TestHTTPSamplersAgainstHttpMirrorServer.
Comment 14 Sebb 2007-04-02 14:40:40 UTC
(In reply to comment #13)
> Created an attachment (id=19892) [edit]
> Fix bug in PostWriter2 for content transfer encoding, and add unit tests
> Attached is patch for PostWriter, to fix bug where "Content-Transfer-
Encoding:
> 8bit" was sent instead of the correct "Content-Transfer-Encoding: binary" for
> the file multipart. This makes PostWriter work like HTTPSampler2.
> I have also enabled more of the unit tests in
> TestHTTPSamplersAgainstHttpMirrorServer, which were commented out. And I have
> also added unit tests for file upload in that class, although I am not doing 
as
> much testing on the post body sent as I wish I had.
> I have also changed the ENCODING of PostWriter to "ISO-8859-1", i.e. 
uppercase.
> I do not think it is case sensitive, but I think it is most common to use
> uppercase.
> See for example :
> http://www.iana.org/assignments/character-sets which says "Alias: ISO-8859-1
> (preferred MIME name)"
> I have also made a constant HTTP_ENCODING for both PostWriterTest and
> TestHTTPSamplersAgainstHttpMirrorServer.

Applied
Comment 15 Alf Hogemark 2007-04-12 06:15:21 UTC
Created attachment 19936 [details]
Suggested patch to allow user to choose to use multipart/form-data for post, as well as HTTP Proxy server

The suggested patch adds an option in the HTTP Request GUI, to allow the user
to choose wether a HTTP POST request should use multipart/form-data or
application/x-www-form-urlencoded.
If a file is uploaded, the multipart/form-data is used whatever the flag is set
to.

This patch also changes the HTTP Proxy server, so that if the proxy sees a HTTP
POST multipart/form-data request, it will insert such a request into the test
plan.

If this patch is applied, I think this bug can be closed as "fixed".
Comment 16 Sebb 2007-04-14 04:59:17 UTC
Patch 19936 applied to SVN in r528796