Bug 41837 - Parameter value corruption
Parameter value corruption
Status: RESOLVED FIXED
Product: JMeter
Classification: Unclassified
Component: HTTP
2.2
All All
: P4 major (vote)
: ---
Assigned To: JMeter issues mailing list
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2007-03-14 07:24 UTC by olivier.schmitt
Modified: 2007-04-18 12:53 UTC (History)
0 users



Attachments
Contains 2 JSPs for testing purpose (800 bytes, application/octet-stream)
2007-03-20 03:06 UTC, olivier.schmitt
Details
Test plan recorded by the proxy with multipart encoding (9.93 KB, text/plain)
2007-03-20 03:08 UTC, olivier.schmitt
Details
Suggested patch (806 bytes, patch)
2007-03-20 11:44 UTC, Alf Hogemark
Details | Diff
Updated web app for testing multipart/form-data (119.81 KB, application/octet-stream)
2007-03-20 11:47 UTC, Alf Hogemark
Details
Updated patch, which also updates content-type header (3.12 KB, patch)
2007-03-20 12:52 UTC, Alf Hogemark
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description olivier.schmitt 2007-03-14 07:24:03 UTC
I try to record a test plan with the proxy server.
My web app uses JSF 1.1.
The view state is saved in HTML in hidden input control.
A view state is a tree of JSF component state saved as a gzipped base64 string :

Excerpt from ResponseStateManagerImpl of jsf-impl-1.1.jar
...
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
        if (compress) {
            zos = new GZIPOutputStream(bos);
            oos = new ObjectOutputStream(zos);
        } else {
            oos = new ObjectOutputStream(bos);
        }
        oos.writeObject(view.getStructure());
        oos.writeObject(view.getState());
        oos.close();
        if (compress) {
            zos.close();
        }
        bos.close();

        hiddenField = " <input type=\"hidden\" name=\""
            + RIConstants.FACES_VIEW + "\"" + " value=\"" +
            (new String(Base64.encode(bos.toByteArray()), "ISO-8859-1")) +
            "\" />\n ";

...

sample :

H4sIAAAAAAAAAO19CXgcxZV/ayRZh2UsyZLlS/YQ29gGPDrm0OGQIMuyLVY+sM3...

When the current page is submitted by the browser, the proxy update the string
into :

H4sIAAAAAAAAAO19CXgcxZV/ayRZh2UsyZLlS/YQ29gGPDrm0OGQIMuyLVY sM3...

the "+" char is missing at the end of the string !!!

JSF can not deserialized the string into objects : it crashes.
(UTFDataFormatException, ZipException, ...)

I've remotly debugged jmeter to check when the value is altered.

When Proxy class record an HTTP request every parameter of the request is 
always decoded.

The HTTPArgument class is used to represent a request parameter.

    public HTTPArgument(String name, String value, boolean alreadyEncoded) {
        setAlwaysEncoded(true);
        if (alreadyEncoded) {
            try {
                name = JOrphanUtils.decode(name, "UTF-8");
                value = JOrphanUtils.decode(value, "UTF-8");
            } catch (UnsupportedEncodingException e) {
                // UTF-8 unsupported? You must be joking!
                log.error("UTF-8 encoding not supported!");
                throw new Error(e.toString());
            }
        }
        setName(name);
        setValue(value);
        setMetaData("=");
    }

At least every parameter value is decoded with  URLDecoder.decode().
My parameter value is corrupted because of this call :

URLDecoder.decode("H4sIAAAAAAAAAO19CXgcxZVwz+iWZcuSJdv4HDDGds", "UTF-8")

returns H4sIAAAAAAAAAO19CXgcxZVwz iWZcuSJdv4HDDGds
Comment 1 Alf Hogemark 2007-03-15 02:06:46 UTC
(In reply to comment #0)

Are you using a POST or GET request to the web server ?

From you description, it sounds like the problem is somewhere in the data flow
from the web browser via the proxy to the Jmeter test plan. And not in the data
flow from executing the test plan and sending request to the web server.
Do you agree with that ?

I've seen from the open bugs that there seems to be multiple problem with the
proxy and handling of enconding, and I think your problem might be related.

Comment 2 olivier.schmitt 2007-03-16 09:00:47 UTC
(In reply to comment #1)
> (In reply to comment #0)
> 
> Are you using a POST or GET request to the web server ?

POST.

> From you description, it sounds like the problem is somewhere in the data flow
> from the web browser via the proxy to the Jmeter test plan. And not in the data
> flow from executing the test plan and sending request to the web server.
> Do you agree with that ?

yes.

> 
> I've seen from the open bugs that there seems to be multiple problem with the
> proxy and handling of enconding, and I think your problem might be related.
> 

I've not found any but some special characters encoding problems (Japanese, ...).

You can get a more complete description of my problem here :
http://mail-archives.apache.org/mod_mbox/jakarta-jmeter-user/200703.mbox/browser
(thread is "Parameter value corruption" )
Comment 3 Alf Hogemark 2007-03-19 11:35:22 UTC
I've looked into this now, but haven't gotten much further.

If you can point us to a public web server where we can test, that would help a
lot. Perhaps you know of some public sites which has demos of applications using
JSF, where you and we could recreate the problem.

If not, I suggest you try the following :
Use the proxy to record the test plan. Then execute the test plan, and see which
sample fails. Then I suggest you fix the value of that Request parameter by hand
in the GUI, ie change the space to a +, and then re-execute the test to see if
it works. That should work. If it still does not work, then I'm not sure if it
is a Proxy decoding problem.

Alternative, you could add a method in the
org.apache.jmeter.protocol.http.util.TestHTTPArgument (the unit test for
HTTPArgument), and write a unit test for the encoding and decoding of your
problematic values, and supply that code, and we could have a look at it.
Something like :
public void testDecoding() throws Exception {
  // Test to try to recreate problem in bug 41837
  // This is how the MultipartUrlConfig.addEncodedArgument method
  // creates and HTTPArgument for a parameter value that is 
  // received as part of a POST request
  HTTPArgument arg = new HTTPArgument("test_for_bug_41837",
"H4sIAAAAAAAAAO19CXgcxZV/ayRZh2UsyZLlS/YQ29gGPDrm0OGQIMuyLVY+sM3", true);
  if (arg.getName().equals(arg.getEncodedName()) &&
arg.getValue().equals(arg.getEncodedValue())) {
    arg.setAlwaysEncoded(false);
}
  // The value must be encoded before it is sent to web server
  assertTrue(arg.isAlwaysEncoded());
}

I've run that test code, and it seems to behave correct to me.

Spaces as encoded as + in an URL, and then when the Proxy gets the value, it
decodes the value, and put it into the HTTPSampler that is put into the test
plan, because we prefer to show unencoded values in the GUI.

What is the "Encode?" property value for the request parameter in the HTTP
Request node that is put into the test plan ?
If "Encode?" is true, then I don't think it is a problem that the "+" is shown
as a " " in the GUI, because when the test is run again, it will be encoded into
"+".
Perhaps you could attach a test plan created by the proxy, which shows the problem ?
Or you could attach a screengrab of the problematic HTTP Request in the test plan ?

It would be good if you could run with the latest nightly build of jmeter.

What is the "content-type" and "charset" for the web pages you are serving ?
If you are using UTF-8, perhaps you could try to add "UTF-8" as the value of the
"Content encoding" in the HTTP Request, you need the latest nightly build to
have that field. I think you should try that first. That functionality was added
as part of bug 41705. So if your application is expecting UTF8 data, but you
have not set the "UTF-8" "Content encoding" on the HTTP Request, then you might
get problems.

I'll look more into this issue, if you can provide us with information so we get
a few steps closer to solving this.
Comment 4 olivier.schmitt 2007-03-20 03:06:23 UTC
Created attachment 19750 [details]
Contains 2 JSPs for testing purpose

testviewid.jsp : contains 2 form with 2 different encodings.
result.jsp: display the value of the submitted viewid
When recording with JMeter proxy, the value encoded in first form is not
corrupted while the same value encoded in second form is.
Comment 5 olivier.schmitt 2007-03-20 03:08:34 UTC
Created attachment 19751 [details]
Test plan recorded by the proxy with multipart encoding

The test plan shows the corrupted value '+'  replaced by ' ' for the
test_for_bug_41837 value.
Comment 6 olivier.schmitt 2007-03-20 03:13:04 UTC
Thanks for you long analysis.
I've news !
The form encoding seems to be the cause of the corruption.
You will find two useful attachments for testing purpose.
When using "application/x-www-form-urlencoded" and post method, everything is fine.

When using "multipart/form-data" and post method, the value is corrupted.

As you can see with result.jsp, the proxy corrupts the value and then send the
value to tomcat server. 
Comment 7 Alf Hogemark 2007-03-20 11:38:34 UTC
Thanks for your info, I think we have solved this problem now.

Analysis :
Your jsp pages was helpful in analyzing this. I've added a bit of functionality
to  the results.jps page. I've added use of the commons-io.jar and
commons-fileupload.ja, so that I can see what is posted when doing a
"multipart/from-data" post. I'll attach the updated war file.

First of all, if you are testing pages which are using multipart/form-data, then
you should turn of the "Capture HTTP Headers" setting in the Proxy Server. 
See bug 27780. Bug 27780 also explains that when running the test plan after you
have captured it, jmeter will issue a "application/x-www-form-urlencoded"
request even though the proxy server received a multipart/form-data.
But if you are a bit lucky, that should does not cause problems for you.

I think the bug is in the MultipartUrlConfig class, which is handling the
parsing of a multipart/form-data post request for the proxy server.
If you are in a multipart/form-data post request, then the parameter values
which are part of the multipart is not encoded by the browser.
But the MultipartUrlConfig does the following :
"
} else if (parts[i].indexOf("name=") > -1) {
  int index = parts[i].indexOf("name=\"") + 6;
  String name = parts[i].substring(index, parts[i].indexOf("\"", index));
  index = parts[i].indexOf("\n", index) + 2;
  String value = parts[i].substring(index).trim();
  this.addEncodedArgument(name, value);
}
"
So the MultipartUrlConfig assumes that parameter values which are part of a
multipart/form-data has been URL encoded. I think that is the wrong assumption.
I think it should do a "this.addArgument(name, value);", to add a parameter
value which has not been URL encoded by the browser.
I'll attach a patch to fix this.

Read more on http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4
And perhaps read section 6 og http://www.ietf.org/rfc/rfc2045.txt
So the multipart/form-data can be encoded by specifying the
"Content-Transfer-Encoding" in the multipart. But let us say that for now, the
Proxy does not support that.

It has been useful to do this investigation. I was already planning to fix bugs
27780, 33435, 41518 (which I think is the same as 27780), and 19128.
I hope to have 27780 fixed within a couple of weeks.

However, what I plan to do first, is to reorganize the "HTTP Request" GUI
slightly, because I need to add an "HTTP POST" parameters section, so that the
user can choose if he wants to do a "x-www-form-urlencoded" or a "multipart"
post if only parameters are present. There also needs to be room in the GUI to
add more files.
I'm thinking about using tabs in the lower part of the GUI, one tab names
"Options", one tab named "Parameters", one tab named "File uploads".

That was all for now.
It would be good if you could test the patch I'll add shortly, and see if things
works then. And I guess Sebb will also look at the patch, and apply it if he agrees.
Comment 8 Alf Hogemark 2007-03-20 11:44:15 UTC
Created attachment 19758 [details]
Suggested patch

This patch makes parameter values which are part of a multipart/form-data post
request treated as not url encoded, in fact not encoded at all.
We assume the "default" "Content-Transfer-Encoding" has been used for the
multipart/form-data. Support for different "Content-Transfer-Encoding" might
come in the future.
Comment 9 Alf Hogemark 2007-03-20 11:47:24 UTC
Created attachment 19759 [details]
Updated web app for testing multipart/form-data

Added a bit more code to the jsp, and included commons-io.jar and
commons-fileupload.jar
Comment 10 Alf Hogemark 2007-03-20 12:13:06 UTC
My updated test app is available at :
http://www.kanonbra.com/testjmeter/testviewid.jsp

at least for a couple of weeks.

I testet recording a test plan using the patched proxy server now. I also
attached  a "View Results Tree" listener to the proxy server.
I see that the browser/proxy has trouble sending the "multipart/form-data".
I think this is because the "http headers" sent to the web server says
"Content-Type: multipart/form-data;", while in fact the proxy is currently
always sending HTTP Post  "Content-Type: application/x-www-form-urlencoded"
request. I'll look more into this now, because I guess you are not able to
record much of a plan without having that fixed.

But the recorded test plan is possible to play back correctly.
Comment 11 Alf Hogemark 2007-03-20 12:52:42 UTC
Created attachment 19760 [details]
Updated patch, which also updates content-type header

This updated patch also changes the "Content-Type" http header receives from
web browser to "application/x-www-form-urlencoded" in the case where it is a
multipart/form-data POST, and no file is being uploaded.
With this patch, I'm able to record a test plan, and the browser shows the
correct values.

I'm not sure if this is the nicest patch, but I think it should be sufficient
for you to record your plan, and test using jmeter, assuming that you web
application allows "application/x-www-form-urlencoded".

So this patch translates a multipart/form-data HTTP Post request into a HTTP
Post "application/x-www-form-urlencoded" request, if no file is uploaded as
part of the post request.
Comment 12 Sebb 2007-03-21 16:53:09 UTC
Thanks for the patch, which I have applied to SVN.

It will be in the nightly builds after 0
Comment 13 Alf Hogemark 2007-04-18 05:26:21 UTC
I think this bug should be closed as "Fixed".
The HTTP Proxy server no longer changes values that contains "+" or other values
that change when URL encoded, when it is a multipart/form-data request.

The HTTP Proxy server has also been enhanced to support multipart/form-data even
if files are not being uploaded, so I think all in this bug is fixed.