Bug 42506 - JMeter threads all use the same SSL session
JMeter threads all use the same SSL session
Status: RESOLVED FIXED
Product: JMeter
Classification: Unclassified
Component: HTTP
2.2
All All
: P2 major (vote)
: ---
Assigned To: JMeter issues mailing list
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2007-05-24 00:57 UTC by Anatoli Tubman
Modified: 2007-06-02 06:59 UTC (History)
0 users



Attachments
Patch (per thread SSL context) - take 1 (14.61 KB, patch)
2007-05-27 16:33 UTC, Oleg Kalnichevski
Details | Diff
Patch (per thread SSL context) - take 1 + SSL session ID debug logging (17.01 KB, patch)
2007-05-31 02:21 UTC, Oleg Kalnichevski
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Anatoli Tubman 2007-05-24 00:57:26 UTC
When issuing HTTPS requests, JMeter threads all use the same SSL session. This
prevents JMeter from testing real-world performance of SSL sites. (One can look
at SSL session IDs in the Apache httpd custom access log with %{SSL_SESSION_ID}x
format specifier).

This means that JMeter only does the SSL handshake once with the server in the
beginning of the test, not at the beginning of each thread. The handshake is the
expensive part of SSL protocol, and bypassing it can seriously skew the
performance figures. I've done measurements with other tools that do SSL
handshake properly, and in some cases the performance difference is 4-fold.
Comment 1 Sebb 2007-05-24 04:21:08 UTC
Which HTTP sampler are you using? The default, or HttpClient?

The default Java sampler offers no control over connection re-use.
Comment 2 Anatoli Tubman 2007-05-24 05:28:24 UTC
I'm using HttpClient (from the menu: Add/Sampler/HTTP Request HTTPClient, I
think it's the right non-default one).
Comment 3 Oleg Kalnichevski 2007-05-26 05:16:59 UTC
JSSE (and hence HttpClient) uses SSL session caching per default as this results
in a significant performance improvement. Cached SSL sessions can be re-used
when opening new SSL sockets. So, just disabling connection re-use will not be
enough. 

One can plug in to HttpClient a custom SSL socket factory in order to exercise a
greater control over SSL session re-use. I could contribute one if necessary.

Oleg
Comment 4 Anatoli Tubman 2007-05-27 00:40:29 UTC
>One can plug in to HttpClient a custom SSL socket factory in order to exercise a
>greater control over SSL session re-use. I could contribute one if necessary.

It would be great, as it's impossible to benchmark SSL servers without such control.
Comment 5 Sebb 2007-05-27 06:31:49 UTC
What control are you expecting to see?

Would it be enough to use a separate session for each test thread?

Or is there any need to use multiple sessions in the same thread for the same 
host? 

Should the session always be dropped when the connection is closed?
Comment 6 Oleg Kalnichevski 2007-05-27 09:23:33 UTC
Sebastian,

I see two ways of solving the problem, one which requires quite a bit of code
refactoring and another one that does not. The former option would involve
keeping a custom HostConfiguration object per thread / user, which entails some
changes to the internal structure of JMeter. The latter option would make use of
a ThreadLocal to store an instance of SSLContext on a per thread / user basis.
All changes would be confined to HttpSSLProtocolSocketFactory.

I personally prefer to not use ThreadLocals, unless absolutely necessary,
especially in libraries. (disclaimer: I do tend to see JMeter as a library + a
default UI), but this is your call.

Oleg 
Comment 7 Sebb 2007-05-27 09:31:51 UTC
JMeter already uses ThreadLocal in a few places (and its performance impact is 
much reduced in recent JVMs), so I don't see that as a negative.

I think the best would be to use ThreadLocal, at least initially.

If necessary this can be replaced later.
Comment 8 Oleg Kalnichevski 2007-05-27 16:33:23 UTC
Created attachment 20279 [details]
Patch (per thread SSL context) - take 1

Sebastian
Here's the first take at solving the problem. The fix is sub-optimal in my
opinion. The patch simply causes a new instance of SSLContext to be created on
a per thread of execution basis. This should also result in a different SSL
session cache per thread / user. 

I also tried to clean things up a little in JsseSSLManager class.

Please review.

Oleg
Comment 9 Anatoli Tubman 2007-05-29 02:42:34 UTC
> What control are you expecting to see?
> Would it be enough to use a separate session for each test thread?

Yes.

> Or is there any need to use multiple sessions in the same thread for the same 
> host? 

No, this is not necessary.

> Should the session always be dropped when the connection is closed?

No, the session should NOT be dropped. That's the whole point of having sessions
-- that they will survive a reconnect.

It would be nice if there's an *option* to drop the session every N connections
and/or K seconds, N and K being configurable parameters, but that's advanced
stuff. (Older Microsoft systems used to drop SSL sessions every two minutes.
This was considered a bug and fixed. But it would be nice to be able to test for
this behaviour as well.)
Comment 10 Sebb 2007-05-30 14:51:28 UTC
How about an additional checkbox on the Sampler: when enabled, the cached 
session would be cleared.

Would that be useful?
Comment 11 Anatoli Tubman 2007-05-31 00:46:53 UTC
Sebb: I'm not sure I understand. The session should be per thread. Each thread's
session will be cleared when the thread stops. Am I missing something?

Oleg: I tried to use your patch. No luck, there's still a single SSL session for
all threads. Please correct me if I'm wrong, but it seems that a separate
SSLSession object needs to be created for each thread. I'm not sure what class
is responsible for tracking SSLSession objects, there's very little I can find
in the Java docs, but it's probably not the socket factory.
Comment 12 Oleg Kalnichevski 2007-05-31 02:19:16 UTC
(In reply to comment #11)

> Oleg: I tried to use your patch. No luck, there's still a single SSL session for
> all threads. Please correct me if I'm wrong, but it seems that a separate
> SSLSession object needs to be created for each thread. 

Anatoli,

I do not know of a way to create SSLSession objects with JSSE API. One can
either invalidate an existing SSL session or set a session timeout value in the
SSL session context, but cannot not initiate a new SSL session using JSSE API

I'm not sure what class
> is responsible for tracking SSLSession objects, there's very little I can find
> in the Java docs, but it's probably not the socket factory.

SSLSessionContext is probably what you are looking for.

Anyways, I clearly see different SSL session ids per execution thread on the
client side:

2007/05/31 11:05:41 INFO  - jmeter.threads.JMeterThread: Running PostProcessors
in forward order 
2007/05/31 11:05:41 INFO  - jmeter.threads.JMeterThread: Thread Thread Group 1-1
started 
2007/05/31 11:05:41 INFO  - jmeter.util.JsseSSLManager: Using default SSL
protocol: TLS 
2007/05/31 11:05:41 INFO  - jmeter.util.SSLManager: JmeterKeyStore Location:
/home/oleg/.keystore 
2007/05/31 11:05:41 INFO  - jmeter.util.SSLManager: KeyStore created OK, Type: JKS 
2007/05/31 11:05:41 WARN  - jmeter.util.SSLManager: Keystore file not found,
loading empty keystore 
2007/05/31 11:05:42 INFO  - jmeter.threads.JMeterThread: Thread Thread Group 1-2
started 
2007/05/31 11:05:42 INFO  - jmeter.threads.JMeterThread: Thread Thread Group 1-3
started 
2007/05/31 11:05:43 INFO  - jmeter.util.HttpSSLProtocolSocketFactory: SSL
session id: 1B76A4E7198259C06AD11EB1892A9540C27C661DFC7226574020E96A12D92C60 
2007/05/31 11:05:43 INFO  - jmeter.util.HttpSSLProtocolSocketFactory: SSL
session id: AC1D816001BAB6FC94FBA0754514F13528FE7E20C7B203BBA372BC30248575A8 
2007/05/31 11:05:43 INFO  - jmeter.threads.JMeterThread: Thread Thread Group 1-3
is done 
2007/05/31 11:05:43 INFO  - jmeter.engine.StandardJMeterEngine: Ending thread
Thread Group 1-3 
2007/05/31 11:05:43 INFO  - jmeter.threads.JMeterThread: Thread Thread Group 1-2
is done 
2007/05/31 11:05:43 INFO  - jmeter.engine.StandardJMeterEngine: Ending thread
Thread Group 1-2 
2007/05/31 11:05:43 INFO  - jmeter.util.HttpSSLProtocolSocketFactory: SSL
session id: 0A7CE7BADA54C493980570B80AD804280E59C2FF5FFA7C88C71323594DEB75CD 
2007/05/31 11:05:44 INFO  - jmeter.threads.JMeterThread: Thread Thread Group 1-1
is done 
2007/05/31 11:05:44 INFO  - jmeter.engine.StandardJMeterEngine: Ending thread
Thread Group 1-1 
2007/05/31 11:05:44 INFO  - jmeter.engine.StandardJMeterEngine: Stopping test 
2007/05/31 11:05:44 INFO  - jmeter.engine.StandardJMeterEngine: Notifying test
listeners of end of test 
2007/05/31 11:05:44 INFO  - jmeter.gui.util.JMeterMenuBar: setRunning(false,local) 
2007/05/31 11:05:44 INFO  - jmeter.engine.StandardJMeterEngine: Test has ended 

Oleg

Comment 13 Oleg Kalnichevski 2007-05-31 02:21:12 UTC
Created attachment 20295 [details]
Patch (per thread SSL context) - take 1 + SSL session ID debug logging

Added SSL session ID debug logging.

Oleg
Comment 14 Anatoli Tubman 2007-05-31 03:41:43 UTC
Oleg,

Please disregard the previous message, it probably was my mistake in building
the project. JMeter indeed uses different session IDs now. Thank you.
Comment 15 Sebb 2007-05-31 15:27:12 UTC
(In reply to comment #11)
> Sebb: I'm not sure I understand. The session should be per thread. Each 
thread's
> session will be cleared when the thread stops. Am I missing something?

If there are multiple samplers in a single thread, then the idea would be to 
allow the session to be cleared before starting the current sample.
Comment 16 Sebb 2007-06-02 06:59:47 UTC
Many thanks.
The second patch has been applied to SVN in r543739.

I made some changes:
- moved the threadLocal processing to JsseSSLManager, so that now deals with 
all the SSL context manipulation.
- added a property to allow reversion to the original behaviour
- renamed some variables and methods to keep original names

Warning: the debug code added in the second patch causes some SSL sites (e.g. 
https://svn.apache.org/) to generate the error:
javax.net.ssl.SSLException: Received fatal alert: unexpected_message
This seems to be due to calling sock.getSession().