When issuing HTTPS requests, JMeter threads all use the same SSL session. This prevents JMeter from testing real-world performance of SSL sites. (One can look at SSL session IDs in the Apache httpd custom access log with %{SSL_SESSION_ID}x format specifier). This means that JMeter only does the SSL handshake once with the server in the beginning of the test, not at the beginning of each thread. The handshake is the expensive part of SSL protocol, and bypassing it can seriously skew the performance figures. I've done measurements with other tools that do SSL handshake properly, and in some cases the performance difference is 4-fold.
Which HTTP sampler are you using? The default, or HttpClient? The default Java sampler offers no control over connection re-use.
I'm using HttpClient (from the menu: Add/Sampler/HTTP Request HTTPClient, I think it's the right non-default one).
JSSE (and hence HttpClient) uses SSL session caching per default as this results in a significant performance improvement. Cached SSL sessions can be re-used when opening new SSL sockets. So, just disabling connection re-use will not be enough. One can plug in to HttpClient a custom SSL socket factory in order to exercise a greater control over SSL session re-use. I could contribute one if necessary. Oleg
>One can plug in to HttpClient a custom SSL socket factory in order to exercise a >greater control over SSL session re-use. I could contribute one if necessary. It would be great, as it's impossible to benchmark SSL servers without such control.
What control are you expecting to see? Would it be enough to use a separate session for each test thread? Or is there any need to use multiple sessions in the same thread for the same host? Should the session always be dropped when the connection is closed?
Sebastian, I see two ways of solving the problem, one which requires quite a bit of code refactoring and another one that does not. The former option would involve keeping a custom HostConfiguration object per thread / user, which entails some changes to the internal structure of JMeter. The latter option would make use of a ThreadLocal to store an instance of SSLContext on a per thread / user basis. All changes would be confined to HttpSSLProtocolSocketFactory. I personally prefer to not use ThreadLocals, unless absolutely necessary, especially in libraries. (disclaimer: I do tend to see JMeter as a library + a default UI), but this is your call. Oleg
JMeter already uses ThreadLocal in a few places (and its performance impact is much reduced in recent JVMs), so I don't see that as a negative. I think the best would be to use ThreadLocal, at least initially. If necessary this can be replaced later.
Created attachment 20279 [details] Patch (per thread SSL context) - take 1 Sebastian Here's the first take at solving the problem. The fix is sub-optimal in my opinion. The patch simply causes a new instance of SSLContext to be created on a per thread of execution basis. This should also result in a different SSL session cache per thread / user. I also tried to clean things up a little in JsseSSLManager class. Please review. Oleg
> What control are you expecting to see? > Would it be enough to use a separate session for each test thread? Yes. > Or is there any need to use multiple sessions in the same thread for the same > host? No, this is not necessary. > Should the session always be dropped when the connection is closed? No, the session should NOT be dropped. That's the whole point of having sessions -- that they will survive a reconnect. It would be nice if there's an *option* to drop the session every N connections and/or K seconds, N and K being configurable parameters, but that's advanced stuff. (Older Microsoft systems used to drop SSL sessions every two minutes. This was considered a bug and fixed. But it would be nice to be able to test for this behaviour as well.)
How about an additional checkbox on the Sampler: when enabled, the cached session would be cleared. Would that be useful?
Sebb: I'm not sure I understand. The session should be per thread. Each thread's session will be cleared when the thread stops. Am I missing something? Oleg: I tried to use your patch. No luck, there's still a single SSL session for all threads. Please correct me if I'm wrong, but it seems that a separate SSLSession object needs to be created for each thread. I'm not sure what class is responsible for tracking SSLSession objects, there's very little I can find in the Java docs, but it's probably not the socket factory.
(In reply to comment #11) > Oleg: I tried to use your patch. No luck, there's still a single SSL session for > all threads. Please correct me if I'm wrong, but it seems that a separate > SSLSession object needs to be created for each thread. Anatoli, I do not know of a way to create SSLSession objects with JSSE API. One can either invalidate an existing SSL session or set a session timeout value in the SSL session context, but cannot not initiate a new SSL session using JSSE API I'm not sure what class > is responsible for tracking SSLSession objects, there's very little I can find > in the Java docs, but it's probably not the socket factory. SSLSessionContext is probably what you are looking for. Anyways, I clearly see different SSL session ids per execution thread on the client side: 2007/05/31 11:05:41 INFO - jmeter.threads.JMeterThread: Running PostProcessors in forward order 2007/05/31 11:05:41 INFO - jmeter.threads.JMeterThread: Thread Thread Group 1-1 started 2007/05/31 11:05:41 INFO - jmeter.util.JsseSSLManager: Using default SSL protocol: TLS 2007/05/31 11:05:41 INFO - jmeter.util.SSLManager: JmeterKeyStore Location: /home/oleg/.keystore 2007/05/31 11:05:41 INFO - jmeter.util.SSLManager: KeyStore created OK, Type: JKS 2007/05/31 11:05:41 WARN - jmeter.util.SSLManager: Keystore file not found, loading empty keystore 2007/05/31 11:05:42 INFO - jmeter.threads.JMeterThread: Thread Thread Group 1-2 started 2007/05/31 11:05:42 INFO - jmeter.threads.JMeterThread: Thread Thread Group 1-3 started 2007/05/31 11:05:43 INFO - jmeter.util.HttpSSLProtocolSocketFactory: SSL session id: 1B76A4E7198259C06AD11EB1892A9540C27C661DFC7226574020E96A12D92C60 2007/05/31 11:05:43 INFO - jmeter.util.HttpSSLProtocolSocketFactory: SSL session id: AC1D816001BAB6FC94FBA0754514F13528FE7E20C7B203BBA372BC30248575A8 2007/05/31 11:05:43 INFO - jmeter.threads.JMeterThread: Thread Thread Group 1-3 is done 2007/05/31 11:05:43 INFO - jmeter.engine.StandardJMeterEngine: Ending thread Thread Group 1-3 2007/05/31 11:05:43 INFO - jmeter.threads.JMeterThread: Thread Thread Group 1-2 is done 2007/05/31 11:05:43 INFO - jmeter.engine.StandardJMeterEngine: Ending thread Thread Group 1-2 2007/05/31 11:05:43 INFO - jmeter.util.HttpSSLProtocolSocketFactory: SSL session id: 0A7CE7BADA54C493980570B80AD804280E59C2FF5FFA7C88C71323594DEB75CD 2007/05/31 11:05:44 INFO - jmeter.threads.JMeterThread: Thread Thread Group 1-1 is done 2007/05/31 11:05:44 INFO - jmeter.engine.StandardJMeterEngine: Ending thread Thread Group 1-1 2007/05/31 11:05:44 INFO - jmeter.engine.StandardJMeterEngine: Stopping test 2007/05/31 11:05:44 INFO - jmeter.engine.StandardJMeterEngine: Notifying test listeners of end of test 2007/05/31 11:05:44 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(false,local) 2007/05/31 11:05:44 INFO - jmeter.engine.StandardJMeterEngine: Test has ended Oleg
Created attachment 20295 [details] Patch (per thread SSL context) - take 1 + SSL session ID debug logging Added SSL session ID debug logging. Oleg
Oleg, Please disregard the previous message, it probably was my mistake in building the project. JMeter indeed uses different session IDs now. Thank you.
(In reply to comment #11) > Sebb: I'm not sure I understand. The session should be per thread. Each thread's > session will be cleared when the thread stops. Am I missing something? If there are multiple samplers in a single thread, then the idea would be to allow the session to be cleared before starting the current sample.
Many thanks. The second patch has been applied to SVN in r543739. I made some changes: - moved the threadLocal processing to JsseSSLManager, so that now deals with all the SSL context manipulation. - added a property to allow reversion to the original behaviour - renamed some variables and methods to keep original names Warning: the debug code added in the second patch causes some SSL sites (e.g. https://svn.apache.org/) to generate the error: javax.net.ssl.SSLException: Received fatal alert: unexpected_message This seems to be due to calling sock.getSession().
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/1946