Created attachment 27940 [details] Test script that exhibits the performance difference between JM 2.5.1 and JM 2.4 I have a test case that works under JMeter 2.4 just fine. But when I run the exact same test under JMeter 2.5.1, it is consistently slower. Specifically, the average response times increase for POST responses, and the throughput drops from ~128.1 under 2.4 to ~79.5 under 2.5.1 with everything else remaining identical (except the actual start time of the test). There is some variability in the test, but rarely more than a couple of percent - so the difference between JM2.4 and JM2.5.1 is rather disconcerting. I have attached a .zip package that contains the test script. The script performs 2 HTTP requests: 1) simple GET from our site's homepage 2) POST to our site's "login" form (with 3 values: username, password, stk (a token from a COOKIE)) In the test, I normally use 'https' for the test, but the same behavior is exhibited using just HTTP. I do not download embedded resources. I use Keepalive, and follow redirects. I tried changing the HTTP Client in JM 2.5.1 to use the new HTTP4 client, but it has a cookie issue, so I couldn't complete the test correctly. (Using HTTP3.1 is where I see the performance issue.)
Created attachment 27941 [details] JMeter2.4 test run screen image Attaching an image of the Summary Report from the JM 2.4 test run on my system
Created attachment 27942 [details] JMeter2.5.1 test run screen image Attaching an image of the JM 2.5.1 summary report from a test run on my system
Created attachment 27943 [details] JMeter2.5.1r1202266 test run screen image using HTTPClient4 Added image showing the use of HTTPClient4 (using the nightly build r1202266 - since it has the fix for the cookie issue with HTTP4).
Re-ran the test using JM 2.5.1 r1202266 (nightly build) - and HTTPClient4 (needed the cookie fix). With HTTP4, it runs even slower than before. Then I tried the r1202266 build with HTTPclient3.1, and it performs nearly identically (within the margin of error) to the 2.5.1 "release" version. Uploaded another screen snap of the summary report from the HTTP4 test run.
Created attachment 27944 [details] Sample Listener
Thanks for the test case. Unfortunately we cannot use it as it requires access to a private server as far as I can tell, but it does at least show what you are running. Not sure why you use an include controller; would it be possible to eliminate that as a possibility? It's easy to do - just select the Include Controller, and right-click Merge. Can you also provide JTL files for a short run (say 50 samples)? CSV format would be best. See attached Listener for a suitable config.
I'm using the Include controller because I use the 'login' with a bunch of different test procedures. By using the Include Controller, I can manage changes to the Login.jmx file separately, and all my test procedures that use it will inherit the changes. But for the sake of simplicity, I've re-run the tests with the Login process merged into the main script. The results are essentially the same (JM2.4 shows throughput of 145/sec, avg resp times of 655; JM2.5.1r1202266 shows throughput of 81/sec, avg resp times of 880. I'll be including the JTL file and the updated "LoginTest.jmx" file as attachments. The JTL file is from the JM2.5.1 + the nightly build (from yesterday) that I setup previously. I set it to run 5 threads, with 10 loops.
Created attachment 27949 [details] JTL output of 5 threads * 10 iterations, no "include" controller used This is the requested JTL file for a test run using JM2.5.1 r1202266 - with 5 threads, and 10 iterations. The test used HttpClient4. Also, the test script was modified to eliminate the 'Include' controller (the "Login" was merged with the main test script - to simplify the test).
Created attachment 27950 [details] Updated test script - without the Include controller. Removed the "Include" controller by merging the necessary Login script with the main script. NOTE: this script still requires the "T1.csv" file - attached separately.
Created attachment 27951 [details] Configuration data file for LoginTest.jmx test script. Configuration file for LoginTest.jmx test script.
(In reply to comment #8) > Created attachment 27949 [details] > JTL output of 5 threads * 10 iterations, no "include" controller used > > This is the requested JTL file for a test run using JM2.5.1 r1202266 - with 5 > threads, and 10 iterations. The test used HttpClient4. Also, the test script > was modified to eliminate the 'Include' controller (the "Login" was merged with > the main test script - to simplify the test). Can you also provide the JTL for the same run with JMeter 2.4?
Created attachment 27954 [details] CSV Output using Sebb's Simple Data Writer controller Sorry, I missed that you needed 'csv' output... Here's the output of the JM2.5.1 r1202266 test with 5 threads * 10 iterations.
Created attachment 27956 [details] CSV Output using Sebb's Simple Data Writer controller - for JM2.4 Re-ran the test on JM2.4, using the simple data writer. The only difference in this test is that the "HttpClient" can't be set in JM2.4...
Thanks. In fact the XML output would also be useful, as there are some redirects that generate additional samples which do not show up in the CSV output (only top-level samples are recorded).
Created attachment 27958 [details] JTL output of 5 threads * 10 iterations, no "include" controller used - for JMeter 2.4 Added JTL (XML) output for test run using JM2.4.
Thanks for all the logs etc. It would also be helpful to have HC3 JTL data for JMeter 2.5.1. AIUI the 2.5.1 data file was created using HC4, whereas for JMeter 2.4 HC4 is not available, so I assume you used HC3? In addition (if you have the time), HTTP Java runs for JM 2.4 and 2.5.1 would be useful.
Created attachment 27964 [details] JMeter2.5.1r1202266 test run screen image using HTTPClient3.1 Ran a new test with the JMeter 2.5.1 using HttpClient3.1
I'm not sure how to switch to the HTTP Java client in JMeter 2.4... I'll run the test with the Java client for 2.5.1r1202266. But I need some directions on how to use the HTTP Request Java Client for JM2.4.
Created attachment 27965 [details] JMeter2.5.1r1202266 test run JTL using JavaClient Re-ran the test using JM2.5.1 r1202266 - but with the HTTP Request Java Client instead of HTTP3.1 or HTTP4.
Created attachment 27966 [details] JMeter2.5.1r1202266 test run JTL using HttpClient3.1 I mis-labeled the JM2.5.1 r120066 JTL file as a screen image... This is just to fix that mis-label. Attached the correct file as a JTL file for the JM2.5.1 r1202266 w/HTTPClient3.1.
I profiled the 2 versions and found the following elements: There is a slight degradation in response time between 2 versions when using Java Impl which is due to at least these 2 changes: - There is a configuration difference between both related to the addition of:sun.net.http.allowRestrictedHeaders=true in system.properties, this impacts response time of connection.getContentLength - There is also the addition of CountingInputStream which is slightly involved in the degradation - There are still differences , reverting gives me now 2% difference. There is also in 2.5.2 a real difference between the 3 HTTP Sampler implementations and clearly performances tops are: - Java impl - HTTP HC 3.1 Impl - HTTP HC 4 Impl (less performing) I didn't have time yet to understand if it is due to JMeter usage or to HTTPClient impl.
Based on our internal micro-benchmark Apache HttpClient 4.x is comfortably faster than both the internal JRE HTTP client or Apache HttpClient 3.1 http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore You are very welcome to run the benchmark locally and let me know whether or not you get consistent results. Oleg
Thanks Oleg for your comment. I will try to give you some feedback as soon as possible. If you have some time, it would be great if you could review usage of HTTPClient in JMeter and make some improvements propositions. Regards Philippe
From Rainer Jung on dev list: I did a simple test using a very small file (2 bytes) to mostly check per request overhead. I let it run with 10 threads for a total of 200.000 samples and only took the last 20.000 samples to calculate results. Configuration was default, JVM was 1.6.0_29, System was Solaris Sparc with 2 CPUs for JMeter and Apache on a separate one CPU system. CPU was not saturated, bandwidth neither. Those tests showed: - results for HttpClient3.1 and HttpClient4 are about the same - results for JMeter 2.4, 2.5.1 and 2.5.2-dev are about the same - response times measured with HttpClient are between 52% and 59% of the old Java Sampler - wallclock time needed for the 20.000 samples was only 0.3% to 2.2% bigger than the sum of the response times, so overhead is minimal - overhead, though minimal was about 2% for HttpClient and about 0.5 for the old Java sampler. Overall it is a big difference, but both numbers are pretty small. - since overhead is small, throughput in requests per second behaves roughly like average response time, namely about 740 requests per second for HttpClient and about 400-440 for the old Java sampler. So throughput is about 70% better for the newer samplers. - CPU was higher for HttpClient, but only about 50-60%, so relative to throughput (per request) it was a bit lower. "about the same" means differences were smaller than variability of test runs, always less than 10%. It could be, that the test results will be very different, for bigger response sizes, KeepAlive turned off, real live tests with cookies etc. etc. At least the base line looks good and I don't see a relevant difference between 2.4 and 2.5.x. Regards, Rainer
I investigated this issue a little more by tuning my local Tomcat server and doing the following comparisons: - Java Impl in 2.4: 430,4 hits/s - Java Impl in Current Trunk version (> 2.5.1): 431,8 hits/s - HC3.1 Impl in 2.4: 399,3 hits/s - HC3.1 Impl in Current Trunk version (> 2.5.1): 397,4 hits/s So no significant degradations at Sampler level. I close issue as investigations don't reproduce what has been reported.
This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/2649