| Summary: | Download large files avoiding outOfMemory | ||
|---|---|---|---|
| Product: | JMeter - Now in Github | Reporter: | Roberto Contiero <contiero_roby> |
| Component: | HTTP | Assignee: | JMeter issues mailing list <issues> |
| Status: | RESOLVED FIXED | ||
| Severity: | enhancement | CC: | p.mouawad |
| Priority: | P2 | ||
| Version: | 2.9 | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Windows XP | ||
| Attachments: |
The Jmeter environment
Conditions before the read of the stream Conditions after the read of the stream Conditions before returning from the readResponse method Conditions after returning from the readResponse method |
||
(In reply to comment #0) > When java call s.getResponseData(), it get an array of byte of the size > 441MB. s.getResponseData() returns a pointer to the data array. > The write method of the FileOutputStream class allocates the same > space for writing data to a file, causing OutOfMemory. Are you sure about that? Which version of Java are you using? Do you have a stack trace? If FOS really does behave that way, the propsed fix would obviously help reduce memory requirements. Created attachment 30299 [details]
The Jmeter environment
In this screeshot you can see the JVM used during test
Created attachment 30300 [details]
Conditions before the read of the stream
Created attachment 30301 [details]
Conditions after the read of the stream
Created attachment 30302 [details]
Conditions before returning from the readResponse method
Created attachment 30303 [details]
Conditions after returning from the readResponse method
Sorry, I didn't explain myself properly. It's obvious to me that s.getResponseData() returns a pointer to the data array, but returning a pointer to entire array, pw.write() allocates that space. I have reproduced the issue and I have attached some screenshots that explain my test. The jvm.jpg is my jmeter environment. The beforeReadStream.jpg shows the memory status before read the stream (on the right of the image you can see a Java VisualVM screenshot). When the stream has been readed, in the afterReadStream.jpg you can see that allocated memory is about 500MB. The w.toByteArray() instruction (see beforeReturnFromReadStream.jpg), at the end of the readResponse method of the class HTTPSamplerBase, requires as much memory (see afterReturnFromReadStream.jpg). Then, when I reach the pw.write(s.getResponseData()); line and execute it, I obtain an OutOfMemoryError (below, the stack trace from the beginning of the test): 2013/05/19 10:53:45 INFO - jmeter.engine.StandardJMeterEngine: Listeners will be started after enabling running version 2013/05/19 10:53:45 INFO - jmeter.engine.StandardJMeterEngine: To revert to the earlier behaviour, define jmeterengine.startlistenerslater=false 2013/05/19 10:53:45 INFO - jmeter.engine.StandardJMeterEngine: Running the test! 2013/05/19 10:53:45 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(true,*local*) 2013/05/19 10:53:45 INFO - jmeter.engine.StandardJMeterEngine: Starting ThreadGroup: 1 : simone 2013/05/19 10:53:45 INFO - jmeter.engine.StandardJMeterEngine: Starting 1 threads for group simone. 2013/05/19 10:53:45 INFO - jmeter.engine.StandardJMeterEngine: Thread will stop on error 2013/05/19 10:53:45 INFO - jmeter.threads.ThreadGroup: Starting thread group number 1 threads 1 ramp-up 60 perThread 60000.0 delayedStart=false 2013/05/19 10:53:45 INFO - jmeter.threads.JMeterThread: jmeterthread.startearlier=true (see jmeter.properties) 2013/05/19 10:53:45 INFO - jmeter.threads.JMeterThread: Running PostProcessors in forward order 2013/05/19 10:53:45 INFO - jmeter.threads.ThreadGroup: Started thread group number 1 2013/05/19 10:53:45 INFO - jmeter.engine.StandardJMeterEngine: All thread groups have been started 2013/05/19 10:53:45 INFO - jmeter.threads.JMeterThread: Thread started: simone 1-1 2013/05/19 10:53:45 INFO - jmeter.protocol.http.sampler.HTTPHCAbstractImpl: Local host = dell 2013/05/19 10:53:45 INFO - jmeter.protocol.http.sampler.HTTPHC3Impl: HTTP request retry count = 1 2013/05/19 10:53:45 INFO - jmeter.util.JsseSSLManager: Using default SSL protocol: SSLv3 2013/05/19 10:53:45 INFO - jmeter.util.JsseSSLManager: SSL session context: per-thread 2013/05/19 10:53:45 INFO - jmeter.util.SSLManager: JmeterKeyStore Location: type JKS 2013/05/19 10:53:45 INFO - jmeter.util.SSLManager: KeyStore created OK 2013/05/19 10:53:45 WARN - jmeter.util.SSLManager: Keystore file not found, loading empty keystore 2013/05/19 11:03:30 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.OutOfMemoryError at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(Unknown Source) at org.apache.jmeter.reporters.ResultSaver.saveSample(ResultSaver.java:189) at org.apache.jmeter.reporters.ResultSaver.processSample(ResultSaver.java:150) at org.apache.jmeter.reporters.ResultSaver.sampleOccurred(ResultSaver.java:140) at org.apache.jmeter.threads.ListenerNotifier.notifyListeners(ListenerNotifier.java:84) at org.apache.jmeter.threads.JMeterThread.notifyListeners(JMeterThread.java:783) at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:442) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) at java.lang.Thread.run(Unknown Source) 2013/05/19 11:03:30 INFO - jmeter.threads.JMeterThread: Thread finished: simone 1-1 2013/05/19 11:03:30 INFO - jmeter.engine.StandardJMeterEngine: Notifying test listeners of end of test 2013/05/19 11:03:30 INFO - jmeter.gui.util.JMeterMenuBar: setRunning(false,*local*) Thanks for the stacktrace etc. Looks like the memory allocation must be in the native method, as that is what causes the OOME in the stacktrace. So I did some research and it seems that native code which accesses a Java array may involve copying the array (depending on the JVM): http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/design.html#wp1265 This is obviously a general issue with passing large arrays to native methods, and presumably affects many other file operations as they are likely to end up calling native methods. Need to check whether there are other parts of JMeter that can write (or possibly read) large arrays. Thank you for the tip, Sebastian. It has been my pleasure to help you. By URL: http://svn.apache.org/r1484366 Log: Download large files avoiding outOfMemory Add utility method to do chunking and use in ResultSaver Bugzilla Id: 54990 Modified: jmeter/trunk/src/core/org/apache/jmeter/reporters/ResultSaver.java jmeter/trunk/src/jorphan/org/apache/jorphan/util/JOrphanUtils.java jmeter/trunk/xdocs/changes.xml Date: Wed May 29 20:07:20 2013 New Revision: 1487627 URL: http://svn.apache.org/r1487627 Log: Bug 54990 - Download large files avoiding outOfMemory Use method from commons-io Bugzilla Id: 54990 Modified: jmeter/trunk/src/jorphan/org/apache/jorphan/util/JOrphanUtils.java Date: Mon Jun 3 19:40:17 2013 New Revision: 1489124 URL: http://svn.apache.org/r1489124 Log: Bug 54990 - Download large files avoiding outOfMemory Rollback as per sebb comment on dev list Bugzilla Id: 54990 Modified: jmeter/trunk/src/jorphan/org/apache/jorphan/util/JOrphanUtils.java This issue has been migrated to GitHub: https://github.com/apache/jmeter/issues/3126 |
During tests of my web application, I need to download large file and save it on disk. To accomplish this I use an http request with a POST method to request the file and a "Save Response to a file" listener to save on disk the file. The size of the file to download is 441MB. My Jmeter (executed inside Windows XP 32 bit) started with -Xms1536m -Xmx1536m but, however, it goes to an OutOfMemory when start to write the file. Debugging the source I notice that the OutOfMemory succeed in the class ResultSaver.java, specifically at this line pw.write(s.getResponseData()); of the method: private void saveSample(SampleResult s, int num) When java call s.getResponseData(), it get an array of byte of the size 441MB. The write method of the FileOutputStream class allocates the same space for writing data to a file, causing OutOfMemory. I have resolved this problem deleting the listener "Save Response to a file" and adding "BeanShell PostProcessor" with this code: final int BUFFER_SIZE = 65536; FileOutputStream fos = new FileOutputStream("c:\\temp\\roberto\\ranemetestdwn.zip"); for(int i = 0; i < data.length; i += BUFFER_SIZE) { fos.write(data, i, Math.min(data.length-i, BUFFER_SIZE)); } fos.close(); Writing the file recursively, with a buffer of 64KB, none OutOfMemory error occur. Then, if no other side effects exist, I propose to use similar code in class ResultSaver.java, in place of pw.write(s.getResponseData());