[SYNAPSE-1065] GC Overhead limit exceeded error during running a performance test on simple Passthough Proxy

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Core, Transports
• Environment: ubuntu 16.04; 4 core 8GB RAM
• Assignee: Isuru Udana
• Reporter: Chanaka Fernando
• Created: 26/Dec/16 17:49
• Updated: 27/Dec/16 16:46
• Resolved: 27/Dec/16 03:02

Description

I have a simple Passthrough Proxy and a back end service which is written in netty and simply echo back the request message. When I run a performance test using Jmeter (with 100 concurrency), I am getting the following errors in the synapse log file and console.

2016-12-26 23:09:32,376 [-] [Timer-1]  WARN TimeoutHandler Expiring message ID : urn:uuid:155fee35-098b-4968-a13f-efd7b755dfc2; dropping message after global timeout of : 180 seconds
2016-12-26 23:09:47,475 [-] [Timer-1]  WARN TimeoutHandler Expiring message ID : urn:uuid:6576e483-4cb3-4111-aaf4-523a7e87781b; dropping message after global timeout of : 180 seconds
2016-12-26 23:09:54,606 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
2016-12-26 23:09:54,802 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP-PT-Listener I/O Dispatcher-3"
2016-12-26 23:09:55,506 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
2016-12-26 23:09:57,620 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
2016-12-26 23:09:58,137 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
Exception in thread "PassThroughMessageProcessor-30" 2016-12-26 23:09:58,532 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-12-26 23:09:59,097 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
2016-12-26 23:09:58,870 [-] [HTTP-PT-Listener I/O Dispatcher-2]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
Exception in thread "HTTP-PT-Listener I/O Dispatcher-2" java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-12-26 23:10:00,459 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
Exception in thread "PassThroughHTTPListener" java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-12-26 23:10:02,742 [-] [Timer-1]  WARN TimeoutHandler Expiring message ID : urn:uuid:3782bf1e-f012-44f1-923b-43dad4050ac4; dropping message after global timeout of : 180 seconds
Exception in thread "PassThroughMessageProcessor-53" 2016-12-26 23:10:03,029 [-] [HTTP-PT-Listener I/O Dispatcher-1]  WARN SourceHandler Connection closed after request is read: [ACTIVE]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "HTTP-PT-Listener I/O Dispatcher-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughHTTPSender" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-34" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "HTTPS-PT-Listener I/O Dispatcher-3" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "HTTP-PT-Listener I/O Dispatcher-4" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-357" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-212" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-247" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "HTTPS-PT-Sender I/O Dispatcher-2" Exception in thread "HTTPS-PT-Listener I/O Dispatcher-4" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "HTTP-PT-Sender I/O Dispatcher-1" Exception in thread "HTTPS-PT-Listener I/O Dispatcher-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-116" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "HTTPS-PT-Listener I/O Dispatcher-2" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-253" java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-12-26 23:10:17,766 [-] [Timer-1]  WARN TimeoutHandler Expiring message ID : urn:uuid:eaf78011-5a01-4ef7-8dff-b92f5251b282; dropping message after global timeout of : 180 seconds
Exception in thread "PassThroughHTTPSSender" Exception in thread "PassThroughMessageProcessor-158" java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Timer-2" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-227" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-345" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-142" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-385" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-136" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Timer-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Timer-3" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-113" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-16" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "PassThroughMessageProcessor-16"
Exception in thread "PassThroughMessageProcessor-100" java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-12-26 23:10:31,465 [-] [HTTP-PT-Sender I/O Dispatcher-2]  WARN TargetHandler Connection closed before receiving the request
Exception in thread "PassThroughMessageProcessor-388" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "PassThroughMessageProcessor-388"
2016-12-26 23:10:33,052 [-] [Timer-1]  WARN TimeoutHandler Expiring message ID : urn:uuid:24d0c863-f89c-4ced-abef-cbf585e8c27c; dropping message after global timeout of : 180 seconds
2016-12-26 23:10:33,517 [-] [PassThroughMessageProcessor-418] ERROR NativeWorkerPool Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-12-26 23:10:33,052 [-] [HTTP-PT-Sender I/O Dispatcher-2]  WARN TargetHandler Connection closed before receiving the request
Exception in thread "PassThroughMessageProcessor-313" java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-12-26 23:10:35,583 [-] [HTTP-PT-Sender I/O Dispatcher-2]  WARN TargetHandler Connection closed before receiving the request
Exception in thread "PassThroughMessageProcessor-29" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-421" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-59" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-415" java.lang.OutOfMemoryError: GC overhead limit exceeded2016-12-26 23:10:37,109 [-] [PassThroughMessageProcessor-59] ERROR NativeWorkerPool Uncaught exception
2016-12-26 23:10:40,244 [-] [PassThroughMessageProcessor-420] ERROR NativeWorkerPool Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "HTTP-PT-Sender I/O Dispatcher-2" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "PassThroughMessageProcessor-420" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "HTTP-PT-Sender I/O Dispatcher-4" java.lang.OutOfMemoryError: GC overhead limit exceeded