This is a followup to the post on the DEV mailing list, http://old.nabble.com/Help-needed-with-OutOfMemory-error-and-or-GC-Issues-Dead-Locks-td28849756.html.
I've even simplified the test cases so now it just has one simple NioServer, and an NioClient. The MinaClient class creates 5 concurrent connections to the RandomDataServer. Upon a successful connection, the server is setup to send 500MB worth of random text data. The MinaClient just saves the received data to a temp file in the working directory. When I run this code with small amounts of data, it works fine, but with 500MB, I did not have success yet. Some times, I get OOM on the server. Some times, nothing happens.
I've declared several constants in each class that you could change to try various settings such as changing the amount of data served by the server, whether or not to use an executor filter/IoEventThrottle etc.
Both classes have main methods, and I was running them as stand alone applications on the same PC.