Uploaded image for project: 'MINA'
  1. MINA
  2. DIRMINA-789

Possible Deadlock/Out of memory when sending large amounts of data using Nio

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 2.0.0-RC1
    • Fix Version/s: 2.0.8
    • Component/s: Core
    • Labels:
      None
    • Environment:
      Windows Vista 64-bit Java 5 and Java 6

      Description

      This is a followup to the post on the DEV mailing list, http://old.nabble.com/Help-needed-with-OutOfMemory-error-and-or-GC-Issues-Dead-Locks-td28849756.html.

      I've even simplified the test cases so now it just has one simple NioServer, and an NioClient. The MinaClient class creates 5 concurrent connections to the RandomDataServer. Upon a successful connection, the server is setup to send 500MB worth of random text data. The MinaClient just saves the received data to a temp file in the working directory. When I run this code with small amounts of data, it works fine, but with 500MB, I did not have success yet. Some times, I get OOM on the server. Some times, nothing happens.

      I've declared several constants in each class that you could change to try various settings such as changing the amount of data served by the server, whether or not to use an executor filter/IoEventThrottle etc.

      Both classes have main methods, and I was running them as stand alone applications on the same PC.

      1. MinaClient.java
        4 kB
        Sai Pullabhotla
      2. RandomDataServer.java
        4 kB
        Sai Pullabhotla

        Activity

        Hide
        psai Sai Pullabhotla added a comment -

        Classes to reproduce the behavior I'm seeing.

        Show
        psai Sai Pullabhotla added a comment - Classes to reproduce the behavior I'm seeing.
        Hide
        elecharny Emmanuel Lecharny added a comment -

        The problem is pretty clear : you are writing as much data as you can, uo to 5 Mb, but you never wit for the data to be sent to the client, so they get stacked in a writing queue until the client has read them.

        You'll get an OOM in a matter of seconds with such a server.
        What you must do is to wait for the message to be sent before sending another chunk of bytes. You can do that by controlling the data sending from the messageSent() event handler.

        Show
        elecharny Emmanuel Lecharny added a comment - The problem is pretty clear : you are writing as much data as you can, uo to 5 Mb, but you never wit for the data to be sent to the client, so they get stacked in a writing queue until the client has read them. You'll get an OOM in a matter of seconds with such a server. What you must do is to wait for the message to be sent before sending another chunk of bytes. You can do that by controlling the data sending from the messageSent() event handler.
        Hide
        psai Sai Pullabhotla added a comment -

        If I change the line 155 in the server code -

        session.write(buffer);

        to

        session.write(buffer).awaitUninterruptibly();

        Shouldn't that help? Apparently, it has no effect on the behavior. Any idea why?

        Show
        psai Sai Pullabhotla added a comment - If I change the line 155 in the server code - session.write(buffer); to session.write(buffer).awaitUninterruptibly(); Shouldn't that help? Apparently, it has no effect on the behavior. Any idea why?
        Hide
        elecharny Emmanuel Lecharny added a comment -

        It does not change the way the server behaves, because the waut last until the data is moved the the IoProcessor queue.

        The key here is to write some more chunk only when the previous message has been really sent.

        Note that for random data, as there is no way to know when a message has been sent (even if you write a chunk of 10Kb, it may be sent in 10 blocks on 1kb), you have to capture the messageSent events and compute the size really sent to know when you can send an extra chunk (here, it will be when the size sent is 10kb)

        Show
        elecharny Emmanuel Lecharny added a comment - It does not change the way the server behaves, because the waut last until the data is moved the the IoProcessor queue. The key here is to write some more chunk only when the previous message has been really sent. Note that for random data, as there is no way to know when a message has been sent (even if you write a chunk of 10Kb, it may be sent in 10 blocks on 1kb), you have to capture the messageSent events and compute the size really sent to know when you can send an extra chunk (here, it will be when the size sent is 10kb)
        Hide
        psai Sai Pullabhotla added a comment -

        Not to criticize or blame any one, but is the JavaDoc for IoSession.write(), is not accurate? It clearly states that -

        /**
        Writes the specified message to remote peer. This operation is asynchronous; IoHandler.messageSent(IoSession, Object) will be invoked when the message is actually sent to remote peer. You can also wait for the returned WriteFuture if you want to wait for the message actually written.
        **/

        As you can see it clearly states that the message is written to the remote peer, not to the IoProcessor queue. So, according to this doc, waiting on the Furture should be sufficient. If not, we need to update the documentation.

        Also, the doc gives me the impression that one call to session.write(<something>) would result in one and only one "messageSent" event with the very same message that was written, assuming that the write was successful. I do understand that at the TCP level, the data might have been broken down into several packets, but messageSent should be fired only once, after all packets are sent.

        This is my interpretation of the doc (and the behavior one would naturally expect), and it contradicts with what you have said, where one call to session.write might result in an unknown number of messageSent events. At least, I hope that's what you said. If not, please correct me understand it.

        Now coming back to my original requirement -

        Client (C), connects to Server 1 (S1) and Server 2 (S2). Let's say S1 is a web server like Apache or App Server like Tomcat. S1 is sending a large file back to C. C has to forward that to S2. S1 is sending data as fast as it can, and C is receiving the messageReceived events. As soon as it sees this event, I simply writes the message to S2 (and possibly wait for the Future to complete, which does not guarantee a successful write?). Transfer rate from C to S2 is a lot slower than what it is from S1 to C. So, how can I prevent OOM in this scenario? Do I've to use some kind of locks/synchronization based on how many bytes were received vs. sent?

        Show
        psai Sai Pullabhotla added a comment - Not to criticize or blame any one, but is the JavaDoc for IoSession.write(), is not accurate? It clearly states that - /** Writes the specified message to remote peer. This operation is asynchronous; IoHandler.messageSent(IoSession, Object) will be invoked when the message is actually sent to remote peer. You can also wait for the returned WriteFuture if you want to wait for the message actually written. **/ As you can see it clearly states that the message is written to the remote peer, not to the IoProcessor queue. So, according to this doc, waiting on the Furture should be sufficient. If not, we need to update the documentation. Also, the doc gives me the impression that one call to session.write(<something>) would result in one and only one "messageSent" event with the very same message that was written, assuming that the write was successful. I do understand that at the TCP level, the data might have been broken down into several packets, but messageSent should be fired only once, after all packets are sent. This is my interpretation of the doc (and the behavior one would naturally expect), and it contradicts with what you have said, where one call to session.write might result in an unknown number of messageSent events. At least, I hope that's what you said. If not, please correct me understand it. Now coming back to my original requirement - Client (C), connects to Server 1 (S1) and Server 2 (S2). Let's say S1 is a web server like Apache or App Server like Tomcat. S1 is sending a large file back to C. C has to forward that to S2. S1 is sending data as fast as it can, and C is receiving the messageReceived events. As soon as it sees this event, I simply writes the message to S2 (and possibly wait for the Future to complete, which does not guarantee a successful write?). Transfer rate from C to S2 is a lot slower than what it is from S1 to C. So, how can I prevent OOM in this scenario? Do I've to use some kind of locks/synchronization based on how many bytes were received vs. sent?
        Hide
        psai Sai Pullabhotla added a comment -

        It looks like there is no easy way to simply wait for a write operation until the data is sent to the remote peer (other than using the messageSent event, which could be one or many events for one write). Is this correct? if so, should I open a JIRA for enhancing the API? Possibly updating the WriteFuture to signal a completion after the data is sent over the socket?

        I'm still looking for the best way to implement my requirement. I did some proof of concept a few months ago, which seemed to have worked fine. Over the past month or so, I spent a lot of time implementing my custom protocol, encoders and decoders around MINA (This is another layer I've that I did not mention before), but now I'm stuck with this major issue. I would appreciate any help you could provide in solving this issue.

        Show
        psai Sai Pullabhotla added a comment - It looks like there is no easy way to simply wait for a write operation until the data is sent to the remote peer (other than using the messageSent event, which could be one or many events for one write). Is this correct? if so, should I open a JIRA for enhancing the API? Possibly updating the WriteFuture to signal a completion after the data is sent over the socket? I'm still looking for the best way to implement my requirement. I did some proof of concept a few months ago, which seemed to have worked fine. Over the past month or so, I spent a lot of time implementing my custom protocol, encoders and decoders around MINA (This is another layer I've that I did not mention before), but now I'm stuck with this major issue. I would appreciate any help you could provide in solving this issue.
        Hide
        elecharny Emmanuel Lecharny added a comment -

        As I sai, unless you manage the write completion with the MessageSent event, there is no way you can do it using the WriteFuture.

        Also note that modifying the API to get a blocking WriteFuture until the data is read by the client defeats the API purpose, as you first break the SEDA principle, plus you may kill the server if the client is slow to read the data, as you will block a thread waiting for the write to be done : you don't have thousands of threads available.

        Just waiting for the messageSent event to pus the next block of data is certainly the best solution :

        • you can free the thread as nothing will be done untill the event occurs
        • you don't suck a lot of memory in MINA
        • it's event driven, so you don't have to be blocking.

        I understand that when you come from a Blocking IO kind of system, this is a bit unusual, and I myself spent some time to get used to an event driven IO sysem, but once you see how it works, it's quite easy to manipulate. It's just a paradigm shift, IMO.

        Show
        elecharny Emmanuel Lecharny added a comment - As I sai, unless you manage the write completion with the MessageSent event, there is no way you can do it using the WriteFuture. Also note that modifying the API to get a blocking WriteFuture until the data is read by the client defeats the API purpose, as you first break the SEDA principle, plus you may kill the server if the client is slow to read the data, as you will block a thread waiting for the write to be done : you don't have thousands of threads available. Just waiting for the messageSent event to pus the next block of data is certainly the best solution : you can free the thread as nothing will be done untill the event occurs you don't suck a lot of memory in MINA it's event driven, so you don't have to be blocking. I understand that when you come from a Blocking IO kind of system, this is a bit unusual, and I myself spent some time to get used to an event driven IO sysem, but once you see how it works, it's quite easy to manipulate. It's just a paradigm shift, IMO.
        Hide
        psai Sai Pullabhotla added a comment -

        I understand what you are saying, and agree with it. However, I'm unable to figure out the purpose of IoThrottle filters. I tried using the WriteRequestFilter, which is what should help with the test server code I posted. But it does not seem to be helping. It just runs forever with little or no data flowing.

        Show
        psai Sai Pullabhotla added a comment - I understand what you are saying, and agree with it. However, I'm unable to figure out the purpose of IoThrottle filters. I tried using the WriteRequestFilter, which is what should help with the test server code I posted. But it does not seem to be helping. It just runs forever with little or no data flowing.
        Hide
        elecharny Emmanuel Lecharny added a comment -

        The throttle filter is a bit different. It tries to limit the incoming data flow mainly (it does not make a lot of sense for outgoing data).

        Let's say it's just a filter, and it stacks on top of the internal logic.

        Not easy ...

        Show
        elecharny Emmanuel Lecharny added a comment - The throttle filter is a bit different. It tries to limit the incoming data flow mainly (it does not make a lot of sense for outgoing data). Let's say it's just a filter, and it stacks on top of the internal logic. Not easy ...
        Hide
        psai Sai Pullabhotla added a comment -

        Well, your statements do not match up with the Javadocs. I guess, we have to do some work on the Javadocs, unless you think I'm misinterpreting them. Below is what the WriteRequestFilter's Java doc says:

        Attaches an IoEventQueueHandler to an IoSession's WriteRequest queue to provide accurate write queue status tracking.

        The biggest difference from OrderedThreadPoolExecutor and UnorderedThreadPoolExecutor is that IoEventQueueHandler.polled(Object, IoEvent) is invoked when the write operation is completed by an IoProcessor, consequently providing the accurate tracking of the write request queue status to the IoEventQueueHandler.

        Most common usage of this filter could be detecting an IoSession which writes too fast which will cause OutOfMemoryError soon:

        session.getFilterChain().addLast(
        "writeThrottle",
        new WriteRequestFilter(new IoEventQueueThrottle()));

        The sentence I would like to highlight is -

        >> Most common usage of this filter could be detecting an IoSession which writes too fast which will cause OutOfMemoryError soon.

        Which is what I'm trying to prevent with this filter.

        Show
        psai Sai Pullabhotla added a comment - Well, your statements do not match up with the Javadocs. I guess, we have to do some work on the Javadocs, unless you think I'm misinterpreting them. Below is what the WriteRequestFilter's Java doc says: Attaches an IoEventQueueHandler to an IoSession's WriteRequest queue to provide accurate write queue status tracking. The biggest difference from OrderedThreadPoolExecutor and UnorderedThreadPoolExecutor is that IoEventQueueHandler.polled(Object, IoEvent) is invoked when the write operation is completed by an IoProcessor, consequently providing the accurate tracking of the write request queue status to the IoEventQueueHandler. Most common usage of this filter could be detecting an IoSession which writes too fast which will cause OutOfMemoryError soon: session.getFilterChain().addLast( "writeThrottle", new WriteRequestFilter(new IoEventQueueThrottle())); The sentence I would like to highlight is - >> Most common usage of this filter could be detecting an IoSession which writes too fast which will cause OutOfMemoryError soon. Which is what I'm trying to prevent with this filter.
        Hide
        elecharny Emmanuel Lecharny added a comment -

        Again, it's all about waiting for the previous message to have been sent to the peer (ie, waiting for the messageSent event to be received).

        Show
        elecharny Emmanuel Lecharny added a comment - Again, it's all about waiting for the previous message to have been sent to the peer (ie, waiting for the messageSent event to be received).

          People

          • Assignee:
            Unassigned
            Reporter:
            psai Sai Pullabhotla
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development