ActiveMQ
  1. ActiveMQ
  2. AMQ-1249

Exception when sending big messages

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 4.1.1
    • Fix Version/s: 5.3.0
    • Component/s: Broker
    • Labels:
      None

      Description

      Scenario:

      used queue to transfer a message of size 10MB, producer and consumer were in the same location, but broker was available through a WAN, so sending a message took some time, client crashed with exception (also added some trace):

      14:58:00,914 DEBUG [main] org.apache.activemq.ActiveMQSession: Sending message: ActiveMQBytesMessage {commandId = 0, responseRequired = false, messageId = ID:itstl007-41444-1179835075925-1:0:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:itstl007-41444-1179835075925-1:0:1:1, destination = queue://Huge.Remote.1, transactionId = null, expiration = 0, timestamp = 1179835080911, arrival = 0, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@7808b9, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = {HASH=80f6524683f484f19f244fc5d290f79f5f1b48c1}, readOnlyProperties = true, readOnlyBody = true, droppable = false} ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null }
      14:58:00,914 DEBUG [main] org.apache.activemq.transport.TransportLogger.Connection:1: SENDING: ActiveMQBytesMessage {commandId = 5, responseRequired = true, messageId = ID:itstl007-41444-1179835075925-1:0:1:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:itstl007-41444-1179835075925-1:0:1:1, destination = queue://Huge.Remote.1, transactionId = null, expiration = 0, timestamp = 1179835080911, arrival = 0, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@7808b9, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = {HASH=80f6524683f484f19f244fc5d290f79f5f1b48c1}, readOnlyProperties = true, readOnlyBody = true, droppable = false} ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null }
      14:58:26,716 DEBUG [ActiveMQ Transport: tcp://gkscoc01.igk.intel.com/172.28.168.10:61616] org.apache.activemq.transport.TransportLogger.Connection:1: RECEIVED: KeepAliveInfo {}
      14:58:41,699 DEBUG [ActiveMQ Transport: tcp://gkscoc01.igk.intel.com/172.28.168.10:61616] org.apache.activemq.transport.TransportLogger.Connection:1: RECEIVED: KeepAliveInfo {}
      14:58:56,701 DEBUG [ActiveMQ Transport: tcp://gkscoc01.igk.intel.com/172.28.168.10:61616] org.apache.activemq.transport.TransportLogger.Connection:1: RECEIVED: KeepAliveInfo {}
      14:58:56,707 DEBUG [ActiveMQ Transport: tcp://gkscoc01.igk.intel.com/172.28.168.10:61616] org.apache.activemq.transport.TransportLogger.Connection:1: RECEIVED Exception: java.io.EOFException
      java.io.EOFException
              at java.io.DataInputStream.readInt(DataInputStream.java:397)
              at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:267)
              at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:156)
              at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:136)
              at java.lang.Thread.run(Thread.java:536)
      14:58:56,709 DEBUG [main] org.apache.activemq.transport.TransportLogger.Connection:1: SENDING: RemoveInfo {commandId = 6, responseRequired = false, objectId = ID:itstl007-41444-1179835075925-1:0:1:1}
      14:58:56,711 DEBUG [main] org.apache.activemq.transport.TransportLogger.Connection:1: SENDING: RemoveInfo {commandId = 7, responseRequired = false, objectId = ID:itstl007-41444-1179835075925-1:0:1}
      14:58:56,711 DEBUG [main] org.apache.activemq.transport.TransportLogger.Connection:1: SENDING: RemoveInfo {commandId = 8, responseRequired = true, objectId = ID:itstl007-41444-1179835075925-1:0}
      Exception in thread "main" javax.jms.JMSException: The transport is not running.
              at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:58)
              at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1213)
              at org.apache.activemq.ActiveMQConnection.close(ActiveMQConnection.java:579)
              at com.intel.ingrid.test.stress.jms.HugeMessageProducer.main(HugeMessageProducer.java:93)
      Caused by: java.io.IOException: The transport is not running.
              at org.apache.activemq.transport.TransportSupport.checkStarted(TransportSupport.java:103)
              at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:117)
              at org.apache.activemq.transport.TransportLogger.oneway(TransportLogger.java:71)
              at org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:141)
              at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:80)
              at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:93)
              at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47)
              at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
              at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:79)
              at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1203)
              ... 2 more
      

      After I changed wireFormat.maxInactivityDuration to 0 it succeeded and sent the message.

      It's strange for me because in the first case (when wireFormat.maxInactivityDuration is not set, so default value is used) there's an activity occuring - the client sends a lot of data so server should not treat it as a hanging client and the server should not close the connection.

        Activity

        Hide
        Kevin Yaussy added a comment -

        I think it is likely that you can't really detect this - the code is blocked on the socket.write. I don't think that can be distinguished from the socket being in a hung
        state, or the target broker being hung. You might have to bump up the maxInactivity value to a higher value than the default to allow for large messages
        and your WAN.

        Show
        Kevin Yaussy added a comment - I think it is likely that you can't really detect this - the code is blocked on the socket.write. I don't think that can be distinguished from the socket being in a hung state, or the target broker being hung. You might have to bump up the maxInactivity value to a higher value than the default to allow for large messages and your WAN.
        Hide
        Pawel Niewiadomski added a comment -

        Let me think loud: if you use socket.write you usually write chunk by chunk so you can actually detect this somehow - you're probably use socket.write to send chunk by chunk, right? So each call to socket.write will return after some time, so you can set flag "I'm still alive" to true and continue sending, if connection fails flag will not be set and application will know that something's wrong.

        Am I thinking right?

        Show
        Pawel Niewiadomski added a comment - Let me think loud: if you use socket.write you usually write chunk by chunk so you can actually detect this somehow - you're probably use socket.write to send chunk by chunk, right? So each call to socket.write will return after some time, so you can set flag "I'm still alive" to true and continue sending, if connection fails flag will not be set and application will know that something's wrong. Am I thinking right?
        Hide
        james strachan added a comment -

        It would be good if the TCP transport implementation kept the transport alive while its still reading chunks of a single message. So even if one huge Command is being read over a slow network, it still kept the activity monitor alive.

        BTW another work around is to use out of band messaging for large messages over 1Mb

        http://activemq.apache.org/blob-messages.html

        Show
        james strachan added a comment - It would be good if the TCP transport implementation kept the transport alive while its still reading chunks of a single message. So even if one huge Command is being read over a slow network, it still kept the activity monitor alive. BTW another work around is to use out of band messaging for large messages over 1Mb http://activemq.apache.org/blob-messages.html
        Hide
        Dejan Bosanac added a comment -

        This issue is duplicate of https://issues.apache.org/activemq/browse/AMQ-2088, which just has been fixed

        Show
        Dejan Bosanac added a comment - This issue is duplicate of https://issues.apache.org/activemq/browse/AMQ-2088 , which just has been fixed

          People

          • Assignee:
            Dejan Bosanac
            Reporter:
            Pawel Niewiadomski
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development