Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10206

Incorrect handling of end-of stream leading to infinite loop in streaming session

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.1.10, 2.2.2, 3.0 beta 2
    • Component/s: None
    • Labels:
      None
    • Environment:

      cassndra 2.1.8 12 nodes cluster in two DC, with enabled ssl encryption.

    • Severity:
      Normal

      Description

      StreamMessage.deserialize (StreamMessage.java:63) function return null when receive end of stream for ReadableByteChannel.read () function. IncomingMessageHandler.run (ConnectionHandler.java:257) interpreting null - as unsupported message and ignore it. It resulting infinite loop when IncomingMessageHandler try to read message from closed steram, and deserialize always return null - to indicate close stream.

      It not happen on normal socket, because when normal socket closed it throw IOException. But happen from time to time when iner-node ssl encryption enabled.

      I'm attach two tread dump from one node with two infinity lopped thread with id 614 and 583

        Attachments

        1. screenshot-1.png
          181 kB
          Alexey Burylov
        2. threaddump-1440686737505.tdump
          278 kB
          Alexey Burylov
        3. threaddump-1440602926719.tdump
          306 kB
          Alexey Burylov
        4. cassandra-2.1-10206.txt
          1 kB
          Alexey Burylov
        5. threaddump-1440672821181.tdump
          261 kB
          Alexey Burylov
        6. threaddump-1440601628223.tdump
          292 kB
          Alexey Burylov

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              a.burylov Alexey Burylov
              Reviewers:
              Yuki Morishita
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: