Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-12844

nodetool drain causing mutiple nodes crashing with hint file corruption in Cassandra 3.9

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: Legacy/Core
    • Labels:
    • Severity:
      Critical
    • Since Version:
      3.9

      Description

      The steps are as follows.
      we have 4/4 node cassandra running in 3.9 version.
      In one node made some changes to cassanra.yaml. issued a nodetool drain
      killed the cassandra process and restarted the node. After sometime nodetool status reported multiple nodes are down in that DC.

      Went and check the system.log of all the files and found the hint corruption occuring(CASSANDRA-12728). nodetool drain causing this corruption and bringing multiple nodes down is a big concern.

      ERROR [HintsDispatcher:2] 2016-10-26 12:17:59,361 HintsDispatchExecutor.java:225 - Failed to dispatch hints file 4d1362f0-053c-4042-80a7-bfc85a26c90f-1477509190999-1.hints: file is corrupted ({})
      org.apache.cassandra.io.FSReadError: java.io.EOFException
      at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:284) ~[apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsReader$BuffersIterator.computeNext(HintsReader.java:254) ~[apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:156) ~[apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:137) ~[apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:119) ~[apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:91) ~[apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:259) [apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:242) [apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:220) [apache-cassandra-3.9.jar:3.9]
      at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:199) [apache-cassandra-3.9.jar:3.9]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hari708 Harikrishnan
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: