Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-5673

NullPointerException on running instances

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 1.2.6
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      Hallo,
      We are having sporadic NullPointerException in some of the cassandra nodes in cluster (See stacktrace).
      We are having two Datacenter, each having 15 nodes with RF = 2, OS is SLES with java-1_6_0-ibm-1.6.0_sr12.0-0.5.1.
      At present only workaround is to stop the application running on same node and run repair tool on cassandra. We are unable to identify the cause of error.

      1)
      INFO|ScheduledTasks:1|org.apache.cassandra.service.GCInspector|GC for MarkSweepCompact: 347 ms for 1 collections, 138398568 used; ma
      x is 1051721728
      2013-06-19T16:25:50:843|ERROR|ReplicateOnWriteStage:115|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReplicateOnWriteStage:115,5,m
      ain]
      java.lang.RuntimeException: java.lang.NullPointerException
      at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
      at java.lang.Thread.run(Thread.java:738)
      Caused by: java.lang.NullPointerException
      at java.util.TreeSet.iterator(TreeSet.java:230)
      at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
      at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
      at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
      at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
      at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
      at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
      at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
      at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
      at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
      at org.apache.cassandra.db.Table.getRow(Table.java:347)
      at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
      at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
      at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
      at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
      ... 3 more
      2013-06-19T16:26:01:001|ERROR|ReadStage:4833|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:4833,5,main]
      java.lang.RuntimeException: java.lang.NullPointerException
      at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
      at java.lang.Thread.run(Thread.java:738)
      Caused by: java.lang.NullPointerException
      at java.util.TreeSet.iterator(TreeSet.java:230)
      at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
      at org.

      2)
      2013-06-19T08:38:23:436| INFO|Thread-2447|org.apache.cassandra.service.StorageService|Starting repair command #2, repairing 1 ranges for keyspace system_auth
      2013-06-19T08:58:25:685|ERROR|ReadStage:9270|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:9270,5,main]
      java.lang.NullPointerException
      at java.util.TreeSet.iterator(TreeSet.java:230)
      at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
      at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
      at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
      at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
      at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
      at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
      at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
      at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
      at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
      at org.apache.cassandra.db.Table.getRow(Table.java:347)
      at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
      at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
      at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
      at java.lang.Thread.run(Thread.java:738)

      Best Regards
      Sanjay

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              analyst Sanjay
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: