[CASSANDRA-5673] NullPointerException on running instances

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.2.6
• Severity: Normal
• Assignee: Unassigned
• Reporter: Sanjay
• Created: 20/Jun/13 06:39
• Updated: 16/Apr/19 09:32
• Resolved: 12/Sep/13 07:34

Description

Hallo,
We are having sporadic NullPointerException in some of the cassandra nodes in cluster (See stacktrace).
We are having two Datacenter, each having 15 nodes with RF = 2, OS is SLES with java-1_6_0-ibm-1.6.0_sr12.0-0.5.1.
At present only workaround is to stop the application running on same node and run repair tool on cassandra. We are unable to identify the cause of error.

1)
INFO|ScheduledTasks:1|org.apache.cassandra.service.GCInspector|GC for MarkSweepCompact: 347 ms for 1 collections, 138398568 used; ma
x is 1051721728
2013-06-19T16:25:50:843|ERROR|ReplicateOnWriteStage:115|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReplicateOnWriteStage:115,5,m
ain]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Caused by: java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
... 3 more
2013-06-19T16:26:01:001|ERROR|ReadStage:4833|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:4833,5,main]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Caused by: java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.

2)
2013-06-19T08:38:23:436| INFO|Thread-2447|org.apache.cassandra.service.StorageService|Starting repair command #2, repairing 1 ranges for keyspace system_auth
2013-06-19T08:58:25:685|ERROR|ReadStage:9270|org.apache.cassandra.service.CassandraDaemon|Exception in thread Thread[ReadStage:9270,5,main]
java.lang.NullPointerException
at java.util.TreeSet.iterator(TreeSet.java:230)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:163)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
at java.lang.Thread.run(Thread.java:738)

Best Regards
Sanjay