Cassandra
  1. Cassandra
  2. CASSANDRA-5000

Null pointer exception in SecondaryIndexManager.getIndexKeyFor in Cassandra 1.1.x

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Not A Problem
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Linux (exact environment doesn't appear to be relevant; reproduced on RedHat and Centos)

      Description

      Cassandra 1.1.0 and following releases gets an NPE in SecondaryIndexManager writing a CF with multiple secondary keys. Problem did not occur in 1.0.x, and can be resolved by downgrading the Cassandra server. Stack trace is:

      ERROR [MutationStage:47] 2012-11-28 11:24:30,865 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MutationStage:47,5,main]
      java.lang.RuntimeException: java.lang.NullPointerException
      at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1254)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: java.lang.NullPointerException
      at org.apache.cassandra.db.index.SecondaryIndexManager.getIndexKeyFor(SecondaryIndexManager.java:299)
      at org.apache.cassandra.db.index.SecondaryIndexManager.applyIndexUpdates(SecondaryIndexManager.java:463)
      at org.apache.cassandra.db.Table.apply(Table.java:459)
      at org.apache.cassandra.db.Table.apply(Table.java:384)
      at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:294)
      at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:453)
      at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1250)

      Client stack trace is:

      me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
      at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)[hector-core-1.0-5.jar:]
      at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)[hector-core-1.0-5.jar:]
      at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)[hector-core-1.0-5.jar:]
      at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)[hector-core-1.0-5.jar:]
      ... 3 more

      1. CassandraDefect5000-1.0-SNAPSHOT-1.tar.gz
        8.05 MB
        David Tootill
      2. CassandraDefect5000-1.0-SNAPSHOT-2.tar.gz
        5.97 MB
        David Tootill
      3. Cassandra Defect 5000 Reproduction and Background.docx
        21 kB
        David Tootill
      4. stdlog.txt
        59 kB
        David Tootill
      5. system.log
        963 kB
        David Tootill

        Activity

        Hide
        David Tootill added a comment -

        system.log is Cassandra 1.1.4 system.log
        stdlog.txt is client log

        Show
        David Tootill added a comment - system.log is Cassandra 1.1.4 system.log stdlog.txt is client log
        Hide
        Jonathan Ellis added a comment -

        Go ahead and give an example cqlsh schema and queries to reproduce.

        Show
        Jonathan Ellis added a comment - Go ahead and give an example cqlsh schema and queries to reproduce.
        Hide
        David Tootill added a comment -

        The two gzipped files contain a reproduction sandbox for this defect. Download both into the same directory, unzip and untar them, and run script run.sh in the CassandraDefect5000-1.0-SNAPSHOT that results. Cassandra 1.1.x gets an NPE; Cassandra 1.0.x does not.

        The Word document describes the schema and includes the Java source that created it.

        Show
        David Tootill added a comment - The two gzipped files contain a reproduction sandbox for this defect. Download both into the same directory, unzip and untar them, and run script run.sh in the CassandraDefect5000-1.0-SNAPSHOT that results. Cassandra 1.1.x gets an NPE; Cassandra 1.0.x does not. The Word document describes the schema and includes the Java source that created it.
        Hide
        Jonathan Ellis added a comment -

        Thanks, David.

        Show
        Jonathan Ellis added a comment - Thanks, David.
        Hide
        Yuki Morishita added a comment -

        David Tootill is "sIntSer" in your docx sample "Int32Type"? Comparator is defined as "CompositeType(IntegerType, UTF8Type, UUIDType)", so it can be type mismatch. This has been working in cassanandra 1.0.x unintentionally, but I consider it is bug in that version.

        Show
        Yuki Morishita added a comment - David Tootill is "sIntSer" in your docx sample "Int32Type"? Comparator is defined as "CompositeType(IntegerType, UTF8Type, UUIDType)", so it can be type mismatch. This has been working in cassanandra 1.0.x unintentionally , but I consider it is bug in that version.
        Hide
        David Tootill added a comment -

        sIntSer is a Serializer<Integer>. Here are the declarations of all class-level variables:

        private static boolean sCassandraActive;
        private static String sClusterName = null;
        private static String sCbmColumnFamily = null;
        private static boolean sCreateKeyspace = false;
        private static String sHostIp = null;
        private static String sContextsColumnFamily = null;
        private static String sKeyspaceName = null;
        private static String sMetricAnalysisColumnFamily = null;
        private static String sSecondaryEventsColumnFamily = null;
        private static String sTenantContextName = null;
        private static final ConcurrentSkipListSet<String> sSecondaryIndexes = new ConcurrentSkipListSet<>();
        private static final AtomicReference<Keyspace> sKeyspace = new AtomicReference<>();
        private static Cluster sCluster = null;
        private static ColumnFamilyDefinition sCbmFamilyDefinition;
        private static final Serializer<String> sStringSer = new StringSerializer();
        private static final Serializer<Integer> sIntSer = new IntegerSerializer();
        private static final Serializer<Composite> sCompositeSer = new CompositeSerializer();
        private static final Serializer<UUID> sUuidSer = new UUIDSerializer();
        private static final ConfigurableConsistencyLevel sConfigurableConsistencyLevel = new ConfigurableConsistencyLevel();
        public static final String kContextCountColumn = "contextCount";
        public static final String kTenantColumn = "Tenant_Context";
        public final static String kCbmSerializedMetricColumnName = "cbm";
        public static final StringIdentifier kMetricNameContext = new StringIdentifier("nm");
        private static final Logger sLogger = Logger.getLogger(CassandraSetup.class);

        I am compiling and testing on 64-bit machines, using 64-bit Java 7. Are you able to reproduce the problem? Are there client changes you would like me to try?

        Show
        David Tootill added a comment - sIntSer is a Serializer<Integer>. Here are the declarations of all class-level variables: private static boolean sCassandraActive; private static String sClusterName = null; private static String sCbmColumnFamily = null; private static boolean sCreateKeyspace = false; private static String sHostIp = null; private static String sContextsColumnFamily = null; private static String sKeyspaceName = null; private static String sMetricAnalysisColumnFamily = null; private static String sSecondaryEventsColumnFamily = null; private static String sTenantContextName = null; private static final ConcurrentSkipListSet<String> sSecondaryIndexes = new ConcurrentSkipListSet<>(); private static final AtomicReference<Keyspace> sKeyspace = new AtomicReference<>(); private static Cluster sCluster = null; private static ColumnFamilyDefinition sCbmFamilyDefinition; private static final Serializer<String> sStringSer = new StringSerializer(); private static final Serializer<Integer> sIntSer = new IntegerSerializer(); private static final Serializer<Composite> sCompositeSer = new CompositeSerializer(); private static final Serializer<UUID> sUuidSer = new UUIDSerializer(); private static final ConfigurableConsistencyLevel sConfigurableConsistencyLevel = new ConfigurableConsistencyLevel(); public static final String kContextCountColumn = "contextCount"; public static final String kTenantColumn = "Tenant_Context"; public final static String kCbmSerializedMetricColumnName = "cbm"; public static final StringIdentifier kMetricNameContext = new StringIdentifier("nm"); private static final Logger sLogger = Logger.getLogger(CassandraSetup.class); I am compiling and testing on 64-bit machines, using 64-bit Java 7. Are you able to reproduce the problem? Are there client changes you would like me to try?
        Hide
        Yuki Morishita added a comment -

        Yes, I did test using your jar and java7, and noticed that your client is sending 32bit integer for your first component of CompositeType.
        Cassandra's IntegerType is variable-length integer (implemented using BigInteger), and Int32Type is for 32bit int. Cassandra 1.0 carries around 32bit int even if you defined CF to use IntegerType(variable length int), and I consider that as a bug in 1.0.

        I'm not an expert at Hector but I think you can change your sIntSer to use BigIntegerSerializer, or you change your CF to use Int32Type.

        Show
        Yuki Morishita added a comment - Yes, I did test using your jar and java7, and noticed that your client is sending 32bit integer for your first component of CompositeType. Cassandra's IntegerType is variable-length integer (implemented using BigInteger), and Int32Type is for 32bit int. Cassandra 1.0 carries around 32bit int even if you defined CF to use IntegerType(variable length int), and I consider that as a bug in 1.0. I'm not an expert at Hector but I think you can change your sIntSer to use BigIntegerSerializer, or you change your CF to use Int32Type.
        Hide
        David Tootill added a comment -

        Changing the serializer to a BigIntegerSerializer did not resolve the issue. However, changing the CF to use Int32Type does work, and I no longer see the NPE in the server or the timeout in the client.

        I see you've closed the issue. I agree my personal problem is resolved, but IMO there's still an issue in the server, as an NPE is not an appropriate response to composite field mismatch, particularly one that that worked as expected in previous releases. It would be better to raise an exception that specifically cites an issue with the column name definition.

        In any case, thank you very much for the quick and accurate response to my issue.

        Show
        David Tootill added a comment - Changing the serializer to a BigIntegerSerializer did not resolve the issue. However, changing the CF to use Int32Type does work, and I no longer see the NPE in the server or the timeout in the client. I see you've closed the issue. I agree my personal problem is resolved, but IMO there's still an issue in the server, as an NPE is not an appropriate response to composite field mismatch, particularly one that that worked as expected in previous releases. It would be better to raise an exception that specifically cites an issue with the column name definition. In any case, thank you very much for the quick and accurate response to my issue.

          People

          • Assignee:
            Yuki Morishita
            Reporter:
            David Tootill
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development