Cassandra
  1. Cassandra
  2. CASSANDRA-4567

Error in log related to Murmur3Partitioner

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.2.0 beta 1
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Using ccm on ubuntu

      Description

      Start a 2-node cluster on cassandra-1.1. Bring down one node, upgrade it to trunk, start it back up. The following error shows up in the log:

      ...
       INFO [main] 2012-08-22 10:44:40,012 CacheService.java (line 170) Scheduling row cache save to each 0 seconds (going to save all keys).
       INFO [SSTableBatchOpen:1] 2012-08-22 10:44:40,106 SSTableReader.java (line 164) Opening /tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-2 (148 bytes)
       INFO [SSTableBatchOpen:2] 2012-08-22 10:44:40,106 SSTableReader.java (line 164) Opening /tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-1 (226 bytes)
       INFO [SSTableBatchOpen:3] 2012-08-22 10:44:40,106 SSTableReader.java (line 164) Opening /tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-3 (89 bytes)
      ERROR [SSTableBatchOpen:3] 2012-08-22 10:44:40,114 CassandraDaemon.java (line 131) Exception in thread Thread[SSTableBatchOpen:3,5,main]
      java.lang.RuntimeException: Cannot open /tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-3 because partitioner does not match org.apache.cassandra.dht.Murmur3Partitioner
              at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
              at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
              at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      ERROR [SSTableBatchOpen:2] 2012-08-22 10:44:40,114 CassandraDaemon.java (line 131) Exception in thread Thread[SSTableBatchOpen:2,5,main]
      java.lang.RuntimeException: Cannot open /tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-1 because partitioner does not match org.apache.cassandra.dht.Murmur3Partitioner
              at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
              at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
              at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      ERROR [SSTableBatchOpen:1] 2012-08-22 10:44:40,114 CassandraDaemon.java (line 131) Exception in thread Thread[SSTableBatchOpen:1,5,main]
      java.lang.RuntimeException: Cannot open /tmp/dtest-IYHWfV/test/node1/data/system/LocationInfo/system-LocationInfo-he-2 because partitioner does not match org.apache.cassandra.dht.Murmur3Partitioner
              at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:175)
              at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
              at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:236)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
       INFO [main] 2012-08-22 10:44:40,486 DatabaseDescriptor.java (line 522) Couldn't detect any schema definitions in local storage.
       INFO [main] 2012-08-22 10:44:40,487 DatabaseDescriptor.java (line 525) Found table data in data directories. Consider using the CLI to define your schema.
      ...
      

      Note that the error does not happen when upgrading from cassandra-1.0 to cassandra-1.1, or when "upgrading" from trunk to trunk.

      This is the exact dtest I used:

      from dtest import Tester, debug
      
      FROM_VERSION = 'git:cassandra-1.1'
      TO_VERSION = 'git:trunk'
      
      class TestUpgradeOneNode(Tester):
      
          def upgrade_test(self):
              # Start a cluster
              cluster = self.cluster
              cluster.set_cassandra_dir(cassandra_version=FROM_VERSION)
              cluster.populate(2).start()
              node1 = cluster.nodelist()[0]
              node1.watch_log_for('Listening for thrift clients...')
      
              # Bring one node down and upgrade it.
              node1.flush()
              node1.stop(wait_other_notice=True)
              node1.set_cassandra_dir(cassandra_version=TO_VERSION)
              node1.start(wait_other_notice=True)
      

        Activity

        Hide
        Vijay added a comment -

        CASSANDRA-3772 modified the default Paritioner to M3P (was RandomP). Users who upgrade the existing clusters need to change the yaml settings.
        May be updating NEWS.txt will help?

        Show
        Vijay added a comment - CASSANDRA-3772 modified the default Paritioner to M3P (was RandomP). Users who upgrade the existing clusters need to change the yaml settings. May be updating NEWS.txt will help?
        Hide
        Pavel Yaskevich added a comment -

        Agreed, I will update the NEWS as part of CASSANDRA-3772 when finished, I don't really see any user-friendly way to change default partitioner, users who use packages shouldn't be affected as configuration is separate but automated tools probably would have to add a check to reset partitioner when testing 1.1 -> 1.2 migration...

        Show
        Pavel Yaskevich added a comment - Agreed, I will update the NEWS as part of CASSANDRA-3772 when finished, I don't really see any user-friendly way to change default partitioner, users who use packages shouldn't be affected as configuration is separate but automated tools probably would have to add a check to reset partitioner when testing 1.1 -> 1.2 migration...
        Hide
        Tyler Patterson added a comment -

        Would it be feasible to provide a more helpful error message for people that didn't read NEWS.txt?

        Show
        Tyler Patterson added a comment - Would it be feasible to provide a more helpful error message for people that didn't read NEWS.txt?
        Hide
        Jonathan Ellis added a comment -

        Should we just ignore the partitioner after first startup, the way we do with initial_token?

        Show
        Jonathan Ellis added a comment - Should we just ignore the partitioner after first startup, the way we do with initial_token?
        Hide
        Vijay added a comment -

        We can implement it like initial token, but the problem is that the User will not notice/fix it…. When a new node is bootstrapped then he/she will run into issues.

        Show
        Vijay added a comment - We can implement it like initial token, but the problem is that the User will not notice/fix it…. When a new node is bootstrapped then he/she will run into issues.
        Hide
        Jonathan Ellis added a comment -

        We could check partitioner on bootstrap the way we do cluster name...

        Show
        Jonathan Ellis added a comment - We could check partitioner on bootstrap the way we do cluster name...
        Hide
        Vijay added a comment -

        Didn't realize that System table requires SS.getPartitioner (Some how thought it uses local partitioner).

        Looks like we cannot store the partitioner information in system table like initial token (because to open the Table/CF we need a partitioner) and we have to store it as a cluster metadata (new file or something). IMHO, We can make the log messages pretty print and live with it?

        BTW: checking the partitioner to be consistent with the cluster is a safe thing which we have to do, hence the attached patch.

        Show
        Vijay added a comment - Didn't realize that System table requires SS.getPartitioner (Some how thought it uses local partitioner). Looks like we cannot store the partitioner information in system table like initial token (because to open the Table/CF we need a partitioner) and we have to store it as a cluster metadata (new file or something). IMHO, We can make the log messages pretty print and live with it? BTW: checking the partitioner to be consistent with the cluster is a safe thing which we have to do, hence the attached patch.
        Hide
        Jonathan Ellis added a comment -

        We can make the log messages pretty print and live with it

        +1

        Show
        Jonathan Ellis added a comment - We can make the log messages pretty print and live with it +1
        Hide
        Pavel Yaskevich added a comment -

        Two things:

        • I think we can cache canonical name instead of calling StorageService.getPartitioner().getClass().getCanonicalName() all the time
        • In "if (gDigestMessage.partioner != null && !gDigestMessage.partioner.equals(DatabaseDescriptor.getClusterName()))" the second condition tries to match partitioner to cluster_name.
        Show
        Pavel Yaskevich added a comment - Two things: I think we can cache canonical name instead of calling StorageService.getPartitioner().getClass().getCanonicalName() all the time In "if (gDigestMessage.partioner != null && !gDigestMessage.partioner.equals(DatabaseDescriptor.getClusterName()))" the second condition tries to match partitioner to cluster_name.
        Hide
        Vijay added a comment -

        Done! (not sure how i missed this with the tests which i did )

        Attached patch also exits the VM if it detects the change in partitioner, earlier we where skiping the SST's and start the node without any data.

        Show
        Vijay added a comment - Done! (not sure how i missed this with the tests which i did ) Attached patch also exits the VM if it detects the change in partitioner, earlier we where skiping the SST's and start the node without any data.
        Hide
        Pavel Yaskevich added a comment -

        I see you added partitionerName to GossipDigestSynVerbHandler but I think better would be to add it to DatabaseDescriptor which would allow us to use it in GossipDigestSynVerbHandler and well as Gossiper

        Show
        Pavel Yaskevich added a comment - I see you added partitionerName to GossipDigestSynVerbHandler but I think better would be to add it to DatabaseDescriptor which would allow us to use it in GossipDigestSynVerbHandler and well as Gossiper
        Hide
        Vijay added a comment -

        Done!

        Show
        Vijay added a comment - Done!
        Hide
        Pavel Yaskevich added a comment -

        +1

        Show
        Pavel Yaskevich added a comment - +1
        Hide
        Vijay added a comment -

        Committed with regenerated test/data/serialization/1.2/gms.Gossip.bin, Thanks!

        Show
        Vijay added a comment - Committed with regenerated test/data/serialization/1.2/gms.Gossip.bin, Thanks!
        Hide
        Jonathan Ellis added a comment -

        looks like this broke SSTableReaderTest.testPersistantStatisticsWithSecondaryIndex

        Show
        Jonathan Ellis added a comment - looks like this broke SSTableReaderTest.testPersistantStatisticsWithSecondaryIndex
        Hide
        Vijay added a comment -

        Fixed in 9cd53fba648ae5a30a181f8a06786f33db95a0fe

        Show
        Vijay added a comment - Fixed in 9cd53fba648ae5a30a181f8a06786f33db95a0fe

          People

          • Assignee:
            Vijay
            Reporter:
            Tyler Patterson
            Reviewer:
            Pavel Yaskevich
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development