Cassandra
  1. Cassandra
  2. CASSANDRA-4195

error in log when upgrading multi-node cluster to 1.1

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.0.11
    • Component/s: Core
    • Labels:
      None
    • Environment:

      ccm, dtest. Ubuntu

      Description

      I upgraded a cluster from 1.0.9 to 1.1.0. The following message shows up in the logs for all but the first node.

      ERROR [GossipStage:1] 2012-04-30 07:37:06,986 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[GossipStage:1,5,main]
      java.lang.UnsupportedOperationException: Not a time-based UUID                  
          at java.util.UUID.timestamp(UUID.java:331)                                  
          at org.apache.cassandra.service.MigrationManager.updateHighestKnown(MigrationManager.java:121)
          at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:99)
          at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:83)
          at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:806)           
          at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:849)
          at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:908)   
          at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:62)
          at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
          at java.lang.Thread.run(Thread.java:679)
      

      this dtest demonstrates the issue. It was added to the cassandra-dtest repository as upgrade_to_11_test:

      from dtest import Tester, debug 
      from tools import * 
       
      class TestUpgradeTo1_1(Tester): 
       
          def upgrade_test(self): 
              self.num_rows = 0 
              cluster = self.cluster 
       
              # Forcing cluster version on purpose 
              cluster.set_cassandra_dir(cassandra_version='1.0.9') 
       
              cluster.populate(3).start() 
              time.sleep(1) 
       
              for node in cluster.nodelist():     
                  node.flush() 
                  time.sleep(.5) 
                  node.stop(wait_other_notice=True) 
                  node.set_cassandra_dir(cassandra_version='1.1.0') 
                  node.start(wait_other_notice=True) 
                  time.sleep(.5)
      
      1. CASSANDRA-4195.patch
        3 kB
        Pavel Yaskevich

        Activity

        Hide
        Jonathan Ellis added a comment -

        I thought we added logic to not send incompatible migration messages to 1.0.x nodes?

        Show
        Jonathan Ellis added a comment - I thought we added logic to not send incompatible migration messages to 1.0.x nodes?
        Hide
        Pavel Yaskevich added a comment -

        Yes, we don't send it and the problem is not about it - here we see the same problem I was talking about at CASSANDRA-3804 - as we no longer use time-based UUID to represent schema version, the "old" nodes (version < 1.1) would fail to join into the ring because they would be stuck on the "highest" version resolution as they assume that all nodes have time-based UUID for version which is no longer the case. We should make a strict regulation about mixed version C* ring otherwise things would continue to break in the different ways, probably to forbid the older node to join if it detects nodes with version >= 1.1, because we can't rely just skip schema agreement checks on startup or on ring changes...

        Show
        Pavel Yaskevich added a comment - Yes, we don't send it and the problem is not about it - here we see the same problem I was talking about at CASSANDRA-3804 - as we no longer use time-based UUID to represent schema version, the "old" nodes (version < 1.1) would fail to join into the ring because they would be stuck on the "highest" version resolution as they assume that all nodes have time-based UUID for version which is no longer the case. We should make a strict regulation about mixed version C* ring otherwise things would continue to break in the different ways, probably to forbid the older node to join if it detects nodes with version >= 1.1, because we can't rely just skip schema agreement checks on startup or on ring changes...
        Hide
        Jonathan Ellis added a comment -

        probably to forbid the older node to join if it detects nodes with version >= 1.1, because we can't rely just skip schema agreement checks on startup or on ring changes

        Doesn't that kill node-at-a-time upgrading dead in the water?

        Show
        Jonathan Ellis added a comment - probably to forbid the older node to join if it detects nodes with version >= 1.1, because we can't rely just skip schema agreement checks on startup or on ring changes Doesn't that kill node-at-a-time upgrading dead in the water?
        Hide
        Pavel Yaskevich added a comment -

        It would, but on the other hand one can't upgrade node-at-a-time to 1.1 because of incompatible schema changes... We of course can report something like "Nodes with C* version >= 1.1 detected, please upgrade!" instead of failing in MM.rectify(...) but if we continue to operate on that node we can't guarantee stability anyway...

        Show
        Pavel Yaskevich added a comment - It would, but on the other hand one can't upgrade node-at-a-time to 1.1 because of incompatible schema changes... We of course can report something like "Nodes with C* version >= 1.1 detected, please upgrade!" instead of failing in MM.rectify(...) but if we continue to operate on that node we can't guarantee stability anyway...
        Hide
        Jonathan Ellis added a comment -

        We need to support read/write ops in a mixed cluster. We don't need to support schema changes.

        Show
        Jonathan Ellis added a comment - We need to support read/write ops in a mixed cluster. We don't need to support schema changes.
        Hide
        Pavel Yaskevich added a comment -

        We need to support read/write ops in a mixed cluster. We don't need to support schema changes.

        See my first comment, old nodes wouldn't be able to join the ring because they can't resolve the schema version correctly and as we can't know what is the difference between old/new schemas they wouldn't be able to properly handle read/write workloads. We can probably add special mechanism to the 1.0 that would ask for schema in Avro format if it detects that schema version is not a TimeUUID, load it and try to operate but that would require from uses to update their nodes to the latest 1.0 version before they can do upgrade to 1.1.

        Show
        Pavel Yaskevich added a comment - We need to support read/write ops in a mixed cluster. We don't need to support schema changes. See my first comment, old nodes wouldn't be able to join the ring because they can't resolve the schema version correctly and as we can't know what is the difference between old/new schemas they wouldn't be able to properly handle read/write workloads. We can probably add special mechanism to the 1.0 that would ask for schema in Avro format if it detects that schema version is not a TimeUUID, load it and try to operate but that would require from uses to update their nodes to the latest 1.0 version before they can do upgrade to 1.1.
        Hide
        Jonathan Ellis added a comment -

        Ugh, how did we miss this in testing?

        How I think it should work:

        1. if node can't parse schema version, it should join ring w/ whatever schema it currently has
        2. nodes should make best-effort to fulfil requests w/ current schema instead of blocking for schema reconciliation

        That way as long as users don't try to modify schema during the upgrade they will be fine.

        I think #2 works as desired, which leaves #1 as a fairly easy update to make in 1.0.11.

        Show
        Jonathan Ellis added a comment - Ugh, how did we miss this in testing? How I think it should work: if node can't parse schema version, it should join ring w/ whatever schema it currently has nodes should make best-effort to fulfil requests w/ current schema instead of blocking for schema reconciliation That way as long as users don't try to modify schema during the upgrade they will be fine. I think #2 works as desired, which leaves #1 as a fairly easy update to make in 1.0.11.
        Hide
        Pavel Yaskevich added a comment -

        Ok, I can do #1 and it would get ring into disagreement until upgrade to 1.1 is finished with is a good thing in our case because users won't be able to mutate it. I was warning about from the time we have decided to change the way version is handled...

        Show
        Pavel Yaskevich added a comment - Ok, I can do #1 and it would get ring into disagreement until upgrade to 1.1 is finished with is a good thing in our case because users won't be able to mutate it. I was warning about from the time we have decided to change the way version is handled...
        Hide
        Pavel Yaskevich added a comment -

        patch against cassandra-1.0 to make it skip trying to migrate the schema if it detects that it's running in mixed node cluster and verbose warning.

        Show
        Pavel Yaskevich added a comment - patch against cassandra-1.0 to make it skip trying to migrate the schema if it detects that it's running in mixed node cluster and verbose warning.
        Hide
        Brandon Williams added a comment -

        Worked for me.

        Show
        Brandon Williams added a comment - Worked for me.
        Hide
        Jonathan Ellis added a comment -

        +1

        Show
        Jonathan Ellis added a comment - +1
        Hide
        Pavel Yaskevich added a comment -

        Committed.

        Show
        Pavel Yaskevich added a comment - Committed.

          People

          • Assignee:
            Pavel Yaskevich
            Reporter:
            Tyler Patterson
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development