Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.1.0
    • Component/s: Core
    • Labels:
      None
    • Environment:

      ubuntu, cluster set up with ccm.

      Description

      A 3-node cluster is on version 0.8.9, 1.0.6, or 1.0.7 and then one and only one node is taken down, upgraded to trunk, and started again. An rpc timeout exception happens if counter-add operations are done. It usually takes between 1 and 500 add operations before the failure occurs. The failure seems to happen sooner if the coordinator node is NOT the one that was upgraded. Here is the error:

      
      ======================================================================
      ERROR: counter_upgrade_test.TestCounterUpgrade.counter_upgrade_test
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "/usr/lib/pymodules/python2.7/nose/case.py", line 187, in runTest
          self.test(*self.arg)
        File "/home/tahooie/cassandra-dtest/counter_upgrade_test.py", line 50, in counter_upgrade_test
          cursor.execute("UPDATE counters SET row = row+1 where key='a'")
        File "/usr/local/lib/python2.7/dist-packages/cql/cursor.py", line 96, in execute
          raise cql.OperationalError("Request did not complete within rpc_timeout.")
      OperationalError: Request did not complete within rpc_timeout.
      
      
      1. CASSANDRA-3804.patch
        1 kB
        Pavel Yaskevich
      2. CASSANDRA-3804-1.1.patch
        0.9 kB
        Pavel Yaskevich
      3. CASSANDRA-3804-1.1-v2.patch
        4 kB
        Pavel Yaskevich
      4. node1.log
        25 kB
        Sylvain Lebresne
      5. node2.log
        20 kB
        Sylvain Lebresne

        Activity

        Hide
        Sylvain Lebresne added a comment -

        This is not counter related (but you have to use CL.ALL to reproduce without counters as otherwise it's hidden by the fact that only the non-upgraded coordinator acknowledges writes) and it is related to CASSANDRA-1391.

        This is due to the inability of doing schema changes in a mixed pre/post-1.1 cluster, if I trust the following log (from the upgraded node):

        java.lang.RuntimeException: java.io.IOException: Can't accept schema migrations from Cassandra versions previous to 1.1, please update first.
                at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:544)
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
                at java.util.concurrent.FutureTask.run(FutureTask.java:138)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)
        Caused by: java.io.IOException: Can't accept schema migrations from Cassandra versions previous to 1.1, please update first.
                at org.apache.cassandra.service.MigrationManager.deserializeMigrationMessage(MigrationManager.java:233)
                at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:231)
                at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48)
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        

        There is however two problems imho:

        1. Not supporting migrations during the upgrade process is one thing, but it should put the cluster in a broken state, which I'm not sure it doesn't do. Ideally, new nodes would still accept old migrations from old nodes, but would refuse to schema changes themselves until they know all nodes are upgraded. We could then throw an UnavailableException with a message.
        2. On top of the exception above, the logs during that test are filled with errors that don't sound too reassuring. On every node (upgraded or not), there is a handful of:
          ERROR [MutationStage:34] 2012-01-30 14:35:39,041 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MutationStage:34,5,main]
          java.io.IOError: java.io.EOFException
                  at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:66)
                  at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          Caused by: java.io.EOFException
                  at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
                  at java.io.DataInputStream.readUTF(DataInputStream.java:572)
                  at java.io.DataInputStream.readUTF(DataInputStream.java:547)
                  at org.apache.cassandra.db.TruncationSerializer.deserialize(Truncation.java:80)
                  at org.apache.cassandra.db.TruncationSerializer.deserialize(Truncation.java:70)
                  at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:44)
                  ... 4 more
          

          On the upgraded node, there is a few:

          ERROR [MutationStage:38] 2012-01-30 14:35:50,772 RowMutationVerbHandler.java (line 61) Error in row mutation
          org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=1000
                  at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
                  at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:401)
                  at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:409)
                  at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:357)
                  at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
                  at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          

          And on the non-upgraded ones, there is a few:

          ERROR [GossipStage:1] 2012-01-30 14:35:13,363 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[GossipStage:1,5,main]
          java.lang.UnsupportedOperationException: Not a time-based UUID
                  at java.util.UUID.timestamp(UUID.java:308)
                  at org.apache.cassandra.service.MigrationManager.updateHighestKnown(MigrationManager.java:121)
                  at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:99)
                  at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:83)
                  at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:806)
                  at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:849)
                  at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:908)
                  at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
                  at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                  at java.lang.Thread.run(Thread.java:662)
          
        Show
        Sylvain Lebresne added a comment - This is not counter related (but you have to use CL.ALL to reproduce without counters as otherwise it's hidden by the fact that only the non-upgraded coordinator acknowledges writes) and it is related to CASSANDRA-1391 . This is due to the inability of doing schema changes in a mixed pre/post-1.1 cluster, if I trust the following log (from the upgraded node): java.lang.RuntimeException: java.io.IOException: Can't accept schema migrations from Cassandra versions previous to 1.1, please update first. at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:544) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Can't accept schema migrations from Cassandra versions previous to 1.1, please update first. at org.apache.cassandra.service.MigrationManager.deserializeMigrationMessage(MigrationManager.java:233) at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:231) at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) There is however two problems imho: Not supporting migrations during the upgrade process is one thing, but it should put the cluster in a broken state, which I'm not sure it doesn't do. Ideally, new nodes would still accept old migrations from old nodes, but would refuse to schema changes themselves until they know all nodes are upgraded. We could then throw an UnavailableException with a message. On top of the exception above, the logs during that test are filled with errors that don't sound too reassuring. On every node (upgraded or not), there is a handful of: ERROR [MutationStage:34] 2012-01-30 14:35:39,041 AbstractCassandraDaemon.java (line 134) Fatal exception in thread Thread[MutationStage:34,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:66) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323) at java.io.DataInputStream.readUTF(DataInputStream.java:572) at java.io.DataInputStream.readUTF(DataInputStream.java:547) at org.apache.cassandra.db.TruncationSerializer.deserialize(Truncation.java:80) at org.apache.cassandra.db.TruncationSerializer.deserialize(Truncation.java:70) at org.apache.cassandra.db.TruncateVerbHandler.doVerb(TruncateVerbHandler.java:44) ... 4 more On the upgraded node, there is a few: ERROR [MutationStage:38] 2012-01-30 14:35:50,772 RowMutationVerbHandler.java (line 61) Error in row mutation org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=1000 at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:401) at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:409) at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:357) at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) And on the non-upgraded ones, there is a few: ERROR [GossipStage:1] 2012-01-30 14:35:13,363 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[GossipStage:1,5,main] java.lang.UnsupportedOperationException: Not a time-based UUID at java.util.UUID.timestamp(UUID.java:308) at org.apache.cassandra.service.MigrationManager.updateHighestKnown(MigrationManager.java:121) at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:99) at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:83) at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:806) at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:849) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:908) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
        Hide
        Pavel Yaskevich added a comment -

        This was discussed with Jonathan on the process of CASSANDRA-1391, users should make sure that all of the nodes are updated to 1.1 before running any schema changes because it's impossible to apply old migrations even if we accept them and users will be getting exceptions from your #2 anyway.

        Show
        Pavel Yaskevich added a comment - This was discussed with Jonathan on the process of CASSANDRA-1391 , users should make sure that all of the nodes are updated to 1.1 before running any schema changes because it's impossible to apply old migrations even if we accept them and users will be getting exceptions from your #2 anyway.
        Hide
        Jonathan Ellis added a comment -

        We can't support mixed-version schema changes, but we should make sure that we don't leave the cluster broken if a user does that anyway.

        Show
        Jonathan Ellis added a comment - We can't support mixed-version schema changes, but we should make sure that we don't leave the cluster broken if a user does that anyway.
        Hide
        Pavel Yaskevich added a comment -

        This exception (taken from Sylvain's #2) explains what will happen when you only partially migrate:

        ERROR [GossipStage:1] 2012-01-30 14:35:13,363 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[GossipStage:1,5,main]
        java.lang.UnsupportedOperationException: Not a time-based UUID
                at java.util.UUID.timestamp(UUID.java:308)
                at org.apache.cassandra.service.MigrationManager.updateHighestKnown(MigrationManager.java:121)
                at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:99)
                at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:83)
                at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:806)
                at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:849)
                at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:908)
                at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
                at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)
        

        As we switched from Time-based UUID for schema versions MigrationManager on the old nodes will fail all the time when nodes with new schema start-up or when they will request migrations from it (because they see that their schema version is different from others). Even if we make a fix in MigrationManager.rectify(...) method for 1.0.x, nodes with new/old schema will never come to agreement because of different types of the UUID and because they unable to run schema mutations anymore.

        Show
        Pavel Yaskevich added a comment - This exception (taken from Sylvain's #2) explains what will happen when you only partially migrate: ERROR [GossipStage:1] 2012-01-30 14:35:13,363 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[GossipStage:1,5,main] java.lang.UnsupportedOperationException: Not a time-based UUID at java.util.UUID.timestamp(UUID.java:308) at org.apache.cassandra.service.MigrationManager.updateHighestKnown(MigrationManager.java:121) at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:99) at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:83) at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:806) at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:849) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:908) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) As we switched from Time-based UUID for schema versions MigrationManager on the old nodes will fail all the time when nodes with new schema start-up or when they will request migrations from it (because they see that their schema version is different from others). Even if we make a fix in MigrationManager.rectify(...) method for 1.0.x, nodes with new/old schema will never come to agreement because of different types of the UUID and because they unable to run schema mutations anymore.
        Hide
        Jonathan Ellis added a comment -

        "Never come to agreement" is fine as long as normal reads/writes (against existing CFs) continue to work.

        Show
        Jonathan Ellis added a comment - "Never come to agreement" is fine as long as normal reads/writes (against existing CFs) continue to work.
        Hide
        Pavel Yaskevich added a comment -

        "Never come to agreement" is fine as long as normal reads/writes (against existing CFs) continue to work.

        reads/writes should work against existing CFs. failure from description and first comment are related to the way how cassandra-dtest works because it tries to re-create schema for every test-case which won't work for in the mixed version cluster, if, for example, it was to create a ColumnFamily before updating one of the nodes to trunk, reads/writes to that ColumnFamily would still work after update even tho nodes will be in schema disagreement.

        Show
        Pavel Yaskevich added a comment - "Never come to agreement" is fine as long as normal reads/writes (against existing CFs) continue to work. reads/writes should work against existing CFs. failure from description and first comment are related to the way how cassandra-dtest works because it tries to re-create schema for every test-case which won't work for in the mixed version cluster, if, for example, it was to create a ColumnFamily before updating one of the nodes to trunk, reads/writes to that ColumnFamily would still work after update even tho nodes will be in schema disagreement.
        Hide
        Jonathan Ellis added a comment -

        Can we log something telling the user "you can't change schema until you finish upgrade everything" instead of a scary-looking "not a time-based UUID" uncaught exception?

        Show
        Jonathan Ellis added a comment - Can we log something telling the user "you can't change schema until you finish upgrade everything" instead of a scary-looking "not a time-based UUID" uncaught exception?
        Hide
        Pavel Yaskevich added a comment -

        patch for cassandra-1.0 branch, which does a UUID version check in MigrationManager.rectify(UUID, InetAddress) to determine if version is time-based UUID if not it would log an error and return.

        Show
        Pavel Yaskevich added a comment - patch for cassandra-1.0 branch, which does a UUID version check in MigrationManager.rectify(UUID, InetAddress) to determine if version is time-based UUID if not it would log an error and return.
        Hide
        Jonathan Ellis added a comment -

        It's the 1.0 side that has the problem? I missed that... I'd rather not require people upgrade to 1.0.8+, before upgrading to 1.1.

        Can we just have 1.1 use MS version info to not send schema to nodes that can't understand it?

        Show
        Jonathan Ellis added a comment - It's the 1.0 side that has the problem? I missed that... I'd rather not require people upgrade to 1.0.8+, before upgrading to 1.1. Can we just have 1.1 use MS version info to not send schema to nodes that can't understand it?
        Hide
        Pavel Yaskevich added a comment -

        We can do that. "java.lang.UnsupportedOperationException: Not a time-based UUID" happens on 1.0 nodes so I thought that it would be appropriate to fix it there also.

        Show
        Pavel Yaskevich added a comment - We can do that. "java.lang.UnsupportedOperationException: Not a time-based UUID" happens on 1.0 nodes so I thought that it would be appropriate to fix it there also.
        Hide
        Pavel Yaskevich added a comment -

        changes the MigrationManager.announce to skip the nodes with versions older than 1.1

        Show
        Pavel Yaskevich added a comment - changes the MigrationManager.announce to skip the nodes with versions older than 1.1
        Hide
        Sylvain Lebresne added a comment -

        Tried the two attached patches. It does remove a bunch of the exceptions. For some reason there is still quite a few EOFExceptions related to truncation, but the test doesn't truncate at all so I'm not sure were that coming from. I'm attaching the logs from the nodes for reference (that's the log after the two patches are applied). node1.log is the 1.1 node and node2.log is a 1.0 node. The thing that triggers those exception is the creation of a CF on node2 (the old one).

        So it'd be nice to figure out what triggers those exception, but if we're going to patch both 1.1 and 1.0, why not just (or rather in addition) have schema changes check the (known) version of all other nodes before doing anything and just throw an InvalidRequestException if we know the schema change will fail?

        Show
        Sylvain Lebresne added a comment - Tried the two attached patches. It does remove a bunch of the exceptions. For some reason there is still quite a few EOFExceptions related to truncation, but the test doesn't truncate at all so I'm not sure were that coming from. I'm attaching the logs from the nodes for reference (that's the log after the two patches are applied). node1.log is the 1.1 node and node2.log is a 1.0 node. The thing that triggers those exception is the creation of a CF on node2 (the old one). So it'd be nice to figure out what triggers those exception, but if we're going to patch both 1.1 and 1.0, why not just (or rather in addition) have schema changes check the (known) version of all other nodes before doing anything and just throw an InvalidRequestException if we know the schema change will fail?
        Hide
        Pavel Yaskevich added a comment -

        I will investigate how truncate is related to the schema modifications. I don't think that we have any intention to patch both 1.0 and 1.1 because we don't want to require people to update to 1.0.8+ before upgrading to 1.1.

        Show
        Pavel Yaskevich added a comment - I will investigate how truncate is related to the schema modifications. I don't think that we have any intention to patch both 1.0 and 1.1 because we don't want to require people to update to 1.0.8+ before upgrading to 1.1.
        Hide
        Sylvain Lebresne added a comment -

        I'm personally fine saying that it is recommended to upgrade to 1.0.8+ before 1.1 if that allow smoother/safer upgrade. Besides, I don't think we really have a choice if we want to avoid scary looking logs issue. And I don't really understand your remark anyway since the patches you attached to this issue targets both 1.0 and 1.1.

        Show
        Sylvain Lebresne added a comment - I'm personally fine saying that it is recommended to upgrade to 1.0.8+ before 1.1 if that allow smoother/safer upgrade. Besides, I don't think we really have a choice if we want to avoid scary looking logs issue. And I don't really understand your remark anyway since the patches you attached to this issue targets both 1.0 and 1.1.
        Hide
        Jonathan Ellis added a comment -

        If 1.1 doesn't send new-style schema changes to 1.0, that should be sufficient to avoid scary-looking logs.

        Show
        Jonathan Ellis added a comment - If 1.1 doesn't send new-style schema changes to 1.0, that should be sufficient to avoid scary-looking logs.
        Hide
        Pavel Yaskevich added a comment -

        if you take a look at the comment from Jonathan at "13/Feb/12 23:34" it will give you a better understanding why there are both 1.0 and 1.1 patches.

        Show
        Pavel Yaskevich added a comment - if you take a look at the comment from Jonathan at "13/Feb/12 23:34" it will give you a better understanding why there are both 1.0 and 1.1 patches.
        Hide
        Sylvain Lebresne added a comment -

        Reattaching the two logs with only the 1.1 patch applied. Putting asides all the truncation related EOFException for now:

        • on node1 (the 1.1 node), we do see a "Can't accept schema migrations from Cassandra versions previous to 1.1, please update first", though I suppose it would be nice to avoid the 'Fatal exception' and stacktrace that tends to scare people.
        • on node2, we still have "UnsupportedOperationException: Not a time-based UUID" exceptions.

        That being said, even if we fix all those exceptions (which we should, there no question on that), it is still the case that the schema change is applied to the 1.0 nodes and no error is returned to the user (that is, outside of the log file), but the added column family is not really usable (the user will get timeouts) at least until the cluster upgrade is complete. Even if the user do watch the log and see the error (and will probably assume the creation failed), the column family is still created. That's not really user friendly. So I still think it would be a good idea to add code to 1.0 and 1.1 to refuse upfront schema changes when the cluster is known to have mixed pre-1.1/post-1.1 versions. That don't necessarily mean we would require an upgrade to 1.0.8+ before upgrading to 1.1, but it would mean that for all those that does upgrade from 1.0.8 (possibly a majority of users), we're being more user friendly.

        Show
        Sylvain Lebresne added a comment - Reattaching the two logs with only the 1.1 patch applied. Putting asides all the truncation related EOFException for now: on node1 (the 1.1 node), we do see a "Can't accept schema migrations from Cassandra versions previous to 1.1, please update first", though I suppose it would be nice to avoid the 'Fatal exception' and stacktrace that tends to scare people. on node2, we still have "UnsupportedOperationException: Not a time-based UUID" exceptions. That being said, even if we fix all those exceptions (which we should, there no question on that), it is still the case that the schema change is applied to the 1.0 nodes and no error is returned to the user (that is, outside of the log file), but the added column family is not really usable (the user will get timeouts) at least until the cluster upgrade is complete. Even if the user do watch the log and see the error (and will probably assume the creation failed), the column family is still created. That's not really user friendly. So I still think it would be a good idea to add code to 1.0 and 1.1 to refuse upfront schema changes when the cluster is known to have mixed pre-1.1/post-1.1 versions. That don't necessarily mean we would require an upgrade to 1.0.8+ before upgrading to 1.1, but it would mean that for all those that does upgrade from 1.0.8 (possibly a majority of users), we're being more user friendly.
        Hide
        Pavel Yaskevich added a comment - - edited

        I have found the problem with EOFException in Truncate - it was caused by MIGRATION_REQUEST misinterpreted by 1.0 as TRUNCATE message, I have added a check into MM.rectifySchema(UUID, InetAddress) to skip that request if node with changed schema is older than 1.1. Two patches in combination now return no exceptions but node 1.0 without patch would still throw UnsupportedOperationException because Gossiper always propagates passive schema announce.

        Edit: I have also changed "Can't accept schema migrations from Cassandra versions previous to 1.1, please update first." in 1.1 to be log error instead of IOException.

        Show
        Pavel Yaskevich added a comment - - edited I have found the problem with EOFException in Truncate - it was caused by MIGRATION_REQUEST misinterpreted by 1.0 as TRUNCATE message, I have added a check into MM.rectifySchema(UUID, InetAddress) to skip that request if node with changed schema is older than 1.1. Two patches in combination now return no exceptions but node 1.0 without patch would still throw UnsupportedOperationException because Gossiper always propagates passive schema announce. Edit: I have also changed "Can't accept schema migrations from Cassandra versions previous to 1.1, please update first." in 1.1 to be log error instead of IOException.
        Hide
        Sylvain Lebresne added a comment -

        it was caused by MIGRATION_REQUEST misinterpreted by 1.0 as TRUNCATE message

        That shouldn't be. In the StorageService.Verb enumeration, MIGRATION_REQUEST needs to be moved to the end of the enum. Otherwise lots of stuff will be misinterpreted.

        Show
        Sylvain Lebresne added a comment - it was caused by MIGRATION_REQUEST misinterpreted by 1.0 as TRUNCATE message That shouldn't be. In the StorageService.Verb enumeration, MIGRATION_REQUEST needs to be moved to the end of the enum. Otherwise lots of stuff will be misinterpreted.
        Hide
        Pavel Yaskevich added a comment -

        yeah, it seems like I have missed the comment about that, but fact is fact it was misinterpreted because of that. Have placed MIGRATION_REQUEST at the end of the Verb enum in updated v2.

        Show
        Pavel Yaskevich added a comment - yeah, it seems like I have missed the comment about that, but fact is fact it was misinterpreted because of that. Have placed MIGRATION_REQUEST at the end of the Verb enum in updated v2.
        Hide
        Pavel Yaskevich added a comment -

        fixed the comment about migration request.

        Show
        Pavel Yaskevich added a comment - fixed the comment about migration request.
        Hide
        Sylvain Lebresne added a comment -

        Ok, it does leave some InsupportedOperationException on the 1.0 but it clearly improve things and the misplaced MIGRATION_REQUEST is a real bug so I've committed v2 (putting MIGRATION_REQUEST before the UNUSED ones as that's where it should be added). I'll open another issue to see if we can make smooth even more the case where people (mistakenly) do schema update in a mixed cluster.

        Show
        Sylvain Lebresne added a comment - Ok, it does leave some InsupportedOperationException on the 1.0 but it clearly improve things and the misplaced MIGRATION_REQUEST is a real bug so I've committed v2 (putting MIGRATION_REQUEST before the UNUSED ones as that's where it should be added). I'll open another issue to see if we can make smooth even more the case where people (mistakenly) do schema update in a mixed cluster.
        Hide
        Tyler Patterson added a comment -

        I removed the dtest because it always fails, and will always fail in a properly running cluster.

        Show
        Tyler Patterson added a comment - I removed the dtest because it always fails, and will always fail in a properly running cluster.

          People

          • Assignee:
            Pavel Yaskevich
            Reporter:
            Tyler Patterson
            Reviewer:
            Sylvain Lebresne
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development