Cassandra
  1. Cassandra
  2. CASSANDRA-3547

Race between cf flush and its secondary indexes flush

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.0.6
    • Component/s: Core
    • Labels:
      None

      Description

      When a CF with indexes is flushed, it's indexes are flushed too. In particular their memtable is switched, but without making the old memtable frozen. This can conflict with a concurrent flush of the index itself, as reported on the user list by Michael Vaknine:

      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 java.lang.AssertionError
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:671)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:745)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilyStore.java:750)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.index.keys.KeysIndex.forceBlockingFlush(KeysIndex.java:119)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.index.SecondaryIndexManager.flushIndexesBlocking(SecondaryIndexManager.java:258)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:123)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:151)
      
      1. 3547.patch
        2 kB
        Sylvain Lebresne

        Activity

        Sylvain Lebresne created issue -
        Hide
        Sylvain Lebresne added a comment -

        Patch to freeze every memtable we actually flush.

        Show
        Sylvain Lebresne added a comment - Patch to freeze every memtable we actually flush.
        Sylvain Lebresne made changes -
        Field Original Value New Value
        Attachment 3547.patch [ 12505754 ]
        Sylvain Lebresne made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Sylvain Lebresne added a comment -

        Actually I see no reason this wouldn't affect 0.8. The patch is against 1.0 but it will be trivial to rebase to 0.8 if needed.

        Show
        Sylvain Lebresne added a comment - Actually I see no reason this wouldn't affect 0.8. The patch is against 1.0 but it will be trivial to rebase to 0.8 if needed.
        Sylvain Lebresne made changes -
        Fix Version/s 0.8.9 [ 12319155 ]
        Affects Version/s 0.8.0 [ 12316403 ]
        Affects Version/s 1.0.0 [ 12316349 ]
        Hide
        Jonathan Ellis added a comment -

        +1, and I think we do need this for 0.8 as well

        Show
        Jonathan Ellis added a comment - +1, and I think we do need this for 0.8 as well
        Hide
        Jonathan Ellis added a comment -

        However... 0.8 is stable enough, and this bug is rare enough, and flush code is tricky enough, that maybe we should leave it be. I could go either way.

        Show
        Jonathan Ellis added a comment - However... 0.8 is stable enough, and this bug is rare enough, and flush code is tricky enough, that maybe we should leave it be. I could go either way.
        Hide
        Sylvain Lebresne added a comment -

        You know what, I agree on 0.8. It's the first time anyone reports this and it's not like it corrupt data or anything bad, so I've committed to 1.0 only.

        Show
        Sylvain Lebresne added a comment - You know what, I agree on 0.8. It's the first time anyone reports this and it's not like it corrupt data or anything bad, so I've committed to 1.0 only.
        Sylvain Lebresne made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Reviewer jbellis
        Fix Version/s 0.8.9 [ 12319155 ]
        Resolution Fixed [ 1 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12644147 ] patch-available, re-open possible [ 12749300 ]
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12749300 ] reopen-resolved, no closed status, patch-avail, testing [ 12756920 ]

          People

          • Assignee:
            Sylvain Lebresne
            Reporter:
            Sylvain Lebresne
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development