Cassandra
  1. Cassandra
  2. CASSANDRA-3547

Race between cf flush and its secondary indexes flush

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.0.6
    • Component/s: Core
    • Labels:
      None

      Description

      When a CF with indexes is flushed, it's indexes are flushed too. In particular their memtable is switched, but without making the old memtable frozen. This can conflict with a concurrent flush of the index itself, as reported on the user list by Michael Vaknine:

      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 java.lang.AssertionError
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:671)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:745)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilyStore.java:750)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.index.keys.KeysIndex.forceBlockingFlush(KeysIndex.java:119)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.index.SecondaryIndexManager.flushIndexesBlocking(SecondaryIndexManager.java:258)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:123)
      TST-Cass2 ERROR [Thread-58] 2011-11-30 20:40:17,449 at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:151)
      
      1. 3547.patch
        2 kB
        Sylvain Lebresne

        Activity

        Hide
        Sylvain Lebresne added a comment -

        Patch to freeze every memtable we actually flush.

        Show
        Sylvain Lebresne added a comment - Patch to freeze every memtable we actually flush.
        Hide
        Sylvain Lebresne added a comment -

        Actually I see no reason this wouldn't affect 0.8. The patch is against 1.0 but it will be trivial to rebase to 0.8 if needed.

        Show
        Sylvain Lebresne added a comment - Actually I see no reason this wouldn't affect 0.8. The patch is against 1.0 but it will be trivial to rebase to 0.8 if needed.
        Hide
        Jonathan Ellis added a comment -

        +1, and I think we do need this for 0.8 as well

        Show
        Jonathan Ellis added a comment - +1, and I think we do need this for 0.8 as well
        Hide
        Jonathan Ellis added a comment -

        However... 0.8 is stable enough, and this bug is rare enough, and flush code is tricky enough, that maybe we should leave it be. I could go either way.

        Show
        Jonathan Ellis added a comment - However... 0.8 is stable enough, and this bug is rare enough, and flush code is tricky enough, that maybe we should leave it be. I could go either way.
        Hide
        Sylvain Lebresne added a comment -

        You know what, I agree on 0.8. It's the first time anyone reports this and it's not like it corrupt data or anything bad, so I've committed to 1.0 only.

        Show
        Sylvain Lebresne added a comment - You know what, I agree on 0.8. It's the first time anyone reports this and it's not like it corrupt data or anything bad, so I've committed to 1.0 only.

          People

          • Assignee:
            Sylvain Lebresne
            Reporter:
            Sylvain Lebresne
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development