Cassandra
  1. Cassandra
  2. CASSANDRA-4772

HintedHandoff fails to deliver hints after first repaired node

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.1.6
    • Component/s: Core
    • Labels:

      Description

      If some node has hints for a few nodes it will deliver hints only for the first one of them. After all hints delivery for the first node compaction process is started. After compaction all data from hints cf is removed.

      target fix for 1.2 version:

      diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
      index e5ff163..c02997e 100644
      --- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java
      +++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
      @@ -189,7 +189,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
               ArrayList<Descriptor> descriptors = new ArrayList<Descriptor>();
               for (SSTable sstable : hintStore.getSSTables())
                   descriptors.add(sstable.descriptor);
      -        return CompactionManager.instance.submitUserDefined(hintStore, descriptors, Integer.MAX_VALUE);
      +        return CompactionManager.instance.submitUserDefined(hintStore, descriptors, (int) System.currentTimeMillis() / 1000);
           }
      
       
           private static boolean pagingFinished(ColumnFamily hintColumnFamily, ByteBuffer startColumn)
      

      Can I expect to see that fix in 1.1.6 version?

      1. cassandra-1.2-4772-hh_compact.txt
        4 kB
        Alexey Zotov
      2. 4772-1.0.txt
        4 kB
        Jonathan Ellis

        Issue Links

          Activity

          Hide
          Jonathan Ellis added a comment -

          committed fix to 1.1 and trunk

          Show
          Jonathan Ellis added a comment - committed fix to 1.1 and trunk
          Hide
          Jonathan Ellis added a comment -

          attaching backport of test to 1.0

          Show
          Jonathan Ellis added a comment - attaching backport of test to 1.0
          Hide
          Jonathan Ellis added a comment -

          You're right, Alexey. Thanks for the followup.

          Thought at first this would affect 1.0 as well – compaction code is the same in 1.0.x – but the bug needs on both compaction w/ MAX_TIMESTAMP and CASSANDRA-3716.

          Show
          Jonathan Ellis added a comment - You're right, Alexey. Thanks for the followup. Thought at first this would affect 1.0 as well – compaction code is the same in 1.0.x – but the bug needs on both compaction w/ MAX_TIMESTAMP and CASSANDRA-3716 .
          Hide
          Alexey Zotov added a comment -

          That timestamp is passed to ColumnFamilyStore.removeDeleted() method. That method removes columns with tombstones and expired columns. All columns in "hints" column family are "expiring columns" and they should be removed if gcBefore is greater than localExpirationTime. We pass gcBefore as Integer.MAX_VALUE, so all columns should be deleted because of Integer.MAX_VALUE >> localExpirationTime.
          I've attached some test, that fails without the fix. Please take a look on it.

          Show
          Alexey Zotov added a comment - That timestamp is passed to ColumnFamilyStore.removeDeleted() method. That method removes columns with tombstones and expired columns. All columns in "hints" column family are "expiring columns" and they should be removed if gcBefore is greater than localExpirationTime. We pass gcBefore as Integer.MAX_VALUE, so all columns should be deleted because of Integer.MAX_VALUE >> localExpirationTime. I've attached some test, that fails without the fix. Please take a look on it.
          Hide
          Jonathan Ellis added a comment -

          the timestamp is "time before which we can throw away deleted columns." columns that have not already been deleted are unaffected.

          Show
          Jonathan Ellis added a comment - the timestamp is "time before which we can throw away deleted columns." columns that have not already been deleted are unaffected.

            People

            • Assignee:
              Alexey Zotov
              Reporter:
              Alexey Zotov
              Reviewer:
              Jonathan Ellis
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development