Cassandra
  1. Cassandra
  2. CASSANDRA-4772

HintedHandoff fails to deliver hints after first repaired node

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.1.6
    • Component/s: Core
    • Labels:

      Description

      If some node has hints for a few nodes it will deliver hints only for the first one of them. After all hints delivery for the first node compaction process is started. After compaction all data from hints cf is removed.

      target fix for 1.2 version:

      diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
      index e5ff163..c02997e 100644
      --- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java
      +++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
      @@ -189,7 +189,7 @@ public class HintedHandOffManager implements HintedHandOffManagerMBean
               ArrayList<Descriptor> descriptors = new ArrayList<Descriptor>();
               for (SSTable sstable : hintStore.getSSTables())
                   descriptors.add(sstable.descriptor);
      -        return CompactionManager.instance.submitUserDefined(hintStore, descriptors, Integer.MAX_VALUE);
      +        return CompactionManager.instance.submitUserDefined(hintStore, descriptors, (int) System.currentTimeMillis() / 1000);
           }
      
       
           private static boolean pagingFinished(ColumnFamily hintColumnFamily, ByteBuffer startColumn)
      

      Can I expect to see that fix in 1.1.6 version?

      1. 4772-1.0.txt
        4 kB
        Jonathan Ellis
      2. cassandra-1.2-4772-hh_compact.txt
        4 kB
        Alexey Zotov

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          40s 1 Alexey Zotov 05/Oct/12 22:02
          Patch Available Patch Available Resolved Resolved
          13m 21s 1 Jonathan Ellis 05/Oct/12 22:15
          Resolved Resolved Reopened Reopened
          2d 18h 24m 1 Jonathan Ellis 08/Oct/12 16:40
          Reopened Reopened Resolved Resolved
          22h 14m 1 Jonathan Ellis 09/Oct/12 14:55
          Gavin made changes -
          Workflow patch-available, re-open possible [ 12753298 ] reopen-resolved, no closed status, patch-avail, testing [ 12758680 ]
          Gavin made changes -
          Workflow no-reopen-closed, patch-avail [ 12728550 ] patch-available, re-open possible [ 12753298 ]
          Jonathan Ellis made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Jonathan Ellis added a comment -

          committed fix to 1.1 and trunk

          Show
          Jonathan Ellis added a comment - committed fix to 1.1 and trunk
          Jonathan Ellis made changes -
          Attachment 4772-1.0.txt [ 12548358 ]
          Hide
          Jonathan Ellis added a comment -

          attaching backport of test to 1.0

          Show
          Jonathan Ellis added a comment - attaching backport of test to 1.0
          Jonathan Ellis made changes -
          Affects Version/s 1.1.0 [ 12317615 ]
          Affects Version/s 1.0.0 [ 12316349 ]
          Hide
          Jonathan Ellis added a comment -

          You're right, Alexey. Thanks for the followup.

          Thought at first this would affect 1.0 as well – compaction code is the same in 1.0.x – but the bug needs on both compaction w/ MAX_TIMESTAMP and CASSANDRA-3716.

          Show
          Jonathan Ellis added a comment - You're right, Alexey. Thanks for the followup. Thought at first this would affect 1.0 as well – compaction code is the same in 1.0.x – but the bug needs on both compaction w/ MAX_TIMESTAMP and CASSANDRA-3716 .
          Jonathan Ellis made changes -
          Fix Version/s 1.1.6 [ 12323257 ]
          Affects Version/s 1.0.0 [ 12316349 ]
          Affects Version/s 1.1.4 [ 12322490 ]
          Priority Blocker [ 1 ] Major [ 3 ]
          Reviewer jbellis
          Jonathan Ellis made changes -
          Resolution Invalid [ 6 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Alexey Zotov made changes -
          Attachment cassandra-1.2-4772-hh_compact.txt [ 12548223 ]
          Hide
          Alexey Zotov added a comment -

          That timestamp is passed to ColumnFamilyStore.removeDeleted() method. That method removes columns with tombstones and expired columns. All columns in "hints" column family are "expiring columns" and they should be removed if gcBefore is greater than localExpirationTime. We pass gcBefore as Integer.MAX_VALUE, so all columns should be deleted because of Integer.MAX_VALUE >> localExpirationTime.
          I've attached some test, that fails without the fix. Please take a look on it.

          Show
          Alexey Zotov added a comment - That timestamp is passed to ColumnFamilyStore.removeDeleted() method. That method removes columns with tombstones and expired columns. All columns in "hints" column family are "expiring columns" and they should be removed if gcBefore is greater than localExpirationTime. We pass gcBefore as Integer.MAX_VALUE, so all columns should be deleted because of Integer.MAX_VALUE >> localExpirationTime. I've attached some test, that fails without the fix. Please take a look on it.
          Jonathan Ellis made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 1.2.0 beta 2 [ 12323284 ]
          Resolution Invalid [ 6 ]
          Hide
          Jonathan Ellis added a comment -

          the timestamp is "time before which we can throw away deleted columns." columns that have not already been deleted are unaffected.

          Show
          Jonathan Ellis added a comment - the timestamp is "time before which we can throw away deleted columns." columns that have not already been deleted are unaffected.
          Alexey Zotov made changes -
          Link This issue relates to CASSANDRA-3972 [ CASSANDRA-3972 ]
          Alexey Zotov made changes -
          Field Original Value New Value
          Status Open [ 1 ] Patch Available [ 10002 ]
          Alexey Zotov created issue -

            People

            • Assignee:
              Alexey Zotov
              Reporter:
              Alexey Zotov
              Reviewer:
              Jonathan Ellis
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development