Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9158

Serious bug in cyclic replication

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.98.0, 0.95.1, 0.94.10
    • Fix Version/s: 0.98.0, 0.95.2, 0.94.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While studying the code for HBASE-7709, I found a serious bug in the current cyclic replication code. The problem is here in HRegion.doMiniBatchMutation:

            Mutation first = batchOp.operations[firstIndex].getFirst();
            txid = this.log.appendNoSync(regionInfo, this.htableDescriptor.getName(),
                     walEdit, first.getClusterId(), now, this.htableDescriptor);
      

      Now note that edits replicated from remote cluster and local edits might interleave in the WAL, we might also receive edit from multiple remote clusters. Hence that <walEdit> might have edits from many clusters in it, but all are just labeled with the clusterId of the first Mutation.

      Fixing this in doMiniBatchMutation seems tricky to do efficiently (imagine we get a batch with cluster1, cluster2, cluster1, cluster2, ..., in that case each edit would have to be its own batch). The coprocessor handling would also be difficult.

      The other option is create batches of Puts grouped by the cluster id in ReplicationSink.replicateEntries(...), this is not as general, but equally correct. This is the approach I would favor.

      Lastly this is very hard to verify in a unittest.

        Attachments

        1. 9158-trunk-v4.txt
          9 kB
          Lars Hofhansl
        2. 9158-trunk-v3.txt
          6 kB
          Lars Hofhansl
        3. 9158-trunk-v2.txt
          6 kB
          Lars Hofhansl
        4. 9158-trunk-v1.txt
          4 kB
          Lars Hofhansl
        5. 9158-0.94-v4.txt
          7 kB
          Lars Hofhansl
        6. 9158-0.94-v3.txt
          7 kB
          Lars Hofhansl
        7. 9158-0.94-v2.txt
          6 kB
          Lars Hofhansl
        8. 9158-0.94.txt
          2 kB
          Lars Hofhansl

          Activity

            People

            • Assignee:
              larsh Lars Hofhansl
              Reporter:
              larsh Lars Hofhansl
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: