Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7709

Infinite loop possible in Master/Master replication

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.6, 0.95.1
    • Fix Version/s: 0.98.0, 0.94.12, 0.96.0
    • Component/s: Replication
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We just discovered the following scenario:

      1. Cluster A and B are setup in master/master replication
      2. By accident we had Cluster C replicate to Cluster A.

      Now all edit originating from C will be bouncing between A and B. Forever!
      The reason is that when the edit come in from C the cluster ID is already set and won't be reset.

      We have a couple of options here:

      1. Optionally only support master/master (not cycles of more than two clusters). In that case we can always reset the cluster ID in the ReplicationSource. That means that now cycles > 2 will have the data cycle forever. This is the only option that requires no changes in the HLog format.
      2. Instead of a single cluster id per edit maintain a (unordered) set of cluster id that have seen this edit. Then in ReplicationSource we drop any edit that the sink has seen already. The is the cleanest approach, but it might need a lot of data stored per edit if there are many clusters involved.
      3. Maintain a configurable counter of the maximum cycle side we want to support. Could default to 10 (even maybe even just). Store a hop-count in the WAL and the ReplicationSource increases that hop-count on each hop. If we're over the max, just drop the edit.

        Attachments

        1. 7709-0.94-rev6.txt
          35 kB
          Lars Hofhansl
        2. 0.95-trunk-rev4.patch
          104 kB
          Vasu Mariyala
        3. HBASE-7709-rev5.patch
          37 kB
          Vasu Mariyala
        4. 0.95-trunk-rev3.patch
          102 kB
          Vasu Mariyala
        5. HBASE-7709-rev4.patch
          36 kB
          Vasu Mariyala
        6. 0.95-trunk-rev2.patch
          103 kB
          Vasu Mariyala
        7. HBASE-7709-rev3.patch
          36 kB
          Vasu Mariyala
        8. 0.95-trunk-rev1.patch
          86 kB
          Vasu Mariyala
        9. 095-trunk.patch
          86 kB
          Vasu Mariyala
        10. HBASE-7709-rev2.patch
          36 kB
          Vasu Mariyala
        11. HBASE-7709-rev1.patch
          16 kB
          Vasu Mariyala
        12. HBASE-7709.patch
          12 kB
          Vasu Mariyala

          Activity

            People

            • Assignee:
              vmariyala Vasu Mariyala
              Reporter:
              larsh Lars Hofhansl
            • Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: