Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-22784

OldWALs not cleared in a replication slave cluster (cyclic replication bw 2 clusters)

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.4.9, 1.4.10
    • 1.5.0, 1.4.11
    • regionserver, Replication
    • None
    • Reviewed

    Description

      When a cluster is passive (receiving edits only via replication) in a cyclic replication setup of 2 clusters, OldWALs size keeps on growing. On analysing, we observed the following behaviour.

      1. New entry is added to WAL (Edit replicated from other cluster).
      2. ReplicationSourceWALReaderThread(RSWALRT) reads and applies the configured filters (due to cyclic replication setup, ClusterMarkingEntryFilter discards new entry from other cluster).
      3. Entry is null, RSWALRT neither updates the batch stats (WALEntryBatch.lastWalPosition) nor puts it in the entryBatchQueue.
      4. ReplicationSource thread is blocked in entryBachQueue.take().
      5. So ReplicationSource#updateLogPosition has never invoked and WAL file is never cleared from ReplicationQueue.
      6. Hence LogCleaner on the master, doesn't deletes the oldWAL files from hadoop.

      NOTE: When a new edit is added via hbase-client, ReplicationSource thread process and clears the oldWAL files from replication queues and hence master cleans up the WALs

      Please provide us a solution

       

      Attachments

        1. HBASE-22784.branch-1.004.patch
          10 kB
          Wellington Chevreuil
        2. HBASE-22784.branch-1.003.patch
          10 kB
          Wellington Chevreuil
        3. HBASE-22784.branch-1.002.patch
          7 kB
          Wellington Chevreuil
        4. HBASE-22784.branch-1.001.patch
          5 kB
          Wellington Chevreuil

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wchevreuil Wellington Chevreuil
            solvannan Solvannan R M
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment