Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9208

ReplicationLogCleaner slow at large scale

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.96.0
    • Component/s: Replication
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      The FileCleanerDelegate interface has changed to operate on a batch of files at a time. Any custom implementations that implemented the interface directly for cleaning HLog's (specified by hbase.master.logcleaner.plugins) or HFile's (specified by hbase.master.hfilecleaner.plugins) will have to be updated to implement the new interface or extend one of the included abstract base classes.
      Show
      The FileCleanerDelegate interface has changed to operate on a batch of files at a time. Any custom implementations that implemented the interface directly for cleaning HLog's (specified by hbase.master.logcleaner.plugins) or HFile's (specified by hbase.master.hfilecleaner.plugins) will have to be updated to implement the new interface or extend one of the included abstract base classes.
    • Tags:
      0.96notable

      Description

      At a large scale the ReplicationLogCleaner fails to clean up .oldlogs as fast as the cluster is producing them. For each old HLog file that has been replicated and should be deleted the ReplicationLogCleaner checks every replication queue in ZooKeeper before removing it. This means that as a cluster scales up the number of files to delete scales as well as the time to delete each file so the cleanup chore scales quadratically. In our case it reached the point where the oldlogs were growing faster than they were being cleaned up.

      We're now running with a patch that allows the ReplicationLogCleaner to refresh its list of files in the replication queues from ZooKeeper just once for each batch of files the CleanerChore wants to evaluate.

      I'd propose updating FileCleanerDelegate to take a List<FileStatus> rather than a single one at a time. This would allow file cleaners that check an external resource for references such as ZooKeeper (for ReplicationLogCleaner) or HDFS (for SnapshotLogCleaner which looks like it may also have similar trouble at scale) to load those references once per batch rather than for every log.

      1. HBASE-9208.patch
        24 kB
        Dave Latham
      2. HBASE-9208-0.94.patch
        27 kB
        Dave Latham
      3. HBASE-9208-0.94-v2.patch
        27 kB
        Dave Latham
      4. HBASE-9208-v2.patch
        24 kB
        Dave Latham
      5. HBASE-9208-v3.patch
        26 kB
        Dave Latham

        Issue Links

          Activity

          Hide
          jdcryans Jean-Daniel Cryans added a comment -

          +1

          Show
          jdcryans Jean-Daniel Cryans added a comment - +1
          Hide
          lhofhansl Lars Hofhansl added a comment -

          Sounds good. Can you post the patch, Dave Latham?

          Show
          lhofhansl Lars Hofhansl added a comment - Sounds good. Can you post the patch, Dave Latham ?
          Hide
          davelatham Dave Latham added a comment -

          My patch is somewhat old now and not what I think would make for the nicest approach going forward any way (rather than passing a List<FileStatus> it added startBatch and endBatch methods which I found simpler to adapt to the existing recursive logic in CleanerChore). Since the approach sounds good to you guys I will work up a trunk patch.

          Show
          davelatham Dave Latham added a comment - My patch is somewhat old now and not what I think would make for the nicest approach going forward any way (rather than passing a List<FileStatus> it added startBatch and endBatch methods which I found simpler to adapt to the existing recursive logic in CleanerChore). Since the approach sounds good to you guys I will work up a trunk patch.
          Hide
          davelatham Dave Latham added a comment -

          The CleanerChore is shared code between cleaning up HLogs in .oldlogs (a single directory with many files) and cleaning up HFiles in archive (which can have many nested subdirectories). The proposal here assumes that as long as we get file listings from HDFS first and then check for references to those files second that we're safe to delete the file if there are no such references (of course the existing code has the same assumption). So that leaves a choice:

          (a) We can do this batch for each directory at a time only which should still solve the issue for HLogs (so long as they stay in a flat directory) but wouldn't allow the HFile archive to use as much of the optimization. Or:

          (b) We can first do a full recursive load of all files in the base directory, and call a batch filter across all those files. Then, in order to remove now empty subdirectories we need to do some bookkeeping to associate which ones we believe may now be empty (of course some subdirectories may have new entries created in the mean time).

          Since I'm focused for the moment on HLogs I'm planning on sticking with the simpler (a) for now, unless I hear some clamoring for the fuller solution (b).

          One other question occurred when doing some work on CleanerChore for this. If an IOException occurs during deleting an entry in the top level directory the chore continues attempting to delete the other entries. However, if an IOException is thrown deleting an entry in a subdirectory it aborts trying other entries in the subdirectory. I'd prefer to see consistency (and it would make it easier to share the code) if we either give up the entire current chore iteration on an unexpected file system IOException (i.e. we still tolerate IOException when deleting a non-empty directory) or alternately attempt to continue crawling all subentries. Does anyone have an opinion on which is better (or disagree about making the behavior consistent between top level entries and sublevel entries)?

          Show
          davelatham Dave Latham added a comment - The CleanerChore is shared code between cleaning up HLogs in .oldlogs (a single directory with many files) and cleaning up HFiles in archive (which can have many nested subdirectories). The proposal here assumes that as long as we get file listings from HDFS first and then check for references to those files second that we're safe to delete the file if there are no such references (of course the existing code has the same assumption). So that leaves a choice: (a) We can do this batch for each directory at a time only which should still solve the issue for HLogs (so long as they stay in a flat directory) but wouldn't allow the HFile archive to use as much of the optimization. Or: (b) We can first do a full recursive load of all files in the base directory, and call a batch filter across all those files. Then, in order to remove now empty subdirectories we need to do some bookkeeping to associate which ones we believe may now be empty (of course some subdirectories may have new entries created in the mean time). Since I'm focused for the moment on HLogs I'm planning on sticking with the simpler (a) for now, unless I hear some clamoring for the fuller solution (b). One other question occurred when doing some work on CleanerChore for this. If an IOException occurs during deleting an entry in the top level directory the chore continues attempting to delete the other entries. However, if an IOException is thrown deleting an entry in a subdirectory it aborts trying other entries in the subdirectory. I'd prefer to see consistency (and it would make it easier to share the code) if we either give up the entire current chore iteration on an unexpected file system IOException (i.e. we still tolerate IOException when deleting a non-empty directory) or alternately attempt to continue crawling all subentries. Does anyone have an opinion on which is better (or disagree about making the behavior consistent between top level entries and sublevel entries)?
          Hide
          lhofhansl Lars Hofhansl added a comment -

          +1 for (a). I assume part of the problem is that we repeatedly check ZK to check whether a file maybe removed. For the HFile archiver we do not need to do that.

          Show
          lhofhansl Lars Hofhansl added a comment - +1 for (a). I assume part of the problem is that we repeatedly check ZK to check whether a file maybe removed. For the HFile archiver we do not need to do that.
          Hide
          jesse_yates Jesse Yates added a comment -

          +1 also on (a) for Lars' reasons

          Seems like it would be best to keep trying for all subdirectories, rather than bailing out.

          Is there any reason why bailing early would be a good thing? I guess it depends on the IOException (which HDFS doesn't describe) - if its an IOException because the file doesn't exist, then keep going; however, if its because HDFS is down, then we should just stop early and save ourselves the effort. Seems like there isn't really a downside to continuing the other directories, as long as we check for stopping/interruptions along the way (b/c someone else on the server found the FS was down).

          Show
          jesse_yates Jesse Yates added a comment - +1 also on (a) for Lars' reasons Seems like it would be best to keep trying for all subdirectories, rather than bailing out. Is there any reason why bailing early would be a good thing? I guess it depends on the IOException (which HDFS doesn't describe) - if its an IOException because the file doesn't exist, then keep going; however, if its because HDFS is down, then we should just stop early and save ourselves the effort. Seems like there isn't really a downside to continuing the other directories, as long as we check for stopping/interruptions along the way (b/c someone else on the server found the FS was down).
          Hide
          davelatham Dave Latham added a comment -

          Attaching HBASE-9208.patch with the following changes:

          • Updated FileCleanerDelegate from boolean isFileDeletable(FileStatus fStat); to Iterable<FileStatus> getDeletableFiles(Iterable<FileStatus> files);
          • Added an abstract BaseFileCleanerDelegate that implements the new batch method in terms of the old method so that existing cleaners will work by extending this base class.
          • Updated CleanerChore to make a single call to each cleaner with a batch of all files for each directory (option (a) in coment above). Also now catches and logs unexpected IOExceptions for each entry of subdirectories rather than aborting to be consistent with behavior of the top level directory as mentioned above and suggested by Jesse.

          It is also available on review board at:
          https://reviews.apache.org/r/13608/diff/

          Have not updated SnapshotLogCleaner to take advantage of the batching interface. I intend to create a separate JIRA for that.

          A couple more questions arose during the work:

          • ReplicationLogCleaner.stop contains:
            // Not sure why we're deleting a connection that we never acquired or used
            HConnectionManager.deleteConnection(this.getConf());
            

            Is this correct with the latest work on connection management?

          • The cleaner hierarchy may now be deeper than needed. In particular there is a FileCleanerDelegate interface which is implemented by a BaseFileCleanerDelegate which is subclased by each of BaseHFileCleanerDelegate (which adds only the stopped field) and BaseLogCleanerDelegate (which includes a deprecated isLogDeletable method). In turn these are subclassed by the concrete implementations. Should the base classes be consolidated?

          Reviews and input would be greatly appreciated.

          Show
          davelatham Dave Latham added a comment - Attaching HBASE-9208 .patch with the following changes: Updated FileCleanerDelegate from boolean isFileDeletable(FileStatus fStat); to Iterable<FileStatus> getDeletableFiles(Iterable<FileStatus> files); Added an abstract BaseFileCleanerDelegate that implements the new batch method in terms of the old method so that existing cleaners will work by extending this base class. Updated CleanerChore to make a single call to each cleaner with a batch of all files for each directory (option (a) in coment above). Also now catches and logs unexpected IOExceptions for each entry of subdirectories rather than aborting to be consistent with behavior of the top level directory as mentioned above and suggested by Jesse. It is also available on review board at: https://reviews.apache.org/r/13608/diff/ Have not updated SnapshotLogCleaner to take advantage of the batching interface. I intend to create a separate JIRA for that. A couple more questions arose during the work: ReplicationLogCleaner.stop contains: // Not sure why we're deleting a connection that we never acquired or used HConnectionManager.deleteConnection(this.getConf()); Is this correct with the latest work on connection management? The cleaner hierarchy may now be deeper than needed. In particular there is a FileCleanerDelegate interface which is implemented by a BaseFileCleanerDelegate which is subclased by each of BaseHFileCleanerDelegate (which adds only the stopped field) and BaseLogCleanerDelegate (which includes a deprecated isLogDeletable method). In turn these are subclassed by the concrete implementations. Should the base classes be consolidated? Reviews and input would be greatly appreciated.
          Hide
          davelatham Dave Latham added a comment -

          Meant to add that the patch also updates ReplicationLogCleaner to use the new batching method to only load the ZK replication queues once per batch rather than for every file.

          Show
          davelatham Dave Latham added a comment - Meant to add that the patch also updates ReplicationLogCleaner to use the new batching method to only load the ZK replication queues once per batch rather than for every file.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12598300/HBASE-9208.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12598300/HBASE-9208.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6770//console This message is automatically generated.
          Hide
          davelatham Dave Latham added a comment -

          Attaching HBASE-9208-v2.patch.

          It contains some minor javadoc, comment, and formatting fixes and additions.

          Thanks, Ted, for the review.

          Show
          davelatham Dave Latham added a comment - Attaching HBASE-9208 -v2.patch. It contains some minor javadoc, comment, and formatting fixes and additions. Thanks, Ted, for the review.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12598781/HBASE-9208-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12598781/HBASE-9208-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6811//console This message is automatically generated.
          Hide
          davelatham Dave Latham added a comment -

          Attaching HBASE-9208-0.94.patch

          Here is a backport to 0.94.

          Show
          davelatham Dave Latham added a comment - Attaching HBASE-9208 -0.94.patch Here is a backport to 0.94.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12598799/HBASE-9208-0.94.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6814//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12598799/HBASE-9208-0.94.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6814//console This message is automatically generated.
          Hide
          lhofhansl Lars Hofhansl added a comment -

          Trunk patch looks. +1

          Show
          lhofhansl Lars Hofhansl added a comment - Trunk patch looks. +1
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          +1 as well.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - +1 as well.
          Hide
          lhofhansl Lars Hofhansl added a comment -

          For the 0.94 patch, why is this needed:

             @Deprecated
          -  public abstract boolean isLogDeletable(FileStatus fStat);
          +  public boolean isLogDeletable(FileStatus fStat) {
          +    return false;
          +  }
          

          Seems safer to leave the method abstract.

          Show
          lhofhansl Lars Hofhansl added a comment - For the 0.94 patch, why is this needed: @Deprecated - public abstract boolean isLogDeletable(FileStatus fStat); + public boolean isLogDeletable(FileStatus fStat) { + return false ; + } Seems safer to leave the method abstract.
          Hide
          davelatham Dave Latham added a comment -

          For the 0.94 patch, why is this needed:

          That is there because the ReplicationLogCleaner no longer implements isLogDeletable and instead implements getDeletableFiles directly. This brings up a good point that I had not carefully considered. What are the compatibility requirements for log cleaners? The current patches break compatibility and would require existing log cleaners to implement the changed interface or extend the BaseFileCleanerDelegate. This seems like a bad idea for 0.94, but probably ok for 0.96. What's the best alternative?

          Perhaps rather than changing FileCleanerDelegate we could introduce a new interface BatchFileCleanerDelegate or some such. Then configured cleaners that implement that interface can use it but others could still work by wrapping them. Anyone agree with that approach or have other suggestions?

          Show
          davelatham Dave Latham added a comment - For the 0.94 patch, why is this needed: That is there because the ReplicationLogCleaner no longer implements isLogDeletable and instead implements getDeletableFiles directly. This brings up a good point that I had not carefully considered. What are the compatibility requirements for log cleaners? The current patches break compatibility and would require existing log cleaners to implement the changed interface or extend the BaseFileCleanerDelegate. This seems like a bad idea for 0.94, but probably ok for 0.96. What's the best alternative? Perhaps rather than changing FileCleanerDelegate we could introduce a new interface BatchFileCleanerDelegate or some such. Then configured cleaners that implement that interface can use it but others could still work by wrapping them. Anyone agree with that approach or have other suggestions?
          Hide
          stack stack added a comment -

          Trunk patch looks good. Does what you have deployed look like this patch Dave Latham?

          It doesn't have unit tests. You are relying on existing tests to ensure that base functionality remains undisturbed?

          Nits:

          Rather than indent whole method, just reverse this test and return early: if (entries != null) {

          Is this right? Delete all files if not configured?

          + // all members of this class are null if replication is disabled,
          + // so we cannot filter the files
          if (this.getConf() == null)

          { - return true; + LOG.warn("Not configured - allowing all hlogs to be deleted"); + return files; }

          Otherwise, patch looks good to me.

          Show
          stack stack added a comment - Trunk patch looks good. Does what you have deployed look like this patch Dave Latham ? It doesn't have unit tests. You are relying on existing tests to ensure that base functionality remains undisturbed? Nits: Rather than indent whole method, just reverse this test and return early: if (entries != null) { Is this right? Delete all files if not configured? + // all members of this class are null if replication is disabled, + // so we cannot filter the files if (this.getConf() == null) { - return true; + LOG.warn("Not configured - allowing all hlogs to be deleted"); + return files; } Otherwise, patch looks good to me.
          Hide
          jdcryans Jean-Daniel Cryans added a comment -

          Is this right? Delete all files if not configured?

          What it means is that this log cleaner says ok for those files, but other log cleaners can say otherwise. As long as one cleaner in the chain says "keep it", we keep it.

          But I'm not sure about this WARN, it could just be removed:

          LOG.warn("Not configured - allowing all hlogs to be deleted");

          Show
          jdcryans Jean-Daniel Cryans added a comment - Is this right? Delete all files if not configured? What it means is that this log cleaner says ok for those files, but other log cleaners can say otherwise. As long as one cleaner in the chain says "keep it", we keep it. But I'm not sure about this WARN, it could just be removed: LOG.warn("Not configured - allowing all hlogs to be deleted");
          Hide
          davelatham Dave Latham added a comment -

          Thanks for the attention Stack, and JD.

          Trunk patch looks good. Does what you have deployed look like this patch Dave Latham?

          What we deployed is a bit different. Added a startBatch and endBatch to the FileCleanerDelegate. That made it a bit less clean of an interface, which also has some hidden dependencies on client behavior (otherwise would have race conditions). However it was a smaller change that we needed rapidly that didn't require a refactor of CleanerChore like this patch does. I like this patch better and look forward to running it if it looks good to others.

          It doesn't have unit tests. You are relying on existing tests to ensure that base functionality remains undisturbed?

          That's right.

          Rather than indent whole method, just reverse this test and return early: if (entries != null) {

          I guess it depends on whether you prefer having indentation or an extra return statement. I'll add the extra return since you spoke up.

          Is this right? Delete all files if not configured?
          ...
          What it means is that this log cleaner says ok for those files, but other log cleaners can say otherwise. As long as one cleaner in the chain says "keep it", we keep it.

          But I'm not sure about this WARN, it could just be removed:
          LOG.warn("Not configured - allowing all hlogs to be deleted");

          This case existed in the previous code with the same behavior that JD explains. However it can only occur if someone manually configures HBase to use the ReplicationLogCleaner but has replication disabled (normally the ReplicationLogCleaner is added programmatically in Replication.decorateMasterConfiguration when replication is enabled). The warning could prove useful to someone in such a case. However I will move it to the setConf method so it is only logged once at startup if there is concern over its level and verbosity.

          Show
          davelatham Dave Latham added a comment - Thanks for the attention Stack, and JD. Trunk patch looks good. Does what you have deployed look like this patch Dave Latham? What we deployed is a bit different. Added a startBatch and endBatch to the FileCleanerDelegate. That made it a bit less clean of an interface, which also has some hidden dependencies on client behavior (otherwise would have race conditions). However it was a smaller change that we needed rapidly that didn't require a refactor of CleanerChore like this patch does. I like this patch better and look forward to running it if it looks good to others. It doesn't have unit tests. You are relying on existing tests to ensure that base functionality remains undisturbed? That's right. Rather than indent whole method, just reverse this test and return early: if (entries != null) { I guess it depends on whether you prefer having indentation or an extra return statement. I'll add the extra return since you spoke up. Is this right? Delete all files if not configured? ... What it means is that this log cleaner says ok for those files, but other log cleaners can say otherwise. As long as one cleaner in the chain says "keep it", we keep it. But I'm not sure about this WARN, it could just be removed: LOG.warn("Not configured - allowing all hlogs to be deleted"); This case existed in the previous code with the same behavior that JD explains. However it can only occur if someone manually configures HBase to use the ReplicationLogCleaner but has replication disabled (normally the ReplicationLogCleaner is added programmatically in Replication.decorateMasterConfiguration when replication is enabled). The warning could prove useful to someone in such a case. However I will move it to the setConf method so it is only logged once at startup if there is concern over its level and verbosity.
          Hide
          davelatham Dave Latham added a comment -

          Attaching HBASE-9208-0.94-v2.patch and HBASE-9208-v3.patch.

          These are new patches for 0.94 and trunk, respectively. They implement the changes discussed above (move the WARN statement to ReplicationLogCleaner.setConf so it only moves once and add the guard clause / remove indentation in CleanerChore.checkAndDeleteEntries)

          Show
          davelatham Dave Latham added a comment - Attaching HBASE-9208 -0.94-v2.patch and HBASE-9208 -v3.patch. These are new patches for 0.94 and trunk, respectively. They implement the changes discussed above (move the WARN statement to ReplicationLogCleaner.setConf so it only moves once and add the guard clause / remove indentation in CleanerChore.checkAndDeleteEntries)
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600055/HBASE-9208-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600055/HBASE-9208-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/6908//console This message is automatically generated.
          Hide
          stack stack added a comment -

          Will commit today unless objection. Lars Hofhansl Ok to add to 0.94?

          Show
          stack stack added a comment - Will commit today unless objection. Lars Hofhansl Ok to add to 0.94?
          Hide
          jdcryans Jean-Daniel Cryans added a comment -

          +1, I'll also give it a spin on 0.95 today.

          Show
          jdcryans Jean-Daniel Cryans added a comment - +1, I'll also give it a spin on 0.95 today.
          Hide
          davelatham Dave Latham added a comment -

          I'm not sure that we should commit to 0.94 as is because of the potential to break compatibility for people who have added their own FileCleanerDelegate's for HFiles or HLogs. See my questions in comment #13745252:

          What are the compatibility requirements for log cleaners? The current patches break compatibility and would require existing log cleaners to implement the changed interface or extend the BaseFileCleanerDelegate. This seems like a bad idea for 0.94, but probably ok for 0.96. What's the best alternative?

          Perhaps rather than changing FileCleanerDelegate we could introduce a new interface BatchFileCleanerDelegate or some such. Then configured cleaners that implement that interface can use it but others could still work by wrapping them. Anyone agree with that approach or have other suggestions?

          Show
          davelatham Dave Latham added a comment - I'm not sure that we should commit to 0.94 as is because of the potential to break compatibility for people who have added their own FileCleanerDelegate's for HFiles or HLogs. See my questions in comment #13745252: What are the compatibility requirements for log cleaners? The current patches break compatibility and would require existing log cleaners to implement the changed interface or extend the BaseFileCleanerDelegate. This seems like a bad idea for 0.94, but probably ok for 0.96. What's the best alternative? Perhaps rather than changing FileCleanerDelegate we could introduce a new interface BatchFileCleanerDelegate or some such. Then configured cleaners that implement that interface can use it but others could still work by wrapping them. Anyone agree with that approach or have other suggestions?
          Hide
          stack stack added a comment -

          Will wait on the Jean-Daniel Cryans test run before committing.

          Can make new issue for whether to commit on 0.94

          Show
          stack stack added a comment - Will wait on the Jean-Daniel Cryans test run before committing. Can make new issue for whether to commit on 0.94
          Hide
          jdcryans Jean-Daniel Cryans added a comment -

          I rigged up a cluster to roll 3x more, tuned the TTL log cleaner to 0, then started writing massively to a master cluster and it seems like it was able to clean 1 log per millisecond. So again +1 from me.

          Show
          jdcryans Jean-Daniel Cryans added a comment - I rigged up a cluster to roll 3x more, tuned the TTL log cleaner to 0, then started writing massively to a master cluster and it seems like it was able to clean 1 log per millisecond. So again +1 from me.
          Hide
          stack stack added a comment -

          Committed to 0.95 branch and to trunk. Thanks for the patch Dave. I made a new issue, HBASE-9377, for backport to 0.94.

          Show
          stack stack added a comment - Committed to 0.95 branch and to trunk. Thanks for the patch Dave. I made a new issue, HBASE-9377 , for backport to 0.94.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in hbase-0.95 #503 (See https://builds.apache.org/job/hbase-0.95/503/)
          HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518465)

          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in hbase-0.95 #503 (See https://builds.apache.org/job/hbase-0.95/503/ ) HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518465) /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #4444 (See https://builds.apache.org/job/HBase-TRUNK/4444/)
          HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518466)

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #4444 (See https://builds.apache.org/job/HBase-TRUNK/4444/ ) HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518466) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in hbase-0.95-on-hadoop2 #277 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/277/)
          HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518465)

          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
          • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in hbase-0.95-on-hadoop2 #277 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/277/ ) HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518465) /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #701 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/701/)
          HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518466)

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #701 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/701/ ) HBASE-9208 ReplicationLogCleaner slow at large scale (stack: rev 1518466) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseFileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
          Hide
          davelatham Dave Latham added a comment -

          Thanks, JD, for the testing, and Stack for driving it in. I added a release note.

          Show
          davelatham Dave Latham added a comment - Thanks, JD, for the testing, and Stack for driving it in. I added a release note.
          Hide
          lars_francke Lars Francke added a comment -

          This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

          Show
          lars_francke Lars Francke added a comment - This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

            People

            • Assignee:
              davelatham Dave Latham
              Reporter:
              davelatham Dave Latham
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development