HBase
  1. HBase
  2. HBASE-5199

Delete out of TTL store files before compaction selection

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      set hbase.store.delete.expired.storefile as true to enable the expired store file deletion

      Description

      Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
      In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.

      1. ASF.LICENSE.NOT.GRANTED--D1311.1.patch
        5 kB
        Phabricator
      2. ASF.LICENSE.NOT.GRANTED--D1311.2.patch
        5 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--D1311.3.patch
        5 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--D1311.4.patch
        7 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--D1311.5.patch
        8 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--D1311.5.patch
        8 kB
        Phabricator
      7. hbase-5199.patch
        12 kB
        Liyin Tang

        Issue Links

          Activity

          Hide
          Lars Hofhansl added a comment -

          Need to pay some special attention to specific delete-marker TTL that FB added (i.e. a delete marker could outlive the TTL of a store). Can't find the jira for that right, but it was committed relatively recently.

          Show
          Lars Hofhansl added a comment - Need to pay some special attention to specific delete-marker TTL that FB added (i.e. a delete marker could outlive the TTL of a store). Can't find the jira for that right, but it was committed relatively recently.
          Hide
          Kannan Muthukkaruppan added a comment -

          I think Lars is referring to HBASE-4721 by Prakash. Yes, worth making sure it plays well with that – though I suspect it should work out just fine.

          Show
          Kannan Muthukkaruppan added a comment - I think Lars is referring to HBASE-4721 by Prakash. Yes, worth making sure it plays well with that – though I suspect it should work out just fine.
          Hide
          Liyin Tang added a comment -

          Thanks Lars and Kannan. I will double check this.

          Show
          Liyin Tang added a comment - Thanks Lars and Kannan. I will double check this.
          Hide
          Phabricator added a comment -

          Liyin requested code review of "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".
          Reviewers: Kannan, JIRA

          Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
          In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.

          TEST PLAN
          TestStore

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java

          Show
          Phabricator added a comment - Liyin requested code review of " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Reviewers: Kannan, JIRA Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. TEST PLAN TestStore REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Phabricator added a comment -

          Kannan has added reviewers to the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".
          Added Reviewers: khemani, aaiyer, Karthik

          adding a few more reviewers in case someone else can get to this earlier.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Kannan has added reviewers to the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Added Reviewers: khemani, aaiyer, Karthik adding a few more reviewers in case someone else can get to this earlier. REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          tedyu has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1053 Should this check be moved to line 1016 ?
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1024 Changing 'because' to 'where' would make the sentence more readable

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - tedyu has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1053 Should this check be moved to line 1016 ? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1024 Changing 'because' to 'where' would make the sentence more readable REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          Karthik has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          Havent reviewed the test code yet...

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1039 Do we need this assignment? Can we make the condition:

          if (nonExpiredFiles.size() < currentStoreFiles.size())

          { //... }

          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1017 Should we exclude files contained in filesCompacting? Otherwise we could end up deleting a file which is currently being compacted...

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Karthik has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Havent reviewed the test code yet... INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1039 Do we need this assignment? Can we make the condition: if (nonExpiredFiles.size() < currentStoreFiles.size()) { //... } src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1017 Should we exclude files contained in filesCompacting? Otherwise we could end up deleting a file which is currently being compacted... REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          Liyin updated the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".
          Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik

          Address Karthik's comments.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java

          Show
          Phabricator added a comment - Liyin updated the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik Address Karthik's comments. REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Phabricator added a comment -

          Liyin updated the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".
          Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik

          Fix a typo.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java

          Show
          Phabricator added a comment - Liyin updated the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik Fix a typo. REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Phabricator added a comment -

          Karthik has accepted the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          LGTM!

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Karthik has accepted the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". LGTM! REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          khemani has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 should Readers be notified via notifyChangeReadersObservers()?
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 Can this be done inside a readLock() (for most of the compactions no storefiles will actually get removed)

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - khemani has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 should Readers be notified via notifyChangeReadersObservers()? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 Can this be done inside a readLock() (for most of the compactions no storefiles will actually get removed) REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          Liyin has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 Since this function is to change <this.storeFiles> by deleting the expired storeFiles, writeLock is needed here.
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 Good Point ! I will update this.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Liyin has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 Since this function is to change <this.storeFiles> by deleting the expired storeFiles, writeLock is needed here. src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 Good Point ! I will update this. REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          tedyu has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 Currently notifyChangedReadersObservers() is called outside lock.writeLock().
          Please add the call after line 1044.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - tedyu has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 Currently notifyChangedReadersObservers() is called outside lock.writeLock(). Please add the call after line 1044. REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          khemani has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 find out the set of files to be deleted in a read lock. If that set is non-empty, then acquire the write lock and remove the to-be-deleted set of files (if they still exist) from the storefiles set.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - khemani has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 find out the set of files to be deleted in a read lock. If that set is non-empty, then acquire the write lock and remove the to-be-deleted set of files (if they still exist) from the storefiles set. REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          Liyin has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          @khemani: I see your point, which inspire me to a new idea: When holding a read lock for selection compaction candidates, it should also select the expired deletion store file candidates. After when processing completeCompaction logic within the write lock, it will delete the expired store files.

          So this approach should NOT introduce any potential dead lock risk since there is no read or write lock introduced by this feature at all.

          @tedyu: Thanks for pointing it out. And no additional notifyChangedReadersObservers() is needed if implementing in this way.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Liyin has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". @khemani: I see your point, which inspire me to a new idea: When holding a read lock for selection compaction candidates, it should also select the expired deletion store file candidates. After when processing completeCompaction logic within the write lock, it will delete the expired store files. So this approach should NOT introduce any potential dead lock risk since there is no read or write lock introduced by this feature at all. @tedyu: Thanks for pointing it out. And no additional notifyChangedReadersObservers() is needed if implementing in this way. REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Ted Yu added a comment -

          This becomes interesting, Liyin.
          Currently we have:

                    override = region.getCoprocessorHost().preCompactSelection(
                        this, candidates);
          

          If I understand your idea correctly, a sub-list of candidates would be extracted for deletion.
          Do you think that we should expose selection of expired store files through coprocessor ?

          Show
          Ted Yu added a comment - This becomes interesting, Liyin. Currently we have: override = region.getCoprocessorHost().preCompactSelection( this , candidates); If I understand your idea correctly, a sub-list of candidates would be extracted for deletion. Do you think that we should expose selection of expired store files through coprocessor ?
          Hide
          Phabricator added a comment -

          Liyin updated the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".
          Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik

          During the minor compaction selection time, if there is any expired store files, these expired store files will be selected to compact directly.
          Since these files have already expired, there would be no-ops during the compaction time and these files will be deleted after the compaction.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java

          Show
          Phabricator added a comment - Liyin updated the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik During the minor compaction selection time, if there is any expired store files, these expired store files will be selected to compact directly. Since these files have already expired, there would be no-ops during the compaction time and these files will be deleted after the compaction. REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Phabricator added a comment -

          Liyin has requested a review of the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          Hi @Karthik,
          The new diff have changed the logic a little bit. There is no need to get the write lock to do the deletion job for these expired store files. It would be much easier to put these expired store files directly to the compaction, which will do the deletion job for free.

          Would you mind reviewing it again? Thanks

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Liyin has requested a review of the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Hi @Karthik, The new diff have changed the logic a little bit. There is no need to get the write lock to do the deletion job for these expired store files. It would be much easier to put these expired store files directly to the compaction, which will do the deletion job for free. Would you mind reviewing it again? Thanks REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          Karthik has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1118 Could we return a new CompactSelection object instead of modifying the existing one and returning a boolean? We can return null or an empty compact selection if there are no such files...

          CompactSelection expiredFiles = compactSelection.selectExpiredStoreFiles(...)

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Karthik has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1118 Could we return a new CompactSelection object instead of modifying the existing one and returning a boolean? We can return null or an empty compact selection if there are no such files... CompactSelection expiredFiles = compactSelection.selectExpiredStoreFiles(...) REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          Liyin updated the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".
          Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik

          Address @Karthik's comments.

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java

          Show
          Phabricator added a comment - Liyin updated the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik Address @Karthik's comments. REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Phabricator added a comment -

          Karthik has accepted the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          Looks good!

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - Karthik has accepted the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Looks good! REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Ted Yu added a comment -

          @Liyin:
          Can you try adding JIRA to the CCs ?

          I have to deal with 6 copies of each review email in my InBox.

          Thanks

          Show
          Ted Yu added a comment - @Liyin: Can you try adding JIRA to the CCs ? I have to deal with 6 copies of each review email in my InBox. Thanks
          Hide
          Phabricator added a comment -

          Liyin added you to the CC list for the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".
          Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik

          Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions.
          In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection.

          TEST PLAN
          TestStore

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java

          Show
          Phabricator added a comment - Liyin added you to the CC list for the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. TEST PLAN TestStore REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12512236/HBASE-5199.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/860//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512236/HBASE-5199.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/860//console This message is automatically generated.
          Hide
          Liyin Tang added a comment -

          Ping committers !

          Show
          Liyin Tang added a comment - Ping committers !
          Hide
          Ted Yu added a comment -

          HBASE-5199.patch doesn't apply cleanly:

          3 out of 6 hunks FAILED -- saving rejects to file src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java.rej
          patching file src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          Reversed (or previously applied) patch detected!  Assume -R? [n] 
          Apply anyway? [n] 
          Skipping patch.
          3 out of 3 hunks ignored -- saving rejects to file src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java.rej
          

          Can you rebase patch ?

          Show
          Ted Yu added a comment - HBASE-5199 .patch doesn't apply cleanly: 3 out of 6 hunks FAILED -- saving rejects to file src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java.rej patching file src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Reversed (or previously applied) patch detected! Assume -R? [n] Apply anyway? [n] Skipping patch. 3 out of 3 hunks ignored -- saving rejects to file src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java.rej Can you rebase patch ?
          Hide
          Liyin Tang added a comment -

          The new patch is rebased on the latest trunk and all the unit tests are passed.

          Show
          Liyin Tang added a comment - The new patch is rebased on the latest trunk and all the unit tests are passed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12514057/hbase-5199.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.TestReplicationPeer
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock
          org.apache.hadoop.hbase.TestDrainingServer
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/934//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/934//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/934//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514057/hbase-5199.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplicationPeer org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.TestDrainingServer org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/934//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/934//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/934//console This message is automatically generated.
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          @Liyin: could you please post the rebased version of the patch so I can commit?

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". @Liyin: could you please post the rebased version of the patch so I can commit? REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          @Liyin: could you please post the rebased version of the patch so I can commit?

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". @Liyin: could you please post the rebased version of the patch so I can commit? REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          @Liyin: could you please post the rebased version of the patch so I can commit?

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". @Liyin: could you please post the rebased version of the patch so I can commit? REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          @Liyin: could you please post the rebased version of the patch so I can commit?

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". @Liyin: could you please post the rebased version of the patch so I can commit? REVISION DETAIL https://reviews.facebook.net/D1311
          Hide
          Ted Yu added a comment -

          The test failures were not related to this patch.

          Integrated to TRUNK.

          Thanks for the patch Liyin.

          Thanks for the review Karthick, Prakash and Mikhail.

          Show
          Ted Yu added a comment - The test failures were not related to this patch. Integrated to TRUNK. Thanks for the patch Liyin. Thanks for the review Karthick, Prakash and Mikhail.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2659 (See https://builds.apache.org/job/HBase-TRUNK/2659/)
          HBASE-5199 Differential revision 1311 Delete out of TTL store files before compaction selection (Liyin)

          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2659 (See https://builds.apache.org/job/HBase-TRUNK/2659/ ) HBASE-5199 Differential revision 1311 Delete out of TTL store files before compaction selection (Liyin) tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #109 (See https://builds.apache.org/job/HBase-TRUNK-security/109/)
          HBASE-5199 Differential revision 1311 Delete out of TTL store files before compaction selection (Liyin) (Revision 1243086)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #109 (See https://builds.apache.org/job/HBase-TRUNK-security/109/ ) HBASE-5199 Differential revision 1311 Delete out of TTL store files before compaction selection (Liyin) (Revision 1243086) Result = FAILURE tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactSelection.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
          Hide
          Phabricator added a comment -

          mbautin has committed the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          COMMIT
          https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786

          Show
          Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". REVISION DETAIL https://reviews.facebook.net/D1311 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
          Hide
          Phabricator added a comment -

          mbautin has committed the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          COMMIT
          https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786

          Show
          Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". REVISION DETAIL https://reviews.facebook.net/D1311 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
          Hide
          Phabricator added a comment -

          mbautin has committed the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          COMMIT
          https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786

          Show
          Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". REVISION DETAIL https://reviews.facebook.net/D1311 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
          Hide
          Phabricator added a comment -

          mbautin has committed the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          COMMIT
          https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786

          Show
          Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". REVISION DETAIL https://reviews.facebook.net/D1311 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786
          Hide
          Phabricator added a comment -

          mbautin has committed the revision "[jira]HBASE-5199 Delete out of TTL store files before compaction selection ".

          REVISION DETAIL
          https://reviews.facebook.net/D1311

          COMMIT
          https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786

          Show
          Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5199 Delete out of TTL store files before compaction selection ". REVISION DETAIL https://reviews.facebook.net/D1311 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1239786

            People

            • Assignee:
              Liyin Tang
              Reporter:
              Liyin Tang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development