Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: hbase-11339
    • Fix Version/s: hbase-11339
    • Component/s: Compaction, master
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      From the design doc, mob files are not involved in the normal HBase compaction process. This means deleted mobs would still take up space and that we never really merge mob files that accrue over time. Currently, MOBs depend on two external tools:

      1) A TTL cleaner that removes mobs that have passed their TTL or exceeded minVersions.
      2) A 'sweep tool' cleaner that remove mobs that have had their references deleted and merges small files into larger ones.

      Today the tools are triggered by admins. The longer term goal would be to integrate them into hbase such that by default mobs are cleaned. The tools will be preserved however so that advanced admins can disable automatic cleanups and manually trigger these compaction like operaitons. #1 would likely be a chore in the master while #2 requires some design work to integrate into hbase.

      1. HBASE-11644.diff
        137 kB
        Jingcheng Du
      2. HBASE-11644-Sep-15.diff
        157 kB
        Jingcheng Du
      3. HBASE-11644-Sep-16.diff
        157 kB
        Jingcheng Du
      4. HBASE-11644-Sep-16.diff
        157 kB
        Jingcheng Du
      5. HBASE-11644-Sep-18.diff
        158 kB
        Jingcheng Du
      6. HBASE-11644-Sep-19.diff
        162 kB
        Jingcheng Du
      7. HBASE-11644-Sep-19-V2.patch
        162 kB
        Jingcheng Du
      8. HBASE-11646-0918-bad.patch
        160 kB
        Jonathan Hsieh

        Activity

        Hide
        Jonathan Hsieh added a comment -

        I reorganized the description to make it easier to follow.

        Show
        Jonathan Hsieh added a comment - I reorganized the description to make it easier to follow.
        Hide
        Jingcheng Du added a comment -

        In this patch, there're two tools, one is a file cleaner which cleans the expired mob files by TTL, the other is a sweep tool which merges the small files to bigger ones.
        The cleaner is run by HMaster periodically, and the sweep tool is triggered by users.

        Show
        Jingcheng Du added a comment - In this patch, there're two tools, one is a file cleaner which cleans the expired mob files by TTL, the other is a sweep tool which merges the small files to bigger ones. The cleaner is run by HMaster periodically, and the sweep tool is triggered by users.
        Hide
        Anoop Sam John added a comment -

        Can u put patch on RB.

        Show
        Anoop Sam John added a comment - Can u put patch on RB.
        Hide
        Jingcheng Du added a comment -

        Now the latest patch is uploaded to RB, you could find it by https://reviews.apache.org/r/23676/.

        Show
        Jingcheng Du added a comment - Now the latest patch is uploaded to RB, you could find it by https://reviews.apache.org/r/23676/ .
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12662941/HBASE-11644.diff
        against trunk revision .
        ATTACHMENT ID: 12662941

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10812//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12662941/HBASE-11644.diff against trunk revision . ATTACHMENT ID: 12662941 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10812//console This message is automatically generated.
        Hide
        Jingcheng Du added a comment -

        Update the patch according to Jon, Anoop and Ram's comments.

        Show
        Jingcheng Du added a comment - Update the patch according to Jon, Anoop and Ram's comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12668748/HBASE-11644-Sep-15.diff
        against trunk revision .
        ATTACHMENT ID: 12668748

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10888//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668748/HBASE-11644-Sep-15.diff against trunk revision . ATTACHMENT ID: 12668748 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 16 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10888//console This message is automatically generated.
        Hide
        Jingcheng Du added a comment -

        Update the patch (Sep-16) according to Anoop's comments in RB. Fix some typos in the patch.

        Show
        Jingcheng Du added a comment - Update the patch (Sep-16) according to Anoop's comments in RB. Fix some typos in the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12668958/HBASE-11644-Sep-16.diff
        against trunk revision .
        ATTACHMENT ID: 12668958

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10910//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668958/HBASE-11644-Sep-16.diff against trunk revision . ATTACHMENT ID: 12668958 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 16 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10910//console This message is automatically generated.
        Hide
        Jingcheng Du added a comment -

        Upload the latest patch( Sep-16), add a necessary comment to the code.

        Show
        Jingcheng Du added a comment - Upload the latest patch( Sep-16), add a necessary comment to the code.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12668985/HBASE-11644-Sep-16.diff
        against trunk revision .
        ATTACHMENT ID: 12668985

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10917//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668985/HBASE-11644-Sep-16.diff against trunk revision . ATTACHMENT ID: 12668985 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 16 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10917//console This message is automatically generated.
        Hide
        Jingcheng Du added a comment -

        Update the patch according to Anoop's comments in RB.
        1. Several nits.
        2. Move the SweepJobNodeTracker to setup() instead of in map().

        Show
        Jingcheng Du added a comment - Update the patch according to Anoop's comments in RB. 1. Several nits. 2. Move the SweepJobNodeTracker to setup() instead of in map().
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12669710/HBASE-11644-Sep-18.diff
        against trunk revision .
        ATTACHMENT ID: 12669710

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10967//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669710/HBASE-11644-Sep-18.diff against trunk revision . ATTACHMENT ID: 12669710 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 16 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10967//console This message is automatically generated.
        Hide
        Jonathan Hsieh added a comment -

        hey JingchengDu, this latest patch has problems when applied to the hbase-11339 branch – significantly more than HBASE-11646 had. Most of them are around TableName vs String in the admin flush argument. (this might also be due to patches being applied in a different order).

        I took a stab at working through most of them, but now several of the mob tests fail. (mvn clean test -Dtest=TestMob*) Can you rebase and please post patches that actually apply against the target hbase-11339 branch properly?

        I've attached the quick version I put together as hbase-11644-0918-bad.patch if you want to work from that.

        Show
        Jonathan Hsieh added a comment - hey JingchengDu , this latest patch has problems when applied to the hbase-11339 branch – significantly more than HBASE-11646 had. Most of them are around TableName vs String in the admin flush argument. (this might also be due to patches being applied in a different order). I took a stab at working through most of them, but now several of the mob tests fail. (mvn clean test -Dtest=TestMob*) Can you rebase and please post patches that actually apply against the target hbase-11339 branch properly? I've attached the quick version I put together as hbase-11644-0918-bad.patch if you want to work from that.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12669884/HBASE-11646-0918-bad.patch
        against trunk revision .
        ATTACHMENT ID: 12669884

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 27 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10983//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669884/HBASE-11646-0918-bad.patch against trunk revision . ATTACHMENT ID: 12669884 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 27 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10983//console This message is automatically generated.
        Hide
        Jingcheng Du added a comment -

        Update the patch based on the patch in "handle mob in compaction".

        Show
        Jingcheng Du added a comment - Update the patch based on the patch in "handle mob in compaction".
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12669957/HBASE-11644-Sep-19.diff
        against trunk revision .
        ATTACHMENT ID: 12669957

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 19 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10992//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669957/HBASE-11644-Sep-19.diff against trunk revision . ATTACHMENT ID: 12669957 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 19 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10992//console This message is automatically generated.
        Hide
        Jingcheng Du added a comment -

        And use "mvn clean test -Dtest=TestMob*" to pass the testing in the patch.

        Show
        Jingcheng Du added a comment - And use "mvn clean test -Dtest=TestMob*" to pass the testing in the patch.
        Hide
        Jingcheng Du added a comment -

        Generate the patch(HBASE-11644-Sep-19-V2.patch) based on the latest code in hbase-11339. Thanks Anoop for the help!

        Show
        Jingcheng Du added a comment - Generate the patch( HBASE-11644 -Sep-19-V2.patch) based on the latest code in hbase-11339. Thanks Anoop for the help!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12669962/HBASE-11644-Sep-19-V2.patch
        against trunk revision .
        ATTACHMENT ID: 12669962

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 19 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10993//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669962/HBASE-11644-Sep-19-V2.patch against trunk revision . ATTACHMENT ID: 12669962 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 19 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10993//console This message is automatically generated.
        Hide
        Jonathan Hsieh added a comment -

        the sept19 v2 version compiles and seems to pass for with the 'mvn clean test -Dtest=test*Mob*'

        Show
        Jonathan Hsieh added a comment - the sept19 v2 version compiles and seems to pass for with the 'mvn clean test -Dtest=test*Mob*'
        Hide
        Jonathan Hsieh added a comment -

        thanks jingcheng, and thans for reviews anoop. I've committed to the hbase-11339 branch.

        Show
        Jonathan Hsieh added a comment - thanks jingcheng, and thans for reviews anoop. I've committed to the hbase-11339 branch.

          People

          • Assignee:
            Jingcheng Du
            Reporter:
            Jingcheng Du
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development