HBase
  1. HBase
  2. HBASE-3695

Some improvements to Hbck to test the entire region chain in Meta and provide better error reporting

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.1
    • Fix Version/s: 0.90.3
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      hbck meta

      Description

      The current Hbck tool will miss some inconsistencies in Meta, and in other cases will detect an issue, but does not provide much in the way of useful feedback.

      • Incorporate the full region chain tests (similar to check_meta.rb). I.e. look for overlaps, holes and cycles. I believe check_meta.rb will be redundant after this change.
      • More unit tests, and better tests that will test the actual error discovered, instead of just errors true/false.
      • In the case of overlaps and holes, output both ends of the broken chain.
      • Previous implementation runs check() twice. This is inefficient and, more importantly, reports redundant errors which could be confusing to the user.

        Activity

        Hide
        stack added a comment -

        @Marc Should I resolve 'HBASE-3602) [hbck] Print out better info if detects overlapping regions in .META.' as dup of this? Thanks boss.

        Show
        stack added a comment - @Marc Should I resolve ' HBASE-3602 ) [hbck] Print out better info if detects overlapping regions in .META.' as dup of this? Thanks boss.
        Hide
        Marc Limotte added a comment -

        @Stack Yea. That one is a dupe. It was earlier, but it is only a subset of what's here.

        Separately, I think there should be a distinct JIRA to cover the merge tool issue (it doesn't work with the low, overlapping sequence numbers generated by the bulk load tool).

        Also (possibly another JIRA), I wonder if the ability to manually create a region should be exposed (through the shell or other tool). We had to do this a couple of times to resolve the issues that had come up; but, on the other hand, if the merge tool worked in those scenarios, maybe this wouldn't have been necessary.

        Show
        Marc Limotte added a comment - @Stack Yea. That one is a dupe. It was earlier, but it is only a subset of what's here. Separately, I think there should be a distinct JIRA to cover the merge tool issue (it doesn't work with the low, overlapping sequence numbers generated by the bulk load tool). Also (possibly another JIRA), I wonder if the ability to manually create a region should be exposed (through the shell or other tool). We had to do this a couple of times to resolve the issues that had come up; but, on the other hand, if the merge tool worked in those scenarios, maybe this wouldn't have been necessary.
        Hide
        stack added a comment -

        Resolved the dupe. There is already an online merge too. And yes, need to make it so region hacking in .META. is a bit easier, perhaps some facility in shell (Saw someone trying to do "put '.META.', 'STRING_VERSION_OF_HREGIONINFO', etc" which plain won't work, but, if you think a second, should work).

        Show
        stack added a comment - Resolved the dupe. There is already an online merge too. And yes, need to make it so region hacking in .META. is a bit easier, perhaps some facility in shell (Saw someone trying to do "put '.META.', 'STRING_VERSION_OF_HREGIONINFO', etc" which plain won't work, but, if you think a second, should work).
        Hide
        stack added a comment -

        Applied to trunk and branch after trying it up on cluster to make sure its not 'broke'. Thanks for the patch Marc.

        Show
        stack added a comment - Applied to trunk and branch after trying it up on cluster to make sure its not 'broke'. Thanks for the patch Marc.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #1909 (See https://builds.apache.org/hudson/job/HBase-TRUNK/1909/)

        Show
        Hudson added a comment - Integrated in HBase-TRUNK #1909 (See https://builds.apache.org/hudson/job/HBase-TRUNK/1909/ )
        Hide
        Wayne added a comment -

        It would be very useful to have the -fix flag actually be able to resolve most issues. Once the region keys get messed up it is very hard to manually fix these issues. This tool should be able to find and fix the major problems that can occur. These issues include no region with an empty start key, no region with an empty end key, a hole in the region chain, overlapping regions, regions with the same start key, etc. etc.

        If this tool can find the issues it should be able to fix them as well.

        Show
        Wayne added a comment - It would be very useful to have the -fix flag actually be able to resolve most issues. Once the region keys get messed up it is very hard to manually fix these issues. This tool should be able to find and fix the major problems that can occur. These issues include no region with an empty start key, no region with an empty end key, a hole in the region chain, overlapping regions, regions with the same start key, etc. etc. If this tool can find the issues it should be able to fix them as well.

          People

          • Assignee:
            Marc Limotte
            Reporter:
            Marc Limotte
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development