HBase
  1. HBase
  2. HBASE-7335

Failed split can cause a region to get stuck in transition

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.92.1
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None
    • Tags:
      hbase regionserver region

      Description

      Trying to reassign a region after a failed split causes a that region to get stuck in transition.

      hdfs dfs -R output

      http://pastebin.com/F4DgTxj1

      hbck output

      http://pastebin.com/BaftESBd

      error on regionserver

      http://pastebin.com/Mye60rUA

      For example, if I remove
      /hbase/mytable/2918ce63a9e0bf48b4f3227d88a992b2/RAW/990e00f1058442b3a79de8e39176b978.e6413e07faefd5801f25867ecbc97590
      the region will successfully assign and hbck does not show errors for this
      region anymore. The contents of the file appear to just be a split key.

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          Could this be the same issue as HBASE-7101 ?
          Dave was using 0.92 as well.

          Show
          Ted Yu added a comment - Could this be the same issue as HBASE-7101 ? Dave was using 0.92 as well.
          Hide
          ramkrishna.s.vasudevan added a comment -

          If there is time i will take a look at these logs later today in the eve. Folks have been reporting this frequently.

          Show
          ramkrishna.s.vasudevan added a comment - If there is time i will take a look at these logs later today in the eve. Folks have been reporting this frequently.
          Hide
          ramkrishna.s.vasudevan added a comment -

          @Kyle
          Could you attach the logs during the time of split? Is it possible ?

          Show
          ramkrishna.s.vasudevan added a comment - @Kyle Could you attach the logs during the time of split? Is it possible ?
          Hide
          Kyle McGovern added a comment -

          ramkrishna.s.vasudevan I'm not sure exactly when the split failed so finding the logs might be difficult. Is there any string in particular I might be able to search for?

          Show
          Kyle McGovern added a comment - ramkrishna.s.vasudevan I'm not sure exactly when the split failed so finding the logs might be difficult. Is there any string in particular I might be able to search for?
          Hide
          Jimmy Xiang added a comment -

          With HBASE-7199, hbck can detect and fix such issue.

          Show
          Jimmy Xiang added a comment - With HBASE-7199 , hbck can detect and fix such issue.
          Hide
          Jimmy Xiang added a comment -

          This region should be a daughter region. The region split should be succeeded. It looks to me the parent region is removed while there are still daughter regions refer to it. Since the parent region is gone, we got no choice and have to remove the reference file.

          Do you have any data loss?

          Show
          Jimmy Xiang added a comment - This region should be a daughter region. The region split should be succeeded. It looks to me the parent region is removed while there are still daughter regions refer to it. Since the parent region is gone, we got no choice and have to remove the reference file. Do you have any data loss?
          Hide
          Kyle McGovern added a comment -

          Thanks for the link to the JIRA. It doesn't appear there was any data loss.

          Show
          Kyle McGovern added a comment - Thanks for the link to the JIRA. It doesn't appear there was any data loss.

            People

            • Assignee:
              Unassigned
              Reporter:
              Kyle McGovern
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development