Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7335

Failed split can cause a region to get stuck in transition

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.92.1
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None
    • Tags:
      hbase regionserver region

      Description

      Trying to reassign a region after a failed split causes a that region to get stuck in transition.

      hdfs dfs -R output

      http://pastebin.com/F4DgTxj1

      hbck output

      http://pastebin.com/BaftESBd

      error on regionserver

      http://pastebin.com/Mye60rUA

      For example, if I remove
      /hbase/mytable/2918ce63a9e0bf48b4f3227d88a992b2/RAW/990e00f1058442b3a79de8e39176b978.e6413e07faefd5801f25867ecbc97590
      the region will successfully assign and hbck does not show errors for this
      region anymore. The contents of the file appear to just be a split key.

        Issue Links

          Activity

          Hide
          kyle.t.mcgovern Kyle McGovern added a comment -

          Thanks for the link to the JIRA. It doesn't appear there was any data loss.

          Show
          kyle.t.mcgovern Kyle McGovern added a comment - Thanks for the link to the JIRA. It doesn't appear there was any data loss.
          Hide
          jxiang Jimmy Xiang added a comment -

          This region should be a daughter region. The region split should be succeeded. It looks to me the parent region is removed while there are still daughter regions refer to it. Since the parent region is gone, we got no choice and have to remove the reference file.

          Do you have any data loss?

          Show
          jxiang Jimmy Xiang added a comment - This region should be a daughter region. The region split should be succeeded. It looks to me the parent region is removed while there are still daughter regions refer to it. Since the parent region is gone, we got no choice and have to remove the reference file. Do you have any data loss?
          Hide
          jxiang Jimmy Xiang added a comment -

          With HBASE-7199, hbck can detect and fix such issue.

          Show
          jxiang Jimmy Xiang added a comment - With HBASE-7199 , hbck can detect and fix such issue.
          Hide
          kyle.t.mcgovern Kyle McGovern added a comment -

          ramkrishna.s.vasudevan I'm not sure exactly when the split failed so finding the logs might be difficult. Is there any string in particular I might be able to search for?

          Show
          kyle.t.mcgovern Kyle McGovern added a comment - ramkrishna.s.vasudevan I'm not sure exactly when the split failed so finding the logs might be difficult. Is there any string in particular I might be able to search for?
          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -

          @Kyle
          Could you attach the logs during the time of split? Is it possible ?

          Show
          ram_krish ramkrishna.s.vasudevan added a comment - @Kyle Could you attach the logs during the time of split? Is it possible ?
          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -

          If there is time i will take a look at these logs later today in the eve. Folks have been reporting this frequently.

          Show
          ram_krish ramkrishna.s.vasudevan added a comment - If there is time i will take a look at these logs later today in the eve. Folks have been reporting this frequently.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          Could this be the same issue as HBASE-7101 ?
          Dave was using 0.92 as well.

          Show
          yuzhihong@gmail.com Ted Yu added a comment - Could this be the same issue as HBASE-7101 ? Dave was using 0.92 as well.

            People

            • Assignee:
              Unassigned
              Reporter:
              kyle.t.mcgovern Kyle McGovern
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development