HBase
  1. HBase
  2. HBASE-6050

HLogSplitter renaming recovered.edits and CJ removing the parent directory race, making the HBCK think cluster is inconsistent.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.1, 0.94.0
    • Fix Version/s: 0.94.1, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The scenario is like this
      -> A region is getting splitted.
      -> The master is still not processed the split .
      -> Region server goes down.
      -> Split log manager starts splitting the logs and creates the recovered.edits in the splitlog path.
      -> CJ starts and deletes the entry from META and also just completes the deletion of the region dir.
      -> in hlogSplitter on final step we rename the recovered.edits to come under the regiondir.
      There if the regiondir doesnot exist we tend to create and then add the recovered.edits.

      Because of this HBCK thinks it to be an orphan region because we have the regiondir but with no regioninfo.
      Ideally cluster is fine but we it is misleading.

              } else {
                Path dstdir = dst.getParent();
                if (!fs.exists(dstdir)) {
                  if (!fs.mkdirs(dstdir)) LOG.warn("mkdir failed on " + dstdir);
                }
              }
              fs.rename(src, dst);
              LOG.debug(" moved " + src + " => " + dst);
            } else {
              LOG.debug("Could not move recovered edits from " + src +
                  " as it doesn't exist");
            }
          }
          archiveLogs(null, corruptedLogs, processedLogs,
              oldLogDir, fs, conf);
      
      1. HBASE-6050.patch
        0.9 kB
        ramkrishna.s.vasudevan

        Activity

        Lars Hofhansl made changes -
        Fix Version/s 0.94.1 [ 12320257 ]
        stack made changes -
        Fix Version/s 0.95.0 [ 12324094 ]
        Fix Version/s 0.92.2 [ 12319888 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Fix Version/s 0.94.1 [ 12320257 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        ramkrishna.s.vasudevan made changes -
        Fix Version/s 0.92.2 [ 12319888 ]
        Fix Version/s 0.96.0 [ 12320040 ]
        Fix Version/s 0.94.1 [ 12320257 ]
        Affects Version/s 0.94.0 [ 12316419 ]
        Affects Version/s 0.92.1 [ 12318551 ]
        ramkrishna.s.vasudevan made changes -
        Assignee ramkrishna.s.vasudevan [ ram_krish ]
        ramkrishna.s.vasudevan made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Ted Yu made changes -
        Summary HLogSplitter renaming recovered.edits and CJ removing the parent directory races, making the HBCK to think cluster is inconsistent. HLogSplitter renaming recovered.edits and CJ removing the parent directory race, making the HBCK think cluster is inconsistent.
        Hadoop Flags Reviewed [ 10343 ]
        ramkrishna.s.vasudevan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        ramkrishna.s.vasudevan made changes -
        Field Original Value New Value
        Attachment HBASE-6050.patch [ 12528231 ]
        ramkrishna.s.vasudevan created issue -

          People

          • Assignee:
            ramkrishna.s.vasudevan
            Reporter:
            ramkrishna.s.vasudevan
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development