Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20583

SplitLogWorker should handle FileNotFoundException when split a wal

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.1, 1.2.7
    • wal
    • None

    Description

      When a split task is finished, master will delete the wal first, then remove the task's zk node. So if master crashed after delelte the wal, the zk task node may be leaved on zk. When master resubmit this task, the task will failed by FileNotFoundException.

      We also handle FileNotFoundException in WALSplitter. But not handle this in SplitLogWorker.

       

        try {
          in = getReader(path, reporter);
        } catch (EOFException e) {
          if (length <= 0) {
            // TODO should we ignore an empty, not-last log file if skip.errors
            // is false? Either way, the caller should decide what to do. E.g.
            // ignore if this is the last log in sequence.
            // TODO is this scenario still possible if the log has been
            // recovered (i.e. closed)
            LOG.warn("Could not open {} for reading. File is empty", path, e);
          }
          // EOFException being ignored
          return null;
        }
      } catch (IOException e) {
        if (e instanceof FileNotFoundException) {
          // A wal file may not exist anymore. Nothing can be recovered so move on
          LOG.warn("File {} does not exist anymore", path, e);
          return null;
        }
      }
      // Here fs.getFileStatus may throw FileNotFoundException, too. We should handle this exception as the WALSplitter.getReader.
      try {
        if (!WALSplitter.splitLogFile(walDir, fs.getFileStatus(new Path(walDir, filename)),
          fs, conf, p, sequenceIdChecker,
            server.getCoordinatedStateManager().getSplitLogWorkerCoordination(), factory)) {
          return Status.PREEMPTED;
        }
      } 
      

       

       

      Attachments

        1. HBASE-20583.master.001.patch
          2 kB
          Guanghao Zhang
        2. HBASE-20583.master.001.patch
          2 kB
          Guanghao Zhang

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zghao Guanghao Zhang
            zghao Guanghao Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment