[MAPREDUCE-7229] Job History Server should ignore continuing move done_intermediate files when rename failed

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: jobhistoryserver
• Assignee: Unassigned
• Reporter: Morty Zhong
• Created: 12/Aug/19 07:12
• Updated: 20/Aug/20 15:39

Description

private void moveToDoneNow(final Path src, final Path target)
    throws IOException {
  LOG.info("Moving " + src.toString() + " to " + target.toString());
  try {
    intermediateDoneDirFc.rename(src, target, Options.Rename.NONE);
  } catch (FileNotFoundException e) {
    if (doneDirFc.util().exists(target)) {
      LOG.info("Source file " + src.toString() + " not found, but target "
          + "file " + target.toString() + " already exists. Move already "
          + "happened.");
    } else {
      throw e;
    }
  }
}

if files in done_intermediate dir were already deleted(deleted manually), not move to done dir. Job history will infinitely rename these files later, and throw exp always.This will cause newly created files in done_intermediate dir can not be moved. Later, when scan done_intermediate dir, FileContext.listStatus(done_intermediate) will so slow, and gc problem