[MAPREDUCE-6044] Fully qualified intermediate done directory will break per-user dir creation on Windows

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0, 3.0.0-alpha1
• Fix Version/s: 2.6.0
• Component/s: jobhistoryserver

Environment

Windows

• Target Version/s: 2.6.0
• Hadoop Flags: Reviewed
• Assignee: Zhijie Shen
• Reporter: Zhijie Shen
• Created: 21/Aug/14 21:35
• Updated: 12/May/16 18:24
• Resolved: 22/Aug/14 17:08

Description

After MAPREDUCE-6032, the string of the intermediate done dir will be a fully qualified path.

The following code in JobHistoryUtils tries to concat this path and user name to create a per-user dir path, using File.separator as the seperator (on Windows, it is "\").

  public static String getHistoryIntermediateDoneDirForUser(Configuration conf) throws IOException {
    return getConfiguredHistoryIntermediateDoneDirPrefix(conf) + File.separator
        + UserGroupInformation.getCurrentUser().getShortUserName();
  }

Therefore, an intermediate done dir for user will become "hdfs://localhost:9201/mapred/history/done_intermediate\user". With the scheme available in the path, Path class will not replace "\" with "/", and finally FS cannot handle this path correctly: it will take "done_intermediateuser" as a single directory name.

Issue Links

• relates to: MAPREDUCE-6032 Unable to check mapreduce job status if submitted using a non-default namenode