[HADOOP-14631] Distcp should add a default atomicWorkPath properties when using atomic

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.3, 3.0.0-alpha3
• Fix Version/s: 2.7.3
• Assignee: Hongyuan Li
• Reporter: Hongyuan Li
• Created: 07/Jul/17 14:41
• Updated: 10/Sep/21 21:27

Description

Distcp should add a default AtomicWorkPath properties when using atomic

Distcp#configureOutputFormat using code below to generate atomic work path,

    if (context.shouldAtomicCommit()) {
      Path workDir = context.getAtomicWorkPath();
      if (workDir == null) {
        workDir = targetPath.getParent();
      }
      workDir = new Path(workDir, WIP_PREFIX + targetPath.getName()
                                + rand.nextInt());

When atomic is set and AtomicWorkPath == null, distcp will get the parent of current WorkDir. In this case, if workdir is "/", the parent will be null, wich means
workDir = new Path(workDir, WIP_PREFIX + targetPath.getName() + rand.nextInt()); will throw a nullpoint exception.

Issue Links

• is depended upon by: HADOOP-15788 Improve Distcp for long-haul/cloud deployments
• is related to: HADOOP-14567 DistCP NullPointerException when -atomic is set but -tmp is not