Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14631

Distcp should add a default atomicWorkPath properties when using atomic

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.3, 3.0.0-alpha3
    • Fix Version/s: 2.7.3
    • Component/s: None
    • Labels:
      None

      Description

      Distcp should add a default AtomicWorkPath properties when using atomic

      Distcp#configureOutputFormat using code below to generate atomic work path,

          if (context.shouldAtomicCommit()) {
            Path workDir = context.getAtomicWorkPath();
            if (workDir == null) {
              workDir = targetPath.getParent();
            }
            workDir = new Path(workDir, WIP_PREFIX + targetPath.getName()
                                      + rand.nextInt());
      

      When atomic is set and AtomicWorkPath == null, distcp will get the parent of current WorkDir. In this case, if workdir is "/", the parent will be null, wich means
      workDir = new Path(workDir, WIP_PREFIX + targetPath.getName() + rand.nextInt()); will throw a nullpoint exception.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Hongyuan Li Hongyuan Li
                Reporter:
                Hongyuan Li Hongyuan Li
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: