Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7077

Pipe mapreduce job fails with Permission denied for jobTokenPassword

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.2.0, 3.1.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Steps:

      Launch wordcount example with pipe

      /usr/hdp/current/hadoop-client/bin/hadoop pipes "-Dhadoop.pipes.java.recordreader=true" "-Dhadoop.pipes.java.recordwriter=true" -input pipeInput -output pipeOutput -program bin/wordcount

      The application fails with below stacktrace

      AM
      attempt_1517534613368_0041_r_000000_2 is : 0.0
      
      2018-02-02 02:40:51,071 ERROR [IPC Server handler 16 on 43391] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1517534613368_0041_r_000000_2 - exited : java.io.FileNotFoundException: /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1517534613368_0041/jobTokenPassword (Permission denied)
      
       at java.io.FileOutputStream.open0(Native Method)
      
       at java.io.FileOutputStream.open(FileOutputStream.java:270)
      
       at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
      
       at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:236)
      
       at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
      
       at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:318)
      
       at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:307)
      
       at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:338)
      
       at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:401)
      
       at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:464)
      
       at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
      
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
      
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
      
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
      
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
      
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:703)
      
       at org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:173)
      
       at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:109)
      
       at org.apache.hadoop.mapred.pipes.PipesReducer.startApplication(PipesReducer.java:87)
      
       at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:65)
      
       at org.apache.hadoop.mapred.pipes.PipesReducer.reduce(PipesReducer.java:38)
      
       at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:445)
      
       at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
      
       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
      
       at java.security.AccessController.doPrivileged(Native Method)
      
       at javax.security.auth.Subject.doAs(Subject.java:422)
      
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
      
       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ajisakaa Akira Ajisaka
                Reporter:
                yeshavora Yesha Vora
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: