Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4000

Create jobTokenPassword file fail when run hadoop pipes locally

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.0.1, 1.0.2, 0.23.3
    • Fix Version/s: None
    • Component/s: pipes
    • Labels:
      None
    • Environment:

      Description

      Create jobTokenPassword file fail when run hadoop pipes locally.

      I have put such settings in my job conf.

       
      <property>
        <name>mapreduce.jobtracker.address</name>
        <value>local</value>
      </property>
      
      <property>
        <name>mapreduce.framework.name</name>
        <value>local</value>
      </property>
      
      <property>
        <name>fs.defaultFS</name>
        <value>file:///</value>
      </property>
      

      This job has one map task ,and one reduce task. It will fail when running PipesReducer, because "jobTokenPassword" file is created at current working directory,with permission 0400.

      org.apache.hadoop.mapred.pipes.Application.java
       
       private void writePasswordToLocalFile(String localPasswordFile,
            byte[] password, JobConf conf) throws IOException {
          FileSystem localFs = FileSystem.getLocal(conf);
          Path localPath = new Path(localPasswordFile);
          FSDataOutputStream out = FileSystem.create(localFs, localPath,
              new FsPermission("400"));
          out.write(password);
          out.close();
        }
      

      So, it will fail at the second time.

      In such situation,the application cannot be initialized, but the PipesReducer.close() method will be called after that ,so there will be NullPointer Exception raise in close() method.

        Activity

        Hide
        khsibr Ryadh Khsib added a comment -

        Hi,

        Looking at java.io.FileNotFoundException: /u/pireddu/Projects/pydoop.git/examples/wordcount/test/jobTokenPassword (Permission denied) it seems that the java process doesn't have the right access to this folder.
        Can you check it?

        Cheers Ryadh

        Show
        khsibr Ryadh Khsib added a comment - Hi, Looking at java.io.FileNotFoundException: /u/pireddu/Projects/pydoop.git/examples/wordcount/test/jobTokenPassword (Permission denied) it seems that the java process doesn't have the right access to this folder. Can you check it? Cheers Ryadh
        Hide
        ilveroluca Luca Pireddu added a comment -

        Stack trace, from Hadoop 1.0.2:

        {{
        12/05/08 16:11:32 INFO mapred.JobClient: Task Id : attempt_201205081608_0001_m_000000_2, Status : FAILED
        java.io.FileNotFoundException: /u/pireddu/Projects/pydoop.git/examples/wordcount/test/jobTokenPassword (Permission denied)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:188)
        at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:184)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:255)
        at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:236)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:315)
        at org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:170)
        at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:106)
        at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
        }}

        Show
        ilveroluca Luca Pireddu added a comment - Stack trace, from Hadoop 1.0.2: {{ 12/05/08 16:11:32 INFO mapred.JobClient: Task Id : attempt_201205081608_0001_m_000000_2, Status : FAILED java.io.FileNotFoundException: /u/pireddu/Projects/pydoop.git/examples/wordcount/test/jobTokenPassword (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:194) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:188) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:184) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:255) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:236) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:315) at org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:170) at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:106) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.mapred.Child.main(Child.java:249) }}

          People

          • Assignee:
            Unassigned
            Reporter:
            snnn Changming Sun
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development