Hadoop Common
  1. Hadoop Common
  2. HADOOP-7631

In mapred-site.xml, stream.tmpdir is mapped to ${mapred.temp.dir} which is undeclared.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: conf
    • Labels:
      None

      Description

      Streaming jobs seem to fail with the following exception:

      Exception in thread "main" java.io.IOException: No such file or directory
              at java.io.UnixFileSystem.createFileExclusively(Native Method)
              at java.io.File.checkAndCreate(File.java:1704)
              at java.io.File.createTempFile(File.java:1792)
              at org.apache.hadoop.streaming.StreamJob.packageJobJar(StreamJob.java:603)
              at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:798)
              at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:117)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
              at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:32)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      

      Eric pointed out that in RPM based installs, in /etc/hadoop/mapred-site.xml, stream.tmpdir is mapped to $

      {mapred.temp.dir}, but ${mapred.temp.dir}

      is not declared

      1. HADOOP-7631-trunk.patch
        0.6 kB
        Eric Yang
      2. HADOOP-7631.patch
        0.5 kB
        Eric Yang

        Activity

        Hide
        Matt Foley added a comment -

        Closed upon release of 0.20.205.0

        Show
        Matt Foley added a comment - Closed upon release of 0.20.205.0
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/834/)
        HADOOP-7631. Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/834/ ) HADOOP-7631 . Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #10 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/10/)
        Merge -r 1172185:1172186 from trunk onto branch-0.23. Fixes HADOOP-7631.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172187
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #10 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/10/ ) Merge -r 1172185:1172186 from trunk onto branch-0.23. Fixes HADOOP-7631 . ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172187 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #12 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/12/)
        Merge -r 1172185:1172186 from trunk onto branch-0.23. Fixes HADOOP-7631.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172187
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #12 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/12/ ) Merge -r 1172185:1172186 from trunk onto branch-0.23. Fixes HADOOP-7631 . ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172187 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #804 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/804/)
        HADOOP-7631. Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #804 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/804/ ) HADOOP-7631 . Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #923 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/923/)
        HADOOP-7631. Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #923 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/923/ ) HADOOP-7631 . Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #908 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/908/)
        HADOOP-7631. Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #908 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/908/ ) HADOOP-7631 . Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #985 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/985/)
        HADOOP-7631. Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang.

        ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #985 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/985/ ) HADOOP-7631 . Fixes a config problem to do with running streaming jobs. Contributed by Eric Yang. ddas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172186 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/mapred-site.xml
        Hide
        Devaraj Das added a comment -

        Committed this. Thanks, Eric!

        Show
        Devaraj Das added a comment - Committed this. Thanks, Eric!
        Hide
        Matt Foley added a comment -

        +1 for trunk and branch. Release audit results are unavailable currently, but unlikely to be due to this patch.

        Show
        Matt Foley added a comment - +1 for trunk and branch. Release audit results are unavailable currently, but unlikely to be due to this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494318/HADOOP-7631-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        -1 release audit. The applied patch generated 9 release audit warnings (more than the trunk's current 0 warnings).

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/178//testReport/
        Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/178//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/178//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494318/HADOOP-7631-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 9 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/178//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/178//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/178//console This message is automatically generated.
        Hide
        Eric Yang added a comment -

        Resubmit to trigger patch test.

        Show
        Eric Yang added a comment - Resubmit to trigger patch test.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12494304/HADOOP-7631.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/174//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494304/HADOOP-7631.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/174//console This message is automatically generated.
        Hide
        Eric Yang added a comment -

        Remove declaration fo stream.tmpdir from mapred-site.xml.

        Show
        Eric Yang added a comment - Remove declaration fo stream.tmpdir from mapred-site.xml.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Ramya Sunil
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development