Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5113

Streaming input/output types are ignored with java mapper/reducer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After MAPREDUCE-1888, with a java mapper or reducer, StreamJob doesn't respect stream.map.output/stream.reduce.output when setting a job's output key/value classes, even if these configs are explicitly set by the user.

      As MAPREDUCE-1888 is not in branch-1, this change is only needed in hadoop 2.

      1. HADOOP-9300.patch
        13 kB
        Sandy Ryza
      2. HADOOP-9300.patch
        12 kB
        Sandy Ryza
      3. HADOOP-9300-1.patch
        14 kB
        Sandy Ryza
      4. HADOOP-9300-2.patch
        17 kB
        Sandy Ryza
      5. HADOOP-9300-2.patch
        18 kB
        Sandy Ryza
      6. HADOOP-9300-2.patch
        18 kB
        Sandy Ryza
      7. HADOOP-9300-3.patch
        16 kB
        Sandy Ryza
      8. MAPREDUCE-5113.patch
        4 kB
        Sandy Ryza

        Activity

        Sandy Ryza created issue -
        Sandy Ryza made changes -
        Field Original Value New Value
        Description When a hadoop streaming job is run with a java class as the reducer, or no reducer specified (which defaults to IdentityReducer), the output key and value classes are not set. This can cause a job to fail down the line. In an effort to avoid overwriting user configs (MAPREDUCE-1888), StreamJob doesn't set a job's output key/value classes unless they are specified in the streaming command line. If the configs aren't specified in either of these places, the streaming defaults (Text) no longer kick in, and the global default LongWritable is used.

        This can cause jobs/output writers that are expecting Text to fail.
        Sandy Ryza made changes -
        Attachment HADOOP-9300.patch [ 12569120 ]
        Sandy Ryza made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569120/HADOOP-9300.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2182//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569120/HADOOP-9300.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2182//console This message is automatically generated.
        Sandy Ryza made changes -
        Attachment HADOOP-9300.patch [ 12569899 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569899/HADOOP-9300.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-tools/hadoop-streaming:

        org.apache.hadoop.streaming.TestMultipleArchiveFiles
        org.apache.hadoop.streaming.TestTypedBytesStreaming
        org.apache.hadoop.streaming.TestStreamReduceNone
        org.apache.hadoop.streaming.TestRawBytesStreaming
        org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes
        org.apache.hadoop.streaming.TestFileArgs
        org.apache.hadoop.streaming.TestStreamingFailure
        org.apache.hadoop.streaming.TestStreamXmlRecordReader

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2200//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2200//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569899/HADOOP-9300.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-tools/hadoop-streaming: org.apache.hadoop.streaming.TestMultipleArchiveFiles org.apache.hadoop.streaming.TestTypedBytesStreaming org.apache.hadoop.streaming.TestStreamReduceNone org.apache.hadoop.streaming.TestRawBytesStreaming org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes org.apache.hadoop.streaming.TestFileArgs org.apache.hadoop.streaming.TestStreamingFailure org.apache.hadoop.streaming.TestStreamXmlRecordReader +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2200//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2200//console This message is automatically generated.
        Sandy Ryza made changes -
        Attachment HADOOP-9300-1.patch [ 12570006 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570006/HADOOP-9300-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-tools/hadoop-streaming:

        org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes
        org.apache.hadoop.streaming.TestStreamReduceNone
        org.apache.hadoop.streaming.TestStreamXmlRecordReader
        org.apache.hadoop.streaming.TestFileArgs
        org.apache.hadoop.streaming.TestMultipleArchiveFiles
        org.apache.hadoop.streaming.TestStreamingFailure

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2206//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2206//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570006/HADOOP-9300-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-tools/hadoop-streaming: org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes org.apache.hadoop.streaming.TestStreamReduceNone org.apache.hadoop.streaming.TestStreamXmlRecordReader org.apache.hadoop.streaming.TestFileArgs org.apache.hadoop.streaming.TestMultipleArchiveFiles org.apache.hadoop.streaming.TestStreamingFailure +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2206//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2206//console This message is automatically generated.
        Sandy Ryza made changes -
        Attachment HADOOP-9300-2.patch [ 12570026 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570026/HADOOP-9300-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2208//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570026/HADOOP-9300-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2208//console This message is automatically generated.
        Sandy Ryza made changes -
        Attachment HADOOP-9300-2.patch [ 12570031 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570031/HADOOP-9300-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 4 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2209//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570031/HADOOP-9300-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2209//console This message is automatically generated.
        Sandy Ryza made changes -
        Attachment HADOOP-9300-2.patch [ 12570034 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570034/HADOOP-9300-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-tools/hadoop-streaming:

        org.apache.hadoop.streaming.TestStreamReduceNone
        org.apache.hadoop.streaming.TestStreamXmlRecordReader

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2210//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2210//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570034/HADOOP-9300-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-tools/hadoop-streaming: org.apache.hadoop.streaming.TestStreamReduceNone org.apache.hadoop.streaming.TestStreamXmlRecordReader +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2210//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2210//console This message is automatically generated.
        Hide
        Sandy Ryza added a comment -

        Sorry for all the noise - uploaded old versions of the patch a couple times. The last two failing tests are due to MAPREDUCE-5006.

        Show
        Sandy Ryza added a comment - Sorry for all the noise - uploaded old versions of the patch a couple times. The last two failing tests are due to MAPREDUCE-5006 .
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12570034/HADOOP-9300-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-streaming.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2370//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2370//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570034/HADOOP-9300-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-streaming. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2370//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2370//console This message is automatically generated.
        Sandy Ryza made changes -
        Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
        Key HADOOP-9300 MAPREDUCE-5113
        Affects Version/s 2.0.2-alpha [ 12322471 ]
        Affects Version/s 2.0.2-alpha [ 12322473 ]
        Component/s tools [ 12319643 ]
        Sandy Ryza made changes -
        Attachment HADOOP-9300-3.patch [ 12575946 ]
        Hide
        Sandy Ryza added a comment -

        Updated patch removes some unused imports and, in order not to change existing behavior, does not override input/output key/value classes if the mapper/reducer is a regular java one.

        Show
        Sandy Ryza added a comment - Updated patch removes some unused imports and, in order not to change existing behavior, does not override input/output key/value classes if the mapper/reducer is a regular java one.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12575946/HADOOP-9300-3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-streaming.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3475//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3475//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575946/HADOOP-9300-3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-streaming. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3475//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3475//console This message is automatically generated.
        Hide
        Sandy Ryza added a comment -

        Realized a lot fewer changes are needed, and updated a patch with a much simpler approach.

        Show
        Sandy Ryza added a comment - Realized a lot fewer changes are needed, and updated a patch with a much simpler approach.
        Sandy Ryza made changes -
        Attachment MAPREDUCE-5113.patch [ 12576117 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12576117/MAPREDUCE-5113.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-streaming.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3481//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3481//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576117/MAPREDUCE-5113.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-streaming. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3481//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3481//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12576117/MAPREDUCE-5113.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-streaming.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3486//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3486//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576117/MAPREDUCE-5113.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-streaming. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3486//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3486//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        due to HADOOP-9448 this JIRA is no relevant anymore.

        Show
        Alejandro Abdelnur added a comment - due to HADOOP-9448 this JIRA is no relevant anymore.
        Hide
        Alejandro Abdelnur added a comment -

        +1, it may be relevant after all. Please update summary/description to reflect exact change.

        Show
        Alejandro Abdelnur added a comment - +1, it may be relevant after all. Please update summary/description to reflect exact change.
        Sandy Ryza made changes -
        Summary Streaming fails to set output key class when reducer is java class Streaming input/output types are ignored with java mapper/reducer
        Sandy Ryza made changes -
        Description In an effort to avoid overwriting user configs (MAPREDUCE-1888), StreamJob doesn't set a job's output key/value classes unless they are specified in the streaming command line. If the configs aren't specified in either of these places, the streaming defaults (Text) no longer kick in, and the global default LongWritable is used.

        This can cause jobs/output writers that are expecting Text to fail.
        After MAPREDUCE-1888, with a java mapper or reducer, StreamJob doesn't respect stream.map.output/stream.reduce.output to set a job's output key/value classes.


        unless they are specified in the streaming command line. If the configs aren't specified in either of these places, the streaming defaults (Text) no longer kick in, and the global default LongWritable is used.

        This can cause jobs/output writers that are expecting Text to fail.
        Sandy Ryza made changes -
        Description After MAPREDUCE-1888, with a java mapper or reducer, StreamJob doesn't respect stream.map.output/stream.reduce.output to set a job's output key/value classes.


        unless they are specified in the streaming command line. If the configs aren't specified in either of these places, the streaming defaults (Text) no longer kick in, and the global default LongWritable is used.

        This can cause jobs/output writers that are expecting Text to fail.
        After MAPREDUCE-1888, with a java mapper or reducer, StreamJob doesn't respect stream.map.output/stream.reduce.output when setting a job's output key/value classes, even if these configs are explicitly set by the user.


        As MAPREDUCE-1888 is not in branch-1, this change is only needed in hadoop 2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3544 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3544/)
        MAPREDUCE-5113. Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3544 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3544/ ) MAPREDUCE-5113 . Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Sandy. committed to trunk, trying to sort out CHANGES.tx being out of sync before committing to branch-2.

        Show
        Alejandro Abdelnur added a comment - Thanks Sandy. committed to trunk, trying to sort out CHANGES.tx being out of sync before committing to branch-2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #173 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/173/)
        MAPREDUCE-5113. Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #173 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/173/ ) MAPREDUCE-5113 . Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1362 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1362/)
        MAPREDUCE-5113. Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1362 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1362/ ) MAPREDUCE-5113 . Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1389 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1389/)
        MAPREDUCE-5113. Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1389 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1389/ ) MAPREDUCE-5113 . Streaming input/output types are ignored with java mapper/reducer. (sandyr via tucu) (Revision 1463307) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1463307 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingOutputKeyValueTypes.java /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TrApp.java
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Sandy. Committed to trunk. Waiting for branch-2 until CHANGES.txt gets fixed.

        Show
        Alejandro Abdelnur added a comment - Thanks Sandy. Committed to trunk. Waiting for branch-2 until CHANGES.txt gets fixed.
        Hide
        Alejandro Abdelnur added a comment -

        Committed to branch-2 (thanks Sid for fixing CHANGES.txt)

        Show
        Alejandro Abdelnur added a comment - Committed to branch-2 (thanks Sid for fixing CHANGES.txt)
        Alejandro Abdelnur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.0.5-beta [ 12324032 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2h 45m 1 Sandy Ryza 13/Feb/13 00:59
        Patch Available Patch Available Resolved Resolved
        57d 12h 6m 1 Alejandro Abdelnur 11/Apr/13 14:05
        Resolved Resolved Closed Closed
        138d 9h 16m 1 Arun C Murthy 27/Aug/13 23:22

          People

          • Assignee:
            Sandy Ryza
            Reporter:
            Sandy Ryza
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development