Hadoop Common
  1. Hadoop Common
  2. HADOOP-6576

TestStreamingStatus is failing on 0.20 branch

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestStreamingStatus fails consistently in the 0.20 branch after HADOOP-5623

      1. C6576-1.patch
        38 kB
        Chris Douglas
      2. hadoop-6576.txt
        42 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        MAPREDUCE-1155 I believe fixes this

        Show
        Todd Lipcon added a comment - MAPREDUCE-1155 I believe fixes this
        Hide
        Todd Lipcon added a comment -

        Here's the patch we've applied against the 0.20 branch of CDH (as a somewhat modified backport of MAPREDUCE-1155). NB this hasn't been tested against Apache branch-20. It's also a pretty big rework of the streaming tests, so perhaps not good for inclusion in Apache. But I figured I would contribute it in case it helps this issue.

        Show
        Todd Lipcon added a comment - Here's the patch we've applied against the 0.20 branch of CDH (as a somewhat modified backport of MAPREDUCE-1155 ). NB this hasn't been tested against Apache branch-20. It's also a pretty big rework of the streaming tests, so perhaps not good for inclusion in Apache. But I figured I would contribute it in case it helps this issue.
        Hide
        Chris Douglas added a comment -

        Thanks, Todd. MAPREDUCE-1155 was a worthwhile refactoring, so I'd have no problem including it in 0.20. What do you think about leaving out the changes to StreamJob for compatibility?

        Show
        Chris Douglas added a comment - Thanks, Todd. MAPREDUCE-1155 was a worthwhile refactoring, so I'd have no problem including it in 0.20. What do you think about leaving out the changes to StreamJob for compatibility?
        Hide
        Todd Lipcon added a comment -

        Hey Chris. Which StreamJob change are you referring to? Are you looking at the patch above or the original from the other issue? This one changes the following files:

        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestGzipInput.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleArchiveFiles.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleCachefiles.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamAggregate.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamDataProtocol.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamReduceNone.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamXmlRecordReader.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamedMerge.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreaming.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingCounters.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingExitStatus.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingKeyValue.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingSeparator.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingStatus.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingStderr.java
        src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestSymLink.java
        src/contrib/test/mapred-site.xml
        src/core/org/apache/hadoop/util/StringUtils.java

        Show
        Todd Lipcon added a comment - Hey Chris. Which StreamJob change are you referring to? Are you looking at the patch above or the original from the other issue? This one changes the following files: src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestGzipInput.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleArchiveFiles.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestMultipleCachefiles.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamAggregate.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamDataProtocol.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamReduceNone.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamXmlRecordReader.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamedMerge.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreaming.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingCounters.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingExitStatus.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingKeyValue.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingSeparator.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingStatus.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingStderr.java src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestSymLink.java src/contrib/test/mapred-site.xml src/core/org/apache/hadoop/util/StringUtils.java
        Hide
        Chris Douglas added a comment -

        Thanks, you're right; I was looking at the patch for MAPREDUCE-1155. +1

        I'd rather not rerun all the 0.20.2 tests to validate the change to the test config or add StringUtils::join just for the unit test. Attaching a version of the original, but with these minor nits addressed. All the streaming tests passed on the 0.20 branch

        Show
        Chris Douglas added a comment - Thanks, you're right; I was looking at the patch for MAPREDUCE-1155 . +1 I'd rather not rerun all the 0.20.2 tests to validate the change to the test config or add StringUtils::join just for the unit test. Attaching a version of the original, but with these minor nits addressed. All the streaming tests passed on the 0.20 branch
        Hide
        Chris Douglas added a comment -

        I committed this. Thanks, Todd!

        Show
        Chris Douglas added a comment - I committed this. Thanks, Todd!

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Chris Douglas
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development