Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: contrib/streaming, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestFileArgs failed once for me with the following error

      expected:<[job.jar
      sidefile
      tmp
      ]> but was:<[]>
      sidefile
      tmp
      ]> but was:<[]>
              at org.apache.hadoop.streaming.TestStreaming.checkOutput(TestStreaming.java:107)
              at org.apache.hadoop.streaming.TestStreaming.testCommandLine(TestStreaming.java:123)
      

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )
        Hide
        Amareshwari Sriramadasu added a comment -

        +1.
        I just committed this. Thanks Todd!

        Show
        Amareshwari Sriramadasu added a comment - +1. I just committed this. Thanks Todd!
        Hide
        Todd Lipcon added a comment -

        That original patch was in fact OK - my backport to 0.20 was incorrect so I thought it was still flaky based, but this fix should be good on trunk, and still applies. Marking patch available again with same patch.

        Show
        Todd Lipcon added a comment - That original patch was in fact OK - my backport to 0.20 was incorrect so I thought it was still flaky based, but this fix should be good on trunk, and still applies. Marking patch available again with same patch.
        Hide
        Todd Lipcon added a comment -

        I take it back - this still seems to occur occasionally, though less often. I'll keep spinning this test until I figure out why. Cancelling patch.

        Show
        Todd Lipcon added a comment - I take it back - this still seems to occur occasionally, though less often. I'll keep spinning this test until I figure out why. Cancelling patch.
        Hide
        Todd Lipcon added a comment -

        This should be good to go - it's trivial and fixes a flaky test.

        Show
        Todd Lipcon added a comment - This should be good to go - it's trivial and fixes a flaky test.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12435635/mapreduce-1375.txt
        against trunk revision 909241.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435635/mapreduce-1375.txt against trunk revision 909241. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/447/console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        I think this patch fixes the problem.

        Show
        Todd Lipcon added a comment - I think this patch fixes the problem.
        Hide
        Todd Lipcon added a comment -

        I think I got this figured out. The issue is that the test actually tries to write some "roses are red" text to ls's stdin. Very infrequently, the ls will actually complete before the data can be flushed, so the task gets a "Broken pipe" exception - see MAPREDUCE-1481. I'm actually unsure whether MAPREDUCE-1481 is a bug, but the easy fix for this test is to make the input "" so no data gets written into ls's stdin.

        I'm running the test in a loop with this fix now. If it keeps going for a couple hours without failure I'll post a patch. (before, this loop would fail after about 10 minutes usually)

        Show
        Todd Lipcon added a comment - I think I got this figured out. The issue is that the test actually tries to write some "roses are red" text to ls's stdin. Very infrequently, the ls will actually complete before the data can be flushed, so the task gets a "Broken pipe" exception - see MAPREDUCE-1481 . I'm actually unsure whether MAPREDUCE-1481 is a bug, but the easy fix for this test is to make the input "" so no data gets written into ls's stdin. I'm running the test in a loop with this fix now. If it keeps going for a couple hours without failure I'll post a patch. (before, this loop would fail after about 10 minutes usually)
        Hide
        Todd Lipcon added a comment -

        It looks like it's running two tasks. Am I interpreting the log wrong?

        10/02/02 23:59:27 INFO FSNamesystem.audit: ugi=gravi ip=/127.0.0.1 cmd=rename src=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/_temporary/_attempt_20100202235916710_0001_m_000000_0/part-00000 dst=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/part-00000 perm=gravi:supergroup:rw-r-r-
        10/02/02 23:59:27 INFO FSNamesystem.audit: ugi=gravi ip=/127.0.0.1 cmd=rename src=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/_temporary/_attempt_20100202235916710_0001_m_000001_0/part-00001 dst=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/part-00001 perm=gravi:supergroup:rw-r-r-

        Show
        Todd Lipcon added a comment - It looks like it's running two tasks. Am I interpreting the log wrong? 10/02/02 23:59:27 INFO FSNamesystem.audit: ugi=gravi ip=/127.0.0.1 cmd=rename src=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/_temporary/_attempt_20100202235916710_0001_m_000000_0/part-00000 dst=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/part-00000 perm=gravi:supergroup:rw-r- r - 10/02/02 23:59:27 INFO FSNamesystem.audit: ugi=gravi ip=/127.0.0.1 cmd=rename src=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/_temporary/_attempt_20100202235916710_0001_m_000001_0/part-00001 dst=/home/gravi/workspace/mr1435/trunk/build/contrib/streaming/test/data/org.apache.hadoop.streaming.TestFileArgs/out/part-00001 perm=gravi:supergroup:rw-r- r -
        Hide
        Ravi Gummadi added a comment -

        Attaching failed test log

        Show
        Ravi Gummadi added a comment - Attaching failed test log
        Hide
        Todd Lipcon added a comment -

        This is very strange - it indicates either that streaming didn't work at all, or that the task wasn't run in the right working directory. If you have some logs that would be great.

        Show
        Todd Lipcon added a comment - This is very strange - it indicates either that streaming didn't work at all, or that the task wasn't run in the right working directory. If you have some logs that would be great.
        Hide
        Ravi Gummadi added a comment -

        Got same error message that is given in the Description now for me also.

        Show
        Ravi Gummadi added a comment - Got same error message that is given in the Description now for me also.
        Hide
        Todd Lipcon added a comment -

        Was this local or on Hudson? Do you have the full test logs by any chance?

        Show
        Todd Lipcon added a comment - Was this local or on Hudson? Do you have the full test logs by any chance?

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development