Hadoop Common
  1. Hadoop Common
  2. HADOOP-5847

Streaming unit tests failing for a while on trunk

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/337/testReport/ shows some failed tests:

      org.apache.hadoop.streaming.TestMultipleCachefiles.testMultipleCachefiles
      org.apache.hadoop.streaming.TestStreamingBadRecords.testSkip
      org.apache.hadoop.streaming.TestStreamingBadRecords.testNarrowDown
      org.apache.hadoop.streaming.TestSymLink.testSymLink

      1. HADOOP-5847.patch
        0.7 kB
        Giridharan Kesavan

        Activity

        Hide
        Sharad Agarwal added a comment -

        The failure started happening since build:
        http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-trunk/835/

        In TestMultipleCachefiles, I am seeing numerous
        java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceDependent
        So it looks to be a library issue.

        Show
        Sharad Agarwal added a comment - The failure started happening since build: http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-trunk/835/ In TestMultipleCachefiles, I am seeing numerous java.lang.NoClassDefFoundError: org/apache/jasper/runtime/JspSourceDependent So it looks to be a library issue.
        Hide
        Giridharan Kesavan added a comment -

        this fixes the test failures.
        tnx

        Show
        Giridharan Kesavan added a comment - this fixes the test failures. tnx
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12408247/HADOOP-5847.patch
        against trunk revision 775812.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12408247/HADOOP-5847.patch against trunk revision 775812. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/346/console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        +1 on this patch. This fixes the failing streaming tests on trunk for me.

        Show
        Todd Lipcon added a comment - +1 on this patch. This fixes the failing streaming tests on trunk for me.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12408247/HADOOP-5847.patch
        against trunk revision 777330.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12408247/HADOOP-5847.patch against trunk revision 777330. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/378/console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        The Hudson QA report shows the test failure in the capacity scheduler. All of the streaming contrib tests passed after this change. Given that this bug has been causing hudson test failures for every other JIRA for an entire week now, could someone please commit this?

        Show
        Todd Lipcon added a comment - The Hudson QA report shows the test failure in the capacity scheduler. All of the streaming contrib tests passed after this change. Given that this bug has been causing hudson test failures for every other JIRA for an entire week now, could someone please commit this?
        Hide
        steve_l added a comment -

        I'm running these tests at home (OSX) , and am seeing a failure
        [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 13.164 sec
        [junit] Test org.apache.hadoop.streaming.TestStreamingExitStatus FAILED

        Looks like it is just an OOM event though.

        09/05/24 12:00:11 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]
        09/05/24 12:00:11 INFO streaming.PipeMapRed: Records R/W=1/1
        09/05/24 12:00:11 INFO streaming.PipeMapRed: MRErrorThread done
        09/05/24 12:00:11 WARN mapred.LocalJobRunner: job_local_0002
        java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
        at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366)
        at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:568)
        at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
        at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:360)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:11 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:11 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:11 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:11 INFO streaming.StreamJob: killJob...
        09/05/24 12:00:11 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
        09/05/24 12:00:11 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
        09/05/24 12:00:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        09/05/24 12:00:12 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
        09/05/24 12:00:12 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:13 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:13 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local]
        09/05/24 12:00:13 INFO streaming.StreamJob: Running job: job_local_0003
        09/05/24 12:00:13 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:13 INFO mapred.MapTask: numReduceTasks: 1
        09/05/24 12:00:13 INFO mapred.MapTask: io.sort.mb = 100
        09/05/24 12:00:13 WARN mapred.LocalJobRunner: job_local_0003
        java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:14 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:14 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:14 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:14 INFO streaming.StreamJob: killJob...
        09/05/24 12:00:14 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead.
        09/05/24 12:00:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
        09/05/24 12:00:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        09/05/24 12:00:14 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
        09/05/24 12:00:14 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:15 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local]
        09/05/24 12:00:15 INFO streaming.StreamJob: Running job: job_local_0004
        09/05/24 12:00:15 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:15 INFO mapred.FileInputFormat: Total input paths to process : 1
        09/05/24 12:00:15 INFO mapred.MapTask: numReduceTasks: 1
        09/05/24 12:00:15 INFO mapred.MapTask: io.sort.mb = 100
        09/05/24 12:00:16 WARN mapred.LocalJobRunner: job_local_0004
        java.lang.OutOfMemoryError: Java heap space
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178)
        09/05/24 12:00:16 INFO streaming.StreamJob: map 0% reduce 0%
        09/05/24 12:00:16 INFO streaming.StreamJob: Job running in-process (local Hadoop)
        09/05/24 12:00:16 ERROR streaming.StreamJob: Job not Successful!
        09/05/24 12:00:16 INFO streaming.StreamJob: killJob...
        ------------- ---------------- ---------------

        Show
        steve_l added a comment - I'm running these tests at home (OSX) , and am seeing a failure [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 13.164 sec [junit] Test org.apache.hadoop.streaming.TestStreamingExitStatus FAILED Looks like it is just an OOM event though. 09/05/24 12:00:11 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s] 09/05/24 12:00:11 INFO streaming.PipeMapRed: Records R/W=1/1 09/05/24 12:00:11 INFO streaming.PipeMapRed: MRErrorThread done 09/05/24 12:00:11 WARN mapred.LocalJobRunner: job_local_0002 java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:366) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:568) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:360) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:11 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:11 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:11 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:11 INFO streaming.StreamJob: killJob... 09/05/24 12:00:11 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. 09/05/24 12:00:11 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 09/05/24 12:00:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/05/24 12:00:12 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 09/05/24 12:00:12 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:13 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:13 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local] 09/05/24 12:00:13 INFO streaming.StreamJob: Running job: job_local_0003 09/05/24 12:00:13 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:13 INFO mapred.MapTask: numReduceTasks: 1 09/05/24 12:00:13 INFO mapred.MapTask: io.sort.mb = 100 09/05/24 12:00:13 WARN mapred.LocalJobRunner: job_local_0003 java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:14 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:14 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:14 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:14 INFO streaming.StreamJob: killJob... 09/05/24 12:00:14 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. 09/05/24 12:00:14 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 09/05/24 12:00:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/05/24 12:00:14 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 09/05/24 12:00:14 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:15 INFO streaming.StreamJob: getLocalDirs(): [/Users/slo/Java/Apache/hadoop-branches/hadoop-5847/build/contrib/streaming/test/mapred/local] 09/05/24 12:00:15 INFO streaming.StreamJob: Running job: job_local_0004 09/05/24 12:00:15 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:15 INFO mapred.FileInputFormat: Total input paths to process : 1 09/05/24 12:00:15 INFO mapred.MapTask: numReduceTasks: 1 09/05/24 12:00:15 INFO mapred.MapTask: io.sort.mb = 100 09/05/24 12:00:16 WARN mapred.LocalJobRunner: job_local_0004 java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:689) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:352) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:309) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:178) 09/05/24 12:00:16 INFO streaming.StreamJob: map 0% reduce 0% 09/05/24 12:00:16 INFO streaming.StreamJob: Job running in-process (local Hadoop) 09/05/24 12:00:16 ERROR streaming.StreamJob: Job not Successful! 09/05/24 12:00:16 INFO streaming.StreamJob: killJob... ------------- ---------------- ---------------
        Hide
        Giridharan Kesavan added a comment -

        should we commit this?

        Show
        Giridharan Kesavan added a comment - should we commit this?
        Hide
        steve_l added a comment -

        I'm willing to do the commit, but just need the contrib test suite to pass on a machine first. I'm testing on linux instead, assuming that the OOM problem is related to that system's configuration

        Show
        steve_l added a comment - I'm willing to do the commit, but just need the contrib test suite to pass on a machine first. I'm testing on linux instead, assuming that the OOM problem is related to that system's configuration
        Hide
        Giridharan Kesavan added a comment -

        I just committed this!

        Show
        Giridharan Kesavan added a comment - I just committed this!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #863 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/863/ )

          People

          • Assignee:
            Giridharan Kesavan
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development