HBase
  1. HBase
  2. HBASE-10782

Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.1, 0.99.0, 0.94.19, 0.96.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hadoop2 MR tests fail occasionally with output like this:

      -------------------------------------------------------------------------------
      Test set: org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
      -------------------------------------------------------------------------------
      Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 347.57 sec <<< FAILURE!
      testScanEmptyToAPP(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1)  Time elapsed: 50.047 sec  <<< ERROR!
      java.io.IOException: java.net.ConnectException: Call From liushaohui-OptiPlex-990/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
      	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
      	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
      	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:524)
      	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
      	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
      	at java.security.AccessController.doPrivileged(Native Method)
           ...
      

      The reason is that when MR job was running, the job client pulled the job status from AppMaster. When the job is completed, the AppMaster will exit. At this time, if the job client have not got the job completed event from AppMaster, it will switch to get job report from history server.

      But in HBaseTestingUtility#startMiniMapReduceCluster, the config: mapreduce.jobhistory.address is not copied to TestUtil's config.

      CRUNCH-249 reported the same problem.

        Activity

        Hide
        Liu Shaohui added a comment -

        Patch for trunk.

        Show
        Liu Shaohui added a comment - Patch for trunk.
        Hide
        Nick Dimiduk added a comment -

        Patch looks good to me. You ran with the patch and it fixes the failures for you, Liu Shaohui?

        Show
        Nick Dimiduk added a comment - Patch looks good to me. You ran with the patch and it fixes the failures for you, Liu Shaohui ?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12635269/HBASE-10782-trunk-v1.diff
        against trunk revision .
        ATTACHMENT ID: 12635269

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12635269/HBASE-10782-trunk-v1.diff against trunk revision . ATTACHMENT ID: 12635269 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9035//console This message is automatically generated.
        Hide
        Liu Shaohui added a comment -

        Nick Dimiduk
        Yes , it fixes the failures.
        Without this patch, the MR tests failed in my dev machine with large probability. I added some debug log and found the problem is wrong history server address. After applied this patch, I ran the tests for server times and they did not failed for this problem.

        Show
        Liu Shaohui added a comment - Nick Dimiduk Yes , it fixes the failures. Without this patch, the MR tests failed in my dev machine with large probability. I added some debug log and found the problem is wrong history server address. After applied this patch, I ran the tests for server times and they did not failed for this problem.
        Hide
        Liu Shaohui added a comment -

        I think the failed build 9035 has no relations with this patch and the pre-commit build always failed in recent several builds.

        Show
        Liu Shaohui added a comment - I think the failed build 9035 has no relations with this patch and the pre-commit build always failed in recent several builds.
        Hide
        Liang Xie added a comment -

        +1, will commit it tomorrow if no objection

        Show
        Liang Xie added a comment - +1, will commit it tomorrow if no objection
        Hide
        stack added a comment -

        +1

        Show
        stack added a comment - +1
        Hide
        Nick Dimiduk added a comment -

        +1

        Show
        Nick Dimiduk added a comment - +1
        Hide
        Liang Xie added a comment -

        Integreated into trunk and 0.94/0.96/0.98 branches. Thanks Liu Shaohui for your finding and patch! Thanks all for review.

        Show
        Liang Xie added a comment - Integreated into trunk and 0.94/0.96/0.98 branches. Thanks Liu Shaohui for your finding and patch! Thanks all for review.
        Hide
        Liang Xie added a comment -

        Oh, Lars Hofhansl Stack, sorry for just see your release RC mail, please feel free to change "Fix Version/s" field if it's not correct...

        Show
        Liang Xie added a comment - Oh, Lars Hofhansl Stack , sorry for just see your release RC mail, please feel free to change "Fix Version/s" field if it's not correct...
        Hide
        Lars Hofhansl added a comment -

        no problem. done.

        Show
        Lars Hofhansl added a comment - no problem. done.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #59 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/59/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523)

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #59 (See https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/59/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-0.94-security #449 (See https://builds.apache.org/job/HBase-0.94-security/449/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523)

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #449 (See https://builds.apache.org/job/HBase-0.94-security/449/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94 #1331 (See https://builds.apache.org/job/HBase-0.94/1331/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523)

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1331 (See https://builds.apache.org/job/HBase-0.94/1331/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98 #244 (See https://builds.apache.org/job/HBase-0.98/244/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579522)

        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98 #244 (See https://builds.apache.org/job/HBase-0.98/244/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579522) /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.94-JDK7 #94 (See https://builds.apache.org/job/HBase-0.94-JDK7/94/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523)

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.94-JDK7 #94 (See https://builds.apache.org/job/HBase-0.94-JDK7/94/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579523) /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #228 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/228/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579522)

        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #228 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/228/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579522) /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #5026 (See https://builds.apache.org/job/HBase-TRUNK/5026/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579520)

        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5026 (See https://builds.apache.org/job/HBase-TRUNK/5026/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579520) /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96 #363 (See https://builds.apache.org/job/hbase-0.96/363/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579521)

        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96 #363 (See https://builds.apache.org/job/hbase-0.96/363/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579521) /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in hbase-0.96-hadoop2 #248 (See https://builds.apache.org/job/hbase-0.96-hadoop2/248/)
        HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579521)

        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        Show
        Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #248 (See https://builds.apache.org/job/hbase-0.96-hadoop2/248/ ) HBASE-10782 Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf (Shaohui Liu) (liangxie: rev 1579521) /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java

          People

          • Assignee:
            Liu Shaohui
            Reporter:
            Liu Shaohui
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development