HBase
  1. HBase
  2. HBASE-10782

Hadoop2 MR tests fail occasionally because of mapreduce.jobhistory.address is no set in job conf

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.1, 0.99.0, 0.94.19, 0.96.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hadoop2 MR tests fail occasionally with output like this:

      -------------------------------------------------------------------------------
      Test set: org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
      -------------------------------------------------------------------------------
      Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 347.57 sec <<< FAILURE!
      testScanEmptyToAPP(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1)  Time elapsed: 50.047 sec  <<< ERROR!
      java.io.IOException: java.net.ConnectException: Call From liushaohui-OptiPlex-990/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
      	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
      	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
      	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:524)
      	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
      	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
      	at java.security.AccessController.doPrivileged(Native Method)
           ...
      

      The reason is that when MR job was running, the job client pulled the job status from AppMaster. When the job is completed, the AppMaster will exit. At this time, if the job client have not got the job completed event from AppMaster, it will switch to get job report from history server.

      But in HBaseTestingUtility#startMiniMapReduceCluster, the config: mapreduce.jobhistory.address is not copied to TestUtil's config.

      CRUNCH-249 reported the same problem.

        Activity

        Liu Shaohui created issue -
        Liu Shaohui made changes -
        Field Original Value New Value
        Description Hadoop2 MR tests fail occasionally with output like this:
        {code}
        -------------------------------------------------------------------------------
        Test set: org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
        -------------------------------------------------------------------------------
        Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 347.57 sec <<< FAILURE!
        testScanEmptyToAPP(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1) Time elapsed: 50.047 sec <<< ERROR!
        java.io.IOException: java.net.ConnectException: Call From liushaohui-OptiPlex-990/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
        at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
        at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
        at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:524)
        at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
        at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
        at java.security.AccessController.doPrivileged(Native Method)
             ...
        {code}
        The reason is that when MR job was running, the job client pulled the job status from AppMaster. When the job is completed, the AppMaster will exit. At this time, if the job client have not got the job completed event from AppMaster, it will try to get job report from history server.

        But in HBaseTestingUtility#startMiniMapReduceCluster, the config: mapreduce.jobhistory.address is not copied to TestUtil's config.
         
        CRUNCH-249 reported the same problem.
        Hadoop2 MR tests fail occasionally with output like this:
        {code}
        -------------------------------------------------------------------------------
        Test set: org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
        -------------------------------------------------------------------------------
        Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 347.57 sec <<< FAILURE!
        testScanEmptyToAPP(org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1) Time elapsed: 50.047 sec <<< ERROR!
        java.io.IOException: java.net.ConnectException: Call From liushaohui-OptiPlex-990/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
        at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
        at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
        at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:524)
        at org.apache.hadoop.mapreduce.Job$1.run(Job.java:314)
        at org.apache.hadoop.mapreduce.Job$1.run(Job.java:311)
        at java.security.AccessController.doPrivileged(Native Method)
             ...
        {code}
        The reason is that when MR job was running, the job client pulled the job status from AppMaster. When the job is completed, the AppMaster will exit. At this time, if the job client have not got the job completed event from AppMaster, it will switch to get job report from history server.

        But in HBaseTestingUtility#startMiniMapReduceCluster, the config: mapreduce.jobhistory.address is not copied to TestUtil's config.
         
        CRUNCH-249 reported the same problem.
        Liu Shaohui made changes -
        Attachment HBASE-10782-trunk-v1.diff [ 12635269 ]
        Liu Shaohui made changes -
        Assignee Liu Shaohui [ liushaohui ]
        Ted Yu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Liang Xie made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.96.2 [ 12325658 ]
        Fix Version/s 0.99.0 [ 12325675 ]
        Fix Version/s 0.94.18 [ 12325952 ]
        Fix Version/s 0.98.2 [ 12326505 ]
        Resolution Fixed [ 1 ]
        Lars Hofhansl made changes -
        Fix Version/s 0.94.19 [ 12326287 ]
        Fix Version/s 0.96.3 [ 12326538 ]
        Fix Version/s 0.96.2 [ 12325658 ]
        Fix Version/s 0.94.18 [ 12325952 ]
        Andrew Purtell made changes -
        Fix Version/s 0.98.1 [ 12325664 ]
        Fix Version/s 0.98.2 [ 12326505 ]
        Lars Hofhansl made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Liu Shaohui
            Reporter:
            Liu Shaohui
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development