HBase
  1. HBase
  2. HBASE-5870

Hadoop 23 compilation broken because JobTrackerRunner#getJobTracker() method is not found

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.95.2
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After HBASE-5861 on 0.94 we are left with this issue on trunk.

      $ mvn clean test -PlocalTests -DskipTests -Dhadoop.profile=23
      ...
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure
      [ERROR] /home/jon/proj/hbase-svn/hbase/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[1333,35] cannot find symbol
      [ERROR] symbol  : method getJobTracker()
      [ERROR] location: class org.apache.hadoop.mapred.MiniMRCluster.JobTrackerRunner
      [ERROR] -> [Help 1]
      
      1. 5870.txt
        4 kB
        Lars Hofhansl
      2. 5870-v2.txt
        5 kB
        Ted Yu

        Issue Links

          Activity

          Jonathan Hsieh created issue -
          Jonathan Hsieh made changes -
          Field Original Value New Value
          Description Afater HBASE-5861 on 0.94 we are left with this issue on trunk.

          {code}
          $ mvn clean test -PlocalTests -DskipTests -Dhadoop.profile=23
          ...
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure
          [ERROR] /home/jon/proj/hbase-svn/hbase/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[1333,35] cannot find symbol
          [ERROR] symbol : method getJobTracker()
          [ERROR] location: class org.apache.hadoop.mapred.MiniMRCluster.JobTrackerRunner
          [ERROR] -> [Help 1]
          {code}
          After HBASE-5861 on 0.94 we are left with this issue on trunk.

          {code}
          $ mvn clean test -PlocalTests -DskipTests -Dhadoop.profile=23
          ...
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure
          [ERROR] /home/jon/proj/hbase-svn/hbase/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[1333,35] cannot find symbol
          [ERROR] symbol : method getJobTracker()
          [ERROR] location: class org.apache.hadoop.mapred.MiniMRCluster.JobTrackerRunner
          [ERROR] -> [Help 1]
          {code}
          Jonathan Hsieh made changes -
          Link This issue relates to HBASE-5861 [ HBASE-5861 ]
          Ted Yu made changes -
          Summary Hadoop 23 compile broken because can't find HBaseTestingUtility#getJobTracker() method Hadoop 23 compilation broken because JobTrackerRunner#getJobTracker() method is not found
          Hide
          Ted Yu added a comment -

          https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/136/console was the last build which didn't show this compilation error.

          Show
          Ted Yu added a comment - https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/136/console was the last build which didn't show this compilation error.
          Hide
          Lars Hofhansl added a comment -

          Ted's proposed patch extracted from parent.

          Show
          Lars Hofhansl added a comment - Ted's proposed patch extracted from parent.
          Lars Hofhansl made changes -
          Attachment 5870.txt [ 12524109 ]
          Lars Hofhansl made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524109/5870.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestRegionRebalancing
          org.apache.hadoop.hbase.replication.TestReplication

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1642//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1642//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1642//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524109/5870.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.replication.TestReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1642//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1642//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1642//console This message is automatically generated.
          Hide
          Jonathan Hsieh added a comment -

          In trunk, this bug was introduced in HBASE-5760.

          Show
          Jonathan Hsieh added a comment - In trunk, this bug was introduced in HBASE-5760 .
          Hide
          Jonathan Hsieh added a comment -

          @LarsH Hm, looks like this got committed here as HBASE-5848:

          ------------------------------------------------------------------------
          r1330105 | larsh | 2012-04-24 22:07:15 -0700 (Tue, 24 Apr 2012) | 1 line
          
          HBASE-5848 Addendum
          

          Maybe we should revert that and then recommit with a more appropriate commit message?

          Show
          Jonathan Hsieh added a comment - @LarsH Hm, looks like this got committed here as HBASE-5848 : ------------------------------------------------------------------------ r1330105 | larsh | 2012-04-24 22:07:15 -0700 (Tue, 24 Apr 2012) | 1 line HBASE-5848 Addendum Maybe we should revert that and then recommit with a more appropriate commit message?
          Hide
          Lars Hofhansl added a comment -

          Lemme clean up this mess.

          Show
          Lars Hofhansl added a comment - Lemme clean up this mess.
          Hide
          Lars Hofhansl added a comment -

          Reverted.

          Show
          Lars Hofhansl added a comment - Reverted.
          Hide
          Lars Hofhansl added a comment -

          Test run looked good, though. So I'm +1 on commit.

          Show
          Lars Hofhansl added a comment - Test run looked good, though. So I'm +1 on commit.
          Ted Yu made changes -
          Assignee Zhihong Yu [ zhihyu@ebaysf.com ]
          Hide
          Ted Yu added a comment -

          I ran the patch against 0.23 profile.
          I got one test failure:

          testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport)  Time elapsed: 2.583 sec  <<< ERROR!
          java.io.FileNotFoundException: File does not exist: /Users/zhihyu/.m2/repository/org/apache/hadoop/hadoop-common/0.23.2-SNAPSHOT/hadoop-common-0.23.2-SNAPSHOT.jar
          

          But the jar was there:

          -rw-r--r--  1 zhihyu  110088321  1768854 Apr 24 11:23 /Users/zhihyu/.m2/repository/org/apache/hadoop/hadoop-common/0.23.2-SNAPSHOT/hadoop-common-0.23.2-SNAPSHOT.jar
          
          Show
          Ted Yu added a comment - I ran the patch against 0.23 profile. I got one test failure: testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time elapsed: 2.583 sec <<< ERROR! java.io.FileNotFoundException: File does not exist: /Users/zhihyu/.m2/repository/org/apache/hadoop/hadoop-common/0.23.2-SNAPSHOT/hadoop-common-0.23.2-SNAPSHOT.jar But the jar was there: -rw-r--r-- 1 zhihyu 110088321 1768854 Apr 24 11:23 /Users/zhihyu/.m2/repository/org/apache/hadoop/hadoop-common/0.23.2-SNAPSHOT/hadoop-common-0.23.2-SNAPSHOT.jar
          Hide
          Jonathan Hsieh added a comment -

          I ran into something similar but it was complaining about a ZK jar – when testing HBASE-5861. I think the error is from trying to find the jar in hdfs for the MR job.

          Is this failure it consistent?

          My guess is that something in HBASE-5760 is causing this.

          Show
          Jonathan Hsieh added a comment - I ran into something similar but it was complaining about a ZK jar – when testing HBASE-5861 . I think the error is from trying to find the jar in hdfs for the MR job. Is this failure it consistent? My guess is that something in HBASE-5760 is causing this.
          Hide
          Ted Yu added a comment -

          The failure is consistent:

          testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport)  Time elapsed: 2.552 sec  <<< ERROR!
          java.io.FileNotFoundException: File does not exist: /Users/zhihyu/.m2/repository/org/apache/hadoop/hadoop-common/0.23.2-SNAPSHOT/hadoop-common-0.23.2-SNAPSHOT.jar
            at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:729)
            at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
            at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
            at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:246)
            at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:284)
            at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:355)
            at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
            at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
            at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
            at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1239)
            at org.apache.hadoop.hbase.mapreduce.TestImportExport.testSimpleCase(TestImportExport.java:114)
          
          Show
          Ted Yu added a comment - The failure is consistent: testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time elapsed: 2.552 sec <<< ERROR! java.io.FileNotFoundException: File does not exist: /Users/zhihyu/.m2/repository/org/apache/hadoop/hadoop-common/0.23.2-SNAPSHOT/hadoop-common-0.23.2-SNAPSHOT.jar at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:729) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:246) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:284) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:355) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1239) at org.apache.hadoop.hbase.mapreduce.TestImportExport.testSimpleCase(TestImportExport.java:114)
          Hide
          Ted Yu added a comment -

          Patch v2 fills in obtainJobConf() for MapreduceV2Shim.

          getJobTrackerConf() creates a new JobConf. So setting config param in the returned JobConf is not effective.

          Show
          Ted Yu added a comment - Patch v2 fills in obtainJobConf() for MapreduceV2Shim. getJobTrackerConf() creates a new JobConf. So setting config param in the returned JobConf is not effective.
          Ted Yu made changes -
          Attachment 5870-v2.txt [ 12524309 ]
          Hide
          Ted Yu added a comment -

          Even in build #136 TestImportExport failed, due to a different exception:
          https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/136/testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/org_apache_hadoop_hbase_mapreduce_TestImportExport/

          I suggest checking in patch v2 and investigate TestImportExport using another JIRA.

          Show
          Ted Yu added a comment - Even in build #136 TestImportExport failed, due to a different exception: https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-23/136/testReport/org.apache.hadoop.hbase.mapreduce/TestImportExport/org_apache_hadoop_hbase_mapreduce_TestImportExport/ I suggest checking in patch v2 and investigate TestImportExport using another JIRA.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524309/5870-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestRegionRebalancing
          org.apache.hadoop.hbase.master.TestSplitLogManager

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1648//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1648//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1648//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524309/5870-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1648//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1648//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1648//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          The two failed tests passed locally.

          Show
          Ted Yu added a comment - The two failed tests passed locally.
          Ted Yu made changes -
          Attachment 5870-v2.txt [ 12524322 ]
          Ted Yu made changes -
          Attachment 5870-v2.txt [ 12524309 ]
          Hide
          Jonathan Hsieh added a comment -

          +1. (either as is or after addressing this following nit).

          nit: Would throwing IllegalXxxException make sense instead of returning null? Since this this is testing only I don't feel strongly about this but if it does end up going into the production code it would matter more.

          Filing follow-on issue sounds good to me.

          Show
          Jonathan Hsieh added a comment - +1. (either as is or after addressing this following nit). nit: Would throwing IllegalXxxException make sense instead of returning null? Since this this is testing only I don't feel strongly about this but if it does end up going into the production code it would matter more. Filing follow-on issue sounds good to me.
          Ted Yu made changes -
          Link This issue is related to HBASE-5876 [ HBASE-5876 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12524322/5870-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1649//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1649//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1649//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524322/5870-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1649//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1649//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1649//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Will integrate later this afternoon if there is no objection.

          Show
          Ted Yu added a comment - Will integrate later this afternoon if there is no objection.
          Hide
          Jonathan Hsieh added a comment -

          This test didn't register – TestSplitTransactionOnCluster – I'm assuming this is a flaky and passes most of the time.

          Show
          Jonathan Hsieh added a comment - This test didn't register – TestSplitTransactionOnCluster – I'm assuming this is a flaky and passes most of the time.
          Hide
          Ted Yu added a comment -

          I ran the test suite and TestSplitTransactionOnCluster passed:

          Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.737 sec
          

          I ran it again standalone and it passed.

          Show
          Ted Yu added a comment - I ran the test suite and TestSplitTransactionOnCluster passed: Running org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.737 sec I ran it again standalone and it passed.
          Hide
          Ted Yu added a comment -

          Integrated to trunk.

          Thanks for the review, Lars and Jon.

          Show
          Ted Yu added a comment - Integrated to trunk. Thanks for the review, Lars and Jon.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2815 (See https://builds.apache.org/job/HBase-TRUNK/2815/)
          HBASE-5870 Hadoop 23 compilation broken because JobTrackerRunner#getJobTracker() method is not found (Revision 1330563)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2815 (See https://builds.apache.org/job/HBase-TRUNK/2815/ ) HBASE-5870 Hadoop 23 compilation broken because JobTrackerRunner#getJobTracker() method is not found (Revision 1330563) Result = SUCCESS tedyu : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java
          Ted Yu made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #184 (See https://builds.apache.org/job/HBase-TRUNK-security/184/)
          HBASE-5870 Hadoop 23 compilation broken because JobTrackerRunner#getJobTracker() method is not found (Revision 1330563)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #184 (See https://builds.apache.org/job/HBase-TRUNK-security/184/ ) HBASE-5870 Hadoop 23 compilation broken because JobTrackerRunner#getJobTracker() method is not found (Revision 1330563) Result = FAILURE tedyu : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/MapreduceTestingShim.java
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Hide
          stack added a comment -

          Marking closed.

          Show
          stack added a comment - Marking closed.
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Ted Yu
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development