HBase
  1. HBase
  2. HBASE-5317

Fix TestHFileOutputFormat to work against hadoop 0.23

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0, 0.94.0
    • Fix Version/s: 0.92.1, 0.94.0
    • Component/s: test
    • Labels:
      None

      Description

      Running
      mvn -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

      yields this on 0.92:
      Failed tests: testColumnFamilyCompression(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): HFile for column family info-A not found

      Tests in error:
      test_TIMERANGE(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): /home/gchanan/workspace/apache92/target/test-data/276cbd0c-c771-4f81-9ba8-c464c9dd7486/test_TIMERANGE_present/_temporary/0/_temporary/_attempt_200707121733_0001_m_000000_0 (Is a directory)
      testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable
      testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable

      It looks like on trunk, this also results in an error:
      testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable

      I have a patch that fixes testColumnFamilyCompression and test_TIMERANGE, but haven't fixed the other 3 yet.

      1. HBASE-5317to0.92.patch
        7 kB
        Gregory Chanan
      2. TEST-org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.xml
        1.49 MB
        Ted Yu
      3. HBASE-5317-v6.patch
        7 kB
        Gregory Chanan
      4. HBASE-5317-v5.patch
        7 kB
        Gregory Chanan
      5. HBASE-5317-v4.patch
        7 kB
        Gregory Chanan
      6. HBASE-5317-v3.patch
        7 kB
        Gregory Chanan
      7. HBASE-5317-v1.patch
        2 kB
        Gregory Chanan
      8. HBASE-5317-v0.patch
        2 kB
        Gregory Chanan
      There are no Sub-Tasks for this issue.

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #96 (See https://builds.apache.org/job/HBase-0.92-security/96/)
        HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23
        (Gregory Taylor) (Revision 1293306)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/pom.xml
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #96 (See https://builds.apache.org/job/HBase-0.92-security/96/ ) HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23 (Gregory Taylor) (Revision 1293306) Result = FAILURE tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/pom.xml /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2669 (See https://builds.apache.org/job/HBase-TRUNK/2669/)
        HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23 (Gregory Chanan) (Revision 1293049)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/trunk/pom.xml
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2669 (See https://builds.apache.org/job/HBase-TRUNK/2669/ ) HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23 (Gregory Chanan) (Revision 1293049) Result = SUCCESS tedyu : Files : /hbase/trunk/pom.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #302 (See https://builds.apache.org/job/HBase-0.92/302/)
        HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23
        (Gregory Taylor) (Revision 1293306)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/pom.xml
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #302 (See https://builds.apache.org/job/HBase-0.92/302/ ) HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23 (Gregory Taylor) (Revision 1293306) Result = SUCCESS tedyu : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/pom.xml /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Hide
        Ted Yu added a comment -

        Integrated to 0.92 branch.

        Show
        Ted Yu added a comment - Integrated to 0.92 branch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515901/HBASE-5317to0.92.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1040//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515901/HBASE-5317to0.92.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1040//console This message is automatically generated.
        Hide
        Gregory Chanan added a comment -

        patch for 0.92

        Show
        Gregory Chanan added a comment - patch for 0.92
        Hide
        stack added a comment -

        It can go into 0.92 if you make a version (I see a bunch of failures trying to apply trunk patch). Thanks Gregory.

        Show
        stack added a comment - It can go into 0.92 if you make a version (I see a bunch of failures trying to apply trunk patch). Thanks Gregory.
        Hide
        Gregory Chanan added a comment -

        Thanks, Ted.

        I'll grant the license in the future.

        Any objection to also getting this into 0.92?

        Show
        Gregory Chanan added a comment - Thanks, Ted. I'll grant the license in the future. Any objection to also getting this into 0.92?
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #121 (See https://builds.apache.org/job/HBase-TRUNK-security/121/)
        HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23 (Gregory Chanan) (Revision 1293049)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/trunk/pom.xml
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #121 (See https://builds.apache.org/job/HBase-TRUNK-security/121/ ) HBASE-5317 Fix TestHFileOutputFormat to work against hadoop 0.23 (Gregory Chanan) (Revision 1293049) Result = SUCCESS tedyu : Files : /hbase/trunk/pom.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java
        Hide
        Ted Yu added a comment -

        Integrated patch v6 to TRUNK.

        Thanks for the patch Gregory.
        In the future, please grant license to Apache when you attach patches.

        Show
        Ted Yu added a comment - Integrated patch v6 to TRUNK. Thanks for the patch Gregory. In the future, please grant license to Apache when you attach patches.
        Hide
        Ted Yu added a comment -

        I was able to get the test to pass on Linux for patch v6.
        I will integrate patch v6 later if I don't hear objection.

        Thanks for the persistent digging, Gregory.

        Show
        Ted Yu added a comment - I was able to get the test to pass on Linux for patch v6. I will integrate patch v6 later if I don't hear objection. Thanks for the persistent digging, Gregory.
        Hide
        Ted Yu added a comment -

        I will try again when I get home.
        I noticed weird connection issues at work - inability to use Colloquy, etc

        Show
        Ted Yu added a comment - I will try again when I get home. I noticed weird connection issues at work - inability to use Colloquy, etc
        Hide
        Gregory Chanan added a comment -

        I am unable to reproduce either on my machine or on a coworker's Mac.

        Is it possible for you to try it on another machine? Perhaps some (security?) setting on your machine makes mapreduce unable to open the required port or something?

        Show
        Gregory Chanan added a comment - I am unable to reproduce either on my machine or on a coworker's Mac. Is it possible for you to try it on another machine? Perhaps some (security?) setting on your machine makes mapreduce unable to open the required port or something?
        Hide
        Ted Yu added a comment -
        LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ find . -name stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1330028372796_0001/container_1330028372796_0001_01_000001/stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1330028372796_0001/container_1330028372796_0001_01_000006/stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1330028372796_0001/container_1330028372796_0001_01_000004/stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1330028372796_0001/container_1330028372796_0001_01_000005/stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_2/application_1330028372796_0001/container_1330028372796_0001_01_000002/stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_3/application_1330028372796_0001/container_1330028372796_0001_01_000003/stderr
        LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/stderr
        WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
        LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_3/application_1330028372796_0001/container_1330028372796_0001_01_000003/stderr
        WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
        

        From org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/syslog:

        2012-02-23 12:22:34,208 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging lm-sjn-00713032.corp.ebay.com:61263 with 1 to fetcher#2
        2012-02-23 12:22:34,208 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to lm-sjn-00713032.corp.ebay.com:61263 to fetcher#2
        2012-02-23 12:22:34,469 WARN [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to lm-sjn-00713032.corp.ebay.com:61263 with 1 map outputs
        java.io.IOException: Server returned HTTP response code: 503 for URL: http://lm-sjn-00713032.corp.ebay.com:61263/mapOutput?job=job_1330028372796_0001&reduce=0&map=attempt_1330028372796_0001_m_000000_1
          at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
          at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:220)
          at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152)
        2012-02-23 12:22:34,471 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Reporting fetch failure for attempt_1330028372796_0001_m_000000_1 to jobtracker.
        2012-02-23 12:22:34,471 FATAL [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Shuffle failed with too many fetch failures and insufficient progress!
        2012-02-23 12:22:34,472 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: lm-sjn-00713032.corp.ebay.com:61263 freed by fetcher#2 in 264s
        2012-02-23 12:22:34,472 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:zhihyu (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2
        2012-02-23 12:22:34,472 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2
          at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123)
          at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371)
          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
        Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
          at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.checkReducerHealth(ShuffleScheduler.java:253)
          at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.copyFailed(ShuffleScheduler.java:187)
          at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:247)
          at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152)
        
        2012-02-23 12:22:34,476 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task
        

        Looks like 503 caused fetch failure.

        Show
        Ted Yu added a comment - LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ find . -name stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1330028372796_0001/container_1330028372796_0001_01_000001/stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1330028372796_0001/container_1330028372796_0001_01_000006/stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1330028372796_0001/container_1330028372796_0001_01_000004/stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1330028372796_0001/container_1330028372796_0001_01_000005/stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_2/application_1330028372796_0001/container_1330028372796_0001_01_000002/stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_3/application_1330028372796_0001/container_1330028372796_0001_01_000003/stderr LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/stderr WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-1_3/application_1330028372796_0001/container_1330028372796_0001_01_000003/stderr WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. From org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_0/application_1330028372796_0001/container_1330028372796_0001_01_000007/syslog: 2012-02-23 12:22:34,208 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging lm-sjn-00713032.corp.ebay.com:61263 with 1 to fetcher#2 2012-02-23 12:22:34,208 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 1 of 1 to lm-sjn-00713032.corp.ebay.com:61263 to fetcher#2 2012-02-23 12:22:34,469 WARN [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to lm-sjn-00713032.corp.ebay.com:61263 with 1 map outputs java.io.IOException: Server returned HTTP response code: 503 for URL: http: //lm-sjn-00713032.corp.ebay.com:61263/mapOutput?job=job_1330028372796_0001&reduce=0&map=attempt_1330028372796_0001_m_000000_1 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:220) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152) 2012-02-23 12:22:34,471 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Reporting fetch failure for attempt_1330028372796_0001_m_000000_1 to jobtracker. 2012-02-23 12:22:34,471 FATAL [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Shuffle failed with too many fetch failures and insufficient progress! 2012-02-23 12:22:34,472 INFO [fetcher#2] org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: lm-sjn-00713032.corp.ebay.com:61263 freed by fetcher#2 in 264s 2012-02-23 12:22:34,472 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:zhihyu (auth:SIMPLE) cause:org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2 2012-02-23 12:22:34,472 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142) Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.checkReducerHealth(ShuffleScheduler.java:253) at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.copyFailed(ShuffleScheduler.java:187) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:247) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152) 2012-02-23 12:22:34,476 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task Looks like 503 caused fetch failure.
        Hide
        Ted Yu added a comment -
        echo ${JAVA_HOME}
        /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
        
        mvn -Dhadoop.profile=23 clean test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        

        Same test failures, e.g.:

        testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)  Time elapsed: 60.857 sec  <<< FAILURE!
        java.lang.AssertionError
          at org.junit.Assert.fail(Assert.java:92)
          at org.junit.Assert.assertTrue(Assert.java:43)
          at org.junit.Assert.assertTrue(Assert.java:54)
          at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.runIncrementalPELoad(TestHFileOutputFormat.java:478)
          at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:390)
          at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoadWithSplit(TestHFileOutputFormat.java:369)
        
        Show
        Ted Yu added a comment - echo ${JAVA_HOME} / System /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home mvn -Dhadoop.profile=23 clean test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Same test failures, e.g.: testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 60.857 sec <<< FAILURE! java.lang.AssertionError at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.runIncrementalPELoad(TestHFileOutputFormat.java:478) at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.doIncrementalLoadTest(TestHFileOutputFormat.java:390) at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testMRIncrementalLoadWithSplit(TestHFileOutputFormat.java:369)
        Hide
        Gregory Chanan added a comment -

        Okay, this looks like a mvn issue.

        On linux, if JAVA_HOME is not set, mvn sets JAVA_HOME to be whatever shows up in "mvn --version"
        On mac, mvn appears to not set JAVA_HOME if it is not already set.

        So if you set JAVA_HOME to "/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home" the test should pass.
        The new version of mapreduce requires JAVA_HOME to be set and tries to execute $JAVA_HOME + "bin/java" – they should probably report a bug if JAVA_HOME is not set.

        Let me know if you want me to do anything else wrt this issue.

        Show
        Gregory Chanan added a comment - Okay, this looks like a mvn issue. On linux, if JAVA_HOME is not set, mvn sets JAVA_HOME to be whatever shows up in "mvn --version" On mac, mvn appears to not set JAVA_HOME if it is not already set. So if you set JAVA_HOME to "/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home" the test should pass. The new version of mapreduce requires JAVA_HOME to be set and tries to execute $JAVA_HOME + "bin/java" – they should probably report a bug if JAVA_HOME is not set. Let me know if you want me to do anything else wrt this issue.
        Hide
        Ted Yu added a comment -
        LM-SJN-00713032:205-hadoop zhihyu$ mvn --version
        Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800)
        Maven home: /usr/share/maven
        Java version: 1.6.0_29, vendor: Apple Inc.
        Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
        Default locale: en_US, platform encoding: MacRoman
        OS name: "mac os x", version: "10.6.8", arch: "x86_64", family: "mac"
        LM-SJN-00713032:205-hadoop zhihyu$ ls /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
        bin	bundle	include	lib	man
        
        Show
        Ted Yu added a comment - LM-SJN-00713032:205-hadoop zhihyu$ mvn --version Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800) Maven home: /usr/share/maven Java version: 1.6.0_29, vendor: Apple Inc. Java home: / System /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home Default locale: en_US, platform encoding: MacRoman OS name: "mac os x" , version: "10.6.8" , arch: "x86_64" , family: "mac" LM-SJN-00713032:205-hadoop zhihyu$ ls / System /Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home bin bundle include lib man
        Hide
        Gregory Chanan added a comment -

        Thanks very much, Ted.

        Could also you post the output of "mvn --version" where you ran the test? I'm particularly interested in the value of "Java home:" – is there something valid at the location it is giving you?

        Show
        Gregory Chanan added a comment - Thanks very much, Ted. Could also you post the output of "mvn --version" where you ran the test? I'm particularly interested in the value of "Java home:" – is there something valid at the location it is giving you?
        Hide
        Ted Yu added a comment -

        I do see similar output:

        LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ find . -name stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1329947593635_0001/container_1329947593635_0001_01_000001/stderr
        LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ find . -name syslog
        LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1329947593635_0001/container_1329947593635_0001_01_000001/stderr
        /bin/bash: /bin/java: No such file or directory
        
        Show
        Ted Yu added a comment - I do see similar output: LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ find . -name stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1329947593635_0001/container_1329947593635_0001_01_000001/stderr LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ find . -name syslog LM-SJN-00713032:org.apache.hadoop.mapred.MiniMRCluster zhihyu$ cat org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_3/application_1329947593635_0001/container_1329947593635_0001_01_000001/stderr /bin/bash: /bin/java: No such file or directory
        Hide
        Gregory Chanan added a comment -

        @Ted:
        Checked it on a mac, seeing the same failure as you. Can you confirm something for me? If you cd to target/org.apache.hadoop.mapred.MiniMRCluster and run the following commands, do you see something similar?

        $ find . -name stderr
        ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1329939539132_0001/container_1329939539132_0001_01_000001/stderr
        find . -name syslog
        $
        $ cat ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1329939539132_0001/container_1329939539132_0001_01_000001/stderr
        /bin/bash: /bin/java: No such file or directory

        Show
        Gregory Chanan added a comment - @Ted: Checked it on a mac, seeing the same failure as you. Can you confirm something for me? If you cd to target/org.apache.hadoop.mapred.MiniMRCluster and run the following commands, do you see something similar? $ find . -name stderr ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1329939539132_0001/container_1329939539132_0001_01_000001/stderr find . -name syslog $ $ cat ./org.apache.hadoop.mapred.MiniMRCluster-logDir-nm-0_2/application_1329939539132_0001/container_1329939539132_0001_01_000001/stderr /bin/bash: /bin/java: No such file or directory
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515531/TEST-org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.xml
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 48 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1003//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515531/TEST-org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.xml against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 48 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1003//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Here is what I did on MacBook:

          527  patch -p0 -i  HBASE-5317-v6.patch 
          528  mvn clean -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        

        I will attach test output.

        Show
        Ted Yu added a comment - Here is what I did on MacBook: 527 patch -p0 -i HBASE-5317-v6.patch 528 mvn clean -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat I will attach test output.
        Hide
        Gregory Chanan added a comment -

        @Ted:
        Hmm...just to check I fetched the latest applied the patch and both:

        mvn -Dhadoop.profile=23 clean test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        

        and

        mvn clean test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        

        passed. Can you clarify exactly what you ran?

        I don't know what is going on. I've noticed those tests have been failing Hadoop QA as well for a few days, and I can't reproduce those either. Do you have any idea what is going on?

        Show
        Gregory Chanan added a comment - @Ted: Hmm...just to check I fetched the latest applied the patch and both: mvn -Dhadoop.profile=23 clean test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat and mvn clean test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat passed. Can you clarify exactly what you ran? I don't know what is going on. I've noticed those tests have been failing Hadoop QA as well for a few days, and I can't reproduce those either. Do you have any idea what is going on?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515510/HBASE-5317-v6.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1000//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1000//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1000//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515510/HBASE-5317-v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1000//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1000//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1000//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Based on latest patch, I got:

        Failed tests:   testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)
          testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)
          testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)
        

        Here is one of the test failures:

        testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)  Time elapsed: 29.375 sec  <<< FAILURE!
        java.lang.AssertionError
          at org.junit.Assert.fail(Assert.java:92)
          at org.junit.Assert.assertTrue(Assert.java:43)
          at org.junit.Assert.assertTrue(Assert.java:54)
          at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.runIncrementalPELoad(TestHFileOutputFormat.java:478)
          at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeMinorCompaction(TestHFileOutputFormat.java:723)
        
        Show
        Ted Yu added a comment - Based on latest patch, I got: Failed tests: testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Here is one of the test failures: testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) Time elapsed: 29.375 sec <<< FAILURE! java.lang.AssertionError at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.runIncrementalPELoad(TestHFileOutputFormat.java:478) at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeMinorCompaction(TestHFileOutputFormat.java:723)
        Hide
        Gregory Chanan added a comment -

        This is just to match up the version on Review Board. Only minor spacing differences.

        Show
        Gregory Chanan added a comment - This is just to match up the version on Review Board. Only minor spacing differences.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3939/
        -----------------------------------------------------------

        (Updated 2012-02-22 00:01:33.392779)

        Review request for hbase, Ted Yu and Michael Stack.

        Changes
        -------

        Small changes as suggested by Ted and stack.

        Also updates a test case that was assuming the root dir was the homedir.

        Summary
        -------

        NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346, which has been reviewed but not committed yet.

        Fixes a few issues interacting with the new mapreduce in 0.23:

        • Maven dependency issue
        • Change the hbaseRootDir so that mrv2 output is not confused with an hbase table
        • Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built)

        This addresses bug HBASE-5317.
        https://issues.apache.org/jira/browse/HBASE-5317

        Diffs (updated)


        pom.xml 066c027
        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2
        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581
        src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java 7c5e108

        Diff: https://reviews.apache.org/r/3939/diff

        Testing
        -------

        Thanks,

        Gregory

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/ ----------------------------------------------------------- (Updated 2012-02-22 00:01:33.392779) Review request for hbase, Ted Yu and Michael Stack. Changes ------- Small changes as suggested by Ted and stack. Also updates a test case that was assuming the root dir was the homedir. Summary ------- NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346 , which has been reviewed but not committed yet. Fixes a few issues interacting with the new mapreduce in 0.23: Maven dependency issue Change the hbaseRootDir so that mrv2 output is not confused with an hbase table Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built) This addresses bug HBASE-5317 . https://issues.apache.org/jira/browse/HBASE-5317 Diffs (updated) pom.xml 066c027 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581 src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java 7c5e108 Diff: https://reviews.apache.org/r/3939/diff Testing ------- Thanks, Gregory
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515326/HBASE-5317-v5.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/996//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/996//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/996//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515326/HBASE-5317-v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/996//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/996//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/996//console This message is automatically generated.
        Hide
        Gregory Chanan added a comment -

        One more try at this...

        Show
        Gregory Chanan added a comment - One more try at this...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515322/HBASE-5317-v4.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/995//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/995//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/995//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515322/HBASE-5317-v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 159 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/995//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/995//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/995//console This message is automatically generated.
        Hide
        Gregory Chanan added a comment -

        @stack:
        I'm new to git and didn't add one of the files I modified .

        Show
        Gregory Chanan added a comment - @stack: I'm new to git and didn't add one of the files I modified .
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #91 (See https://builds.apache.org/job/HBase-0.92-security/91/)
        HBASE-5317 Fix testColumnFamilyCompression and test_TIMERANGE in TestHFileOutputFormat (Revision 1245724)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #91 (See https://builds.apache.org/job/HBase-0.92-security/91/ ) HBASE-5317 Fix testColumnFamilyCompression and test_TIMERANGE in TestHFileOutputFormat (Revision 1245724) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
        Hide
        stack added a comment -

        Gregory it failed with this:

        [INFO] --- maven-compiler-plugin:2.0.2:testCompile (default-testCompile) @ hbase ---
        [INFO] Compiling 331 source files to /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-classes
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD FAILURE
        [INFO] ------------------------------------------------------------------------
        [INFO] Total time: 23.287s
        [INFO] Finished at: Sat Feb 18 01:46:56 UTC 2012
        [INFO] Final Memory: 41M/424M
        [INFO] ------------------------------------------------------------------------
        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java:[79,33] cannot find symbol
        [ERROR] symbol  : method getDefaultRootDirPath()
        [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility
        [ERROR] -> [Help 1]
        

        What you reckon thats about? Is it the 0.23 profile? leaking?

        Show
        stack added a comment - Gregory it failed with this: [INFO] --- maven-compiler-plugin:2.0.2:testCompile ( default -testCompile) @ hbase --- [INFO] Compiling 331 source files to /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-classes [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 23.287s [INFO] Finished at: Sat Feb 18 01:46:56 UTC 2012 [INFO] Final Memory: 41M/424M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile ( default -testCompile) on project hbase: Compilation failure [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/src/test/java/org/apache/hadoop/hbase/client/TestMetaMigrationRemovingHTD.java:[79,33] cannot find symbol [ERROR] symbol : method getDefaultRootDirPath() [ERROR] location: class org.apache.hadoop.hbase.HBaseTestingUtility [ERROR] -> [Help 1] What you reckon thats about? Is it the 0.23 profile? leaking?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12515057/HBASE-5317-v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 7 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 158 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/985//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/985//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/985//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515057/HBASE-5317-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 158 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/985//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/985//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/985//console This message is automatically generated.
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-02-17 00:49:13, Michael Stack wrote:

        > Minor nits. +1 otherwise.

        Thanks for the reviews, Ted and Stack.

        On 2012-02-17 00:49:13, Michael Stack wrote:

        > pom.xml, line 2001

        > <https://reviews.apache.org/r/3939/diff/1/?file=75517#file75517line2001>

        >

        > Whats DS? (Sorry if I'm being extra thick here)

        Distributed Shell (hadoop-yarn-applications-distributedshell)

        On 2012-02-17 00:49:13, Michael Stack wrote:

        > src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java, line 515

        > <https://reviews.apache.org/r/3939/diff/1/?file=75519#file75519line515>

        >

        > Is this because we're running against target/classes dir?

        Right. HBase is a single module project and the maven test phase is before the maven package phase (when the jars are built).

        This is discussed briefly on the JIRA, but I think the correct long-term solution is to make HBase a multi-module project (see HBASE-4336) and make a test module that depends on the core module.

        • Gregory

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3939/#review5180
        -----------------------------------------------------------

        On 2012-02-16 21:57:38, Gregory Chanan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3939/

        -----------------------------------------------------------

        (Updated 2012-02-16 21:57:38)

        Review request for hbase, Ted Yu and Michael Stack.

        Summary

        -------

        NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346, which has been reviewed but not committed yet.

        Fixes a few issues interacting with the new mapreduce in 0.23:

        - Maven dependency issue

        - Change the hbaseRootDir so that mrv2 output is not confused with an hbase table

        - Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built)

        This addresses bug HBASE-5317.

        https://issues.apache.org/jira/browse/HBASE-5317

        Diffs

        -----

        pom.xml f24b389

        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731

        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581

        Diff: https://reviews.apache.org/r/3939/diff

        Testing

        -------

        Thanks,

        Gregory

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-02-17 00:49:13, Michael Stack wrote: > Minor nits. +1 otherwise. Thanks for the reviews, Ted and Stack. On 2012-02-17 00:49:13, Michael Stack wrote: > pom.xml, line 2001 > < https://reviews.apache.org/r/3939/diff/1/?file=75517#file75517line2001 > > > Whats DS? (Sorry if I'm being extra thick here) Distributed Shell (hadoop-yarn-applications-distributedshell) On 2012-02-17 00:49:13, Michael Stack wrote: > src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java, line 515 > < https://reviews.apache.org/r/3939/diff/1/?file=75519#file75519line515 > > > Is this because we're running against target/classes dir? Right. HBase is a single module project and the maven test phase is before the maven package phase (when the jars are built). This is discussed briefly on the JIRA, but I think the correct long-term solution is to make HBase a multi-module project (see HBASE-4336 ) and make a test module that depends on the core module. Gregory ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/#review5180 ----------------------------------------------------------- On 2012-02-16 21:57:38, Gregory Chanan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/ ----------------------------------------------------------- (Updated 2012-02-16 21:57:38) Review request for hbase, Ted Yu and Michael Stack. Summary ------- NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346 , which has been reviewed but not committed yet. Fixes a few issues interacting with the new mapreduce in 0.23: - Maven dependency issue - Change the hbaseRootDir so that mrv2 output is not confused with an hbase table - Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built) This addresses bug HBASE-5317 . https://issues.apache.org/jira/browse/HBASE-5317 Diffs ----- pom.xml f24b389 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581 Diff: https://reviews.apache.org/r/3939/diff Testing ------- Thanks, Gregory
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #287 (See https://builds.apache.org/job/HBase-0.92/287/)
        HBASE-5317 Fix testColumnFamilyCompression and test_TIMERANGE in TestHFileOutputFormat (Revision 1245724)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
        Show
        Hudson added a comment - Integrated in HBase-0.92 #287 (See https://builds.apache.org/job/HBase-0.92/287/ ) HBASE-5317 Fix testColumnFamilyCompression and test_TIMERANGE in TestHFileOutputFormat (Revision 1245724) Result = SUCCESS stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3939/#review5180
        -----------------------------------------------------------

        Minor nits. +1 otherwise.

        pom.xml
        <https://reviews.apache.org/r/3939/#comment11342>

        Whats DS? (Sorry if I'm being extra thick here)

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        <https://reviews.apache.org/r/3939/#comment11343>

        rad!

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        <https://reviews.apache.org/r/3939/#comment11345>

        Excellent

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        <https://reviews.apache.org/r/3939/#comment11346>

        Is this because we're running against target/classes dir?

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        <https://reviews.apache.org/r/3939/#comment11347>

        Something is wrong w/ the indent here.

        • Michael

        On 2012-02-16 21:57:38, Gregory Chanan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3939/

        -----------------------------------------------------------

        (Updated 2012-02-16 21:57:38)

        Review request for hbase, Ted Yu and Michael Stack.

        Summary

        -------

        NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346, which has been reviewed but not committed yet.

        Fixes a few issues interacting with the new mapreduce in 0.23:

        - Maven dependency issue

        - Change the hbaseRootDir so that mrv2 output is not confused with an hbase table

        - Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built)

        This addresses bug HBASE-5317.

        https://issues.apache.org/jira/browse/HBASE-5317

        Diffs

        -----

        pom.xml f24b389

        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731

        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581

        Diff: https://reviews.apache.org/r/3939/diff

        Testing

        -------

        Thanks,

        Gregory

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/#review5180 ----------------------------------------------------------- Minor nits. +1 otherwise. pom.xml < https://reviews.apache.org/r/3939/#comment11342 > Whats DS? (Sorry if I'm being extra thick here) src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java < https://reviews.apache.org/r/3939/#comment11343 > rad! src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java < https://reviews.apache.org/r/3939/#comment11345 > Excellent src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java < https://reviews.apache.org/r/3939/#comment11346 > Is this because we're running against target/classes dir? src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java < https://reviews.apache.org/r/3939/#comment11347 > Something is wrong w/ the indent here. Michael On 2012-02-16 21:57:38, Gregory Chanan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/ ----------------------------------------------------------- (Updated 2012-02-16 21:57:38) Review request for hbase, Ted Yu and Michael Stack. Summary ------- NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346 , which has been reviewed but not committed yet. Fixes a few issues interacting with the new mapreduce in 0.23: - Maven dependency issue - Change the hbaseRootDir so that mrv2 output is not confused with an hbase table - Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built) This addresses bug HBASE-5317 . https://issues.apache.org/jira/browse/HBASE-5317 Diffs ----- pom.xml f24b389 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581 Diff: https://reviews.apache.org/r/3939/diff Testing ------- Thanks, Gregory
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3939/#review5179
        -----------------------------------------------------------

        Good progress.
        Please submit to Hadoop QA for test run.

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
        <https://reviews.apache.org/r/3939/#comment11338>

        Please put a comma or period after avilable which is misspelled.

        • Ted

        On 2012-02-16 21:57:38, Gregory Chanan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3939/

        -----------------------------------------------------------

        (Updated 2012-02-16 21:57:38)

        Review request for hbase, Ted Yu and Michael Stack.

        Summary

        -------

        NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346, which has been reviewed but not committed yet.

        Fixes a few issues interacting with the new mapreduce in 0.23:

        - Maven dependency issue

        - Change the hbaseRootDir so that mrv2 output is not confused with an hbase table

        - Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built)

        This addresses bug HBASE-5317.

        https://issues.apache.org/jira/browse/HBASE-5317

        Diffs

        -----

        pom.xml f24b389

        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2

        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731

        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581

        Diff: https://reviews.apache.org/r/3939/diff

        Testing

        -------

        Thanks,

        Gregory

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/#review5179 ----------------------------------------------------------- Good progress. Please submit to Hadoop QA for test run. src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java < https://reviews.apache.org/r/3939/#comment11338 > Please put a comma or period after avilable which is misspelled. Ted On 2012-02-16 21:57:38, Gregory Chanan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/ ----------------------------------------------------------- (Updated 2012-02-16 21:57:38) Review request for hbase, Ted Yu and Michael Stack. Summary ------- NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346 , which has been reviewed but not committed yet. Fixes a few issues interacting with the new mapreduce in 0.23: - Maven dependency issue - Change the hbaseRootDir so that mrv2 output is not confused with an hbase table - Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built) This addresses bug HBASE-5317 . https://issues.apache.org/jira/browse/HBASE-5317 Diffs ----- pom.xml f24b389 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581 Diff: https://reviews.apache.org/r/3939/diff Testing ------- Thanks, Gregory
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3939/
        -----------------------------------------------------------

        Review request for hbase, Ted Yu and Michael Stack.

        Summary
        -------

        NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346, which has been reviewed but not committed yet.

        Fixes a few issues interacting with the new mapreduce in 0.23:

        • Maven dependency issue
        • Change the hbaseRootDir so that mrv2 output is not confused with an hbase table
        • Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built)

        This addresses bug HBASE-5317.
        https://issues.apache.org/jira/browse/HBASE-5317

        Diffs


        pom.xml f24b389
        src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2
        src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581

        Diff: https://reviews.apache.org/r/3939/diff

        Testing
        -------

        Thanks,

        Gregory

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3939/ ----------------------------------------------------------- Review request for hbase, Ted Yu and Michael Stack. Summary ------- NOTE: To get TestHFileOutputFormat passing also requires HBASE-5346 , which has been reviewed but not committed yet. Fixes a few issues interacting with the new mapreduce in 0.23: Maven dependency issue Change the hbaseRootDir so that mrv2 output is not confused with an hbase table Create Jars necessary to run mrv2 jobs at unit test time (before hbase jars have been built) This addresses bug HBASE-5317 . https://issues.apache.org/jira/browse/HBASE-5317 Diffs pom.xml f24b389 src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java c5222c2 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff38731 src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java ec9d581 Diff: https://reviews.apache.org/r/3939/diff Testing ------- Thanks, Gregory
        Hide
        Ted Yu added a comment -

        I agree with the proposal.

        Another option is to put the tests which depend on hbase jar(s) into a dedicated category. That category would run after hbase jar(s) are built.

        Thanks for digging.

        Show
        Ted Yu added a comment - I agree with the proposal. Another option is to put the tests which depend on hbase jar(s) into a dedicated category. That category would run after hbase jar(s) are built. Thanks for digging.
        Hide
        Gregory Chanan added a comment -

        @Ted:

        I tried approach #1, but it seems like it is not kosher to have the test depend on the maven packaging step. For example:

        mvn -Dhadoop.profile=23 -DskipTests package
        mvn -Dhadoop.profile=23 test ...
        

        works, but

        mvn -Dhadoop.profile=23 clean test ...
        

        fails because the jars aren't even built without the package step.

        #2 will probably work in both cases. It don't think it's great either, because ideally the tests should exercise the jars, because that is what downstream users would be running against. I think testing against the jars requires making hbase a multi-module project and is therefore probably:
        0) outside the scope of this JIRA
        1) a lot of work
        2) should be discussed in the community

        Therefore, I propose getting attempt #2 to work and filing a JIRA to figure out a better packaging story so we can later get rid of #2.

        Show
        Gregory Chanan added a comment - @Ted: I tried approach #1, but it seems like it is not kosher to have the test depend on the maven packaging step. For example: mvn -Dhadoop.profile=23 -DskipTests package mvn -Dhadoop.profile=23 test ... works, but mvn -Dhadoop.profile=23 clean test ... fails because the jars aren't even built without the package step. #2 will probably work in both cases. It don't think it's great either, because ideally the tests should exercise the jars, because that is what downstream users would be running against. I think testing against the jars requires making hbase a multi-module project and is therefore probably: 0) outside the scope of this JIRA 1) a lot of work 2) should be discussed in the community Therefore, I propose getting attempt #2 to work and filing a JIRA to figure out a better packaging story so we can later get rid of #2.
        Hide
        Gregory Chanan added a comment -

        @Evert:
        Yes, that is the idea. I'm hoping to have a patch that does that and solves the jar issue above by the end of today.

        Show
        Gregory Chanan added a comment - @Evert: Yes, that is the idea. I'm hoping to have a patch that does that and solves the jar issue above by the end of today.
        Hide
        Evert Arckens added a comment -

        @Gregory:
        I'm also encountering the "No descriptor for target" issue when using HBaseTestingUtility. And after reading this issue and HBASE-4025, it seems to me having HBaseTestingUtility create a subdirectory "hbase" is the right thing to do here. Are you effectively planning to do so?

        Show
        Evert Arckens added a comment - @Gregory: I'm also encountering the "No descriptor for target" issue when using HBaseTestingUtility. And after reading this issue and HBASE-4025 , it seems to me having HBaseTestingUtility create a subdirectory "hbase" is the right thing to do here. Are you effectively planning to do so?
        Hide
        Ted Yu added a comment -

        @Gregory:
        I think we should try approach #1 first.

        Thanks for your persistence.

        Show
        Ted Yu added a comment - @Gregory: I think we should try approach #1 first. Thanks for your persistence.
        Hide
        Gregory Chanan added a comment -

        Ok, did some more digging.

        In 0.23 the MRAppMaster tries to load up the OutputFileFormat, so we have to load that, which we didn't have to do prior to 0.23.
        It looks like: TableMapReduceUtil.addDependencyJars(job)
        is supposed to do this, but when I run it, no jars can be found for the classes associated with the job.

        By default, it looks like the classpath in the test is:

        {basedir}/target/test-classes:{basedir}

        /target/classes:...
        but the relevant jars are in:
        target/
        target/hbase-

        {VERSION}/hbase-{VERSION}

        /

        By the way, when I add hbase jars manually to "tmpjars" [which is what addDependencyJars does], the test passes.

        So it seems like the solution is either:
        1) Modify the classpath to include those jars
        2) Build a jar myself of the relevant classes

        I'm not sure what the correct answer is; please advise.

        Show
        Gregory Chanan added a comment - Ok, did some more digging. In 0.23 the MRAppMaster tries to load up the OutputFileFormat, so we have to load that, which we didn't have to do prior to 0.23. It looks like: TableMapReduceUtil.addDependencyJars(job) is supposed to do this, but when I run it, no jars can be found for the classes associated with the job. By default, it looks like the classpath in the test is: {basedir}/target/test-classes:{basedir} /target/classes:... but the relevant jars are in: target/ target/hbase- {VERSION}/hbase-{VERSION} / By the way, when I add hbase jars manually to "tmpjars" [which is what addDependencyJars does] , the test passes. So it seems like the solution is either: 1) Modify the classpath to include those jars 2) Build a jar myself of the relevant classes I'm not sure what the correct answer is; please advise.
        Hide
        Gregory Chanan added a comment -

        The new MiniMRCluster in 0.23 copies some new MrApp jars into the working directory of the filesystem (which is the home directory of that file system):

            FileSystem fs = FileSystem.get(conf);
            Path testRootDir = new Path("target", caller.getName() + "-tmpDir")
                .makeQualified(fs);
            Path appJar = new Path(testRootDir, "MRAppJar.jar");
        
            // Copy MRAppJar and make it private.
            Path appMasterJar = new Path(MiniMRYarnCluster.APPJAR);
            fs.copyFromLocalFile(appMasterJar, appJar);
        

        This also happens to be the HBASE_DIR:

             Path hbaseRootdir = fs.makeQualified(fs.getHomeDirectory());
             this.conf.set(HConstants.HBASE_DIR, hbaseRootdir.toString());
        

        I created a subdirectory under fs.getHomeDirectory called "hbase" and made this the HBASE_DIR. This gets past the "TestTable" issue and back to the ClassNotFoundException.

        Since these issues appear to be issues with the test utilities, rather than the test themselves, I am going to create a subtask for fixing the test themselves (which will just be the v0 patch).

        Show
        Gregory Chanan added a comment - The new MiniMRCluster in 0.23 copies some new MrApp jars into the working directory of the filesystem (which is the home directory of that file system): FileSystem fs = FileSystem.get(conf); Path testRootDir = new Path( "target" , caller.getName() + "-tmpDir" ) .makeQualified(fs); Path appJar = new Path(testRootDir, "MRAppJar.jar" ); // Copy MRAppJar and make it private . Path appMasterJar = new Path(MiniMRYarnCluster.APPJAR); fs.copyFromLocalFile(appMasterJar, appJar); This also happens to be the HBASE_DIR: Path hbaseRootdir = fs.makeQualified(fs.getHomeDirectory()); this .conf.set(HConstants.HBASE_DIR, hbaseRootdir.toString()); I created a subdirectory under fs.getHomeDirectory called "hbase" and made this the HBASE_DIR. This gets past the "TestTable" issue and back to the ClassNotFoundException. Since these issues appear to be issues with the test utilities, rather than the test themselves, I am going to create a subtask for fixing the test themselves (which will just be the v0 patch).
        Hide
        Ted Yu added a comment -

        Right.
        Can we find out why this "target" directory was created ?

        Show
        Ted Yu added a comment - Right. Can we find out why this "target" directory was created ?
        Hide
        Gregory Chanan added a comment -

        @Ted:
        FSUtils.getTableDirs() excludes a specific list of directories. Specifically:

             Arrays.asList(new String[]{ HREGION_LOGDIR_NAME, HREGION_OLDLOGDIR_NAME,
                  CORRUPT_DIR_NAME, Bytes.toString(META_TABLE_NAME),
                  Bytes.toString(ROOT_TABLE_NAME), SPLIT_LOGDIR_NAME }));
        

        So if the MiniMRCluster creates a "target" directory, it will be returned.

        Show
        Gregory Chanan added a comment - @Ted: FSUtils.getTableDirs() excludes a specific list of directories. Specifically: Arrays.asList( new String []{ HREGION_LOGDIR_NAME, HREGION_OLDLOGDIR_NAME, CORRUPT_DIR_NAME, Bytes.toString(META_TABLE_NAME), Bytes.toString(ROOT_TABLE_NAME), SPLIT_LOGDIR_NAME })); So if the MiniMRCluster creates a "target" directory, it will be returned.
        Hide
        Ted Yu added a comment -

        Not every directory under hbase.rootdir is deemed corresponding to a table.
        See the following javadoc in FSUtils.getTableDirs():

           * @return All the table directories under <code>rootdir</code>. Ignore non table hbase folders such as
           * .logs, .oldlogs, .corrupt, .META., and -ROOT- folders.
        
        Show
        Ted Yu added a comment - Not every directory under hbase.rootdir is deemed corresponding to a table. See the following javadoc in FSUtils.getTableDirs(): * @ return All the table directories under <code>rootdir</code>. Ignore non table hbase folders such as * .logs, .oldlogs, .corrupt, .META., and -ROOT- folders.
        Hide
        Ted Yu added a comment -

        handleDeprecation() call may not prevent the target directory from being created.
        We should try to find out why 'target' directory was only created for hadoop 0.23.

        Show
        Ted Yu added a comment - handleDeprecation() call may not prevent the target directory from being created. We should try to find out why 'target' directory was only created for hadoop 0.23.
        Hide
        Gregory Chanan added a comment -

        @Ted:
        I agree we shouldn't just catch the TableExistsException – that's why I didn't initially post that part of the patch. My recollection of that issue was that the MiniMRCluster is creating a "target" directory in hbase.rootdir [I could be wrong about the exact location]. When we call "table.getTableDescriptor()" it can't get a table descriptor for "target", so throws a TableExistsException. Can the handleDeprecation call prevent the target directory from being created? Or are you thinking of something else?

        It also seems a little strange to me that calling "table.getTableDescriptor();" tries to get table descriptors for everything in hbase.rootdir. Why should I get an exception thrown if hbase can't find a TableDescriptor for "target" when I am only asking about "table"?

        Show
        Gregory Chanan added a comment - @Ted: I agree we shouldn't just catch the TableExistsException – that's why I didn't initially post that part of the patch. My recollection of that issue was that the MiniMRCluster is creating a "target" directory in hbase.rootdir [I could be wrong about the exact location] . When we call "table.getTableDescriptor()" it can't get a table descriptor for "target", so throws a TableExistsException. Can the handleDeprecation call prevent the target directory from being created? Or are you thinking of something else? It also seems a little strange to me that calling "table.getTableDescriptor();" tries to get table descriptors for everything in hbase.rootdir. Why should I get an exception thrown if hbase can't find a TableDescriptor for "target" when I am only asking about "table"?
        Hide
        Ted Yu added a comment - - edited

        Configuration.handleDeprecation() is private.
        We may need to borrow deprecatedKeyMap and come up with good strategy of providing up-to-date config parameters to MRv2 (when hadoop.profile property carries value of 23).

        Show
        Ted Yu added a comment - - edited Configuration.handleDeprecation() is private. We may need to borrow deprecatedKeyMap and come up with good strategy of providing up-to-date config parameters to MRv2 (when hadoop.profile property carries value of 23).
        Hide
        Ted Yu added a comment -

        For HBASE-5317-v1.patch, I think we shouldn't simply catch TableExistsException.
        We should add missing Configuration parameters in MRv2 so that there is no TableExistsException during test.

        Show
        Ted Yu added a comment - For HBASE-5317 -v1.patch, I think we shouldn't simply catch TableExistsException. We should add missing Configuration parameters in MRv2 so that there is no TableExistsException during test.
        Hide
        Ted Yu added a comment -

        I don't see ClassNotFoundException in test output but the following may provide a clue:

        2012-02-06 09:44:48,377 WARN  [main] mapreduce.JobSubmitter(139): Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        2012-02-06 09:44:48,380 WARN  [main] mapreduce.JobSubmitter(241): No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
        2012-02-06 09:44:51,163 WARN  [ContainersLauncher #0] nodemanager.DefaultContainerExecutor(192): Exit code from task is : 127
        2012-02-06 09:44:51,165 WARN  [ContainersLauncher #0] launcher.ContainerLaunch(273): Container exited with a non-zero exit code 127
        
        Show
        Ted Yu added a comment - I don't see ClassNotFoundException in test output but the following may provide a clue: 2012-02-06 09:44:48,377 WARN [main] mapreduce.JobSubmitter(139): Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2012-02-06 09:44:48,380 WARN [main] mapreduce.JobSubmitter(241): No job jar file set. User classes may not be found. See Job or Job#setJar( String ). 2012-02-06 09:44:51,163 WARN [ContainersLauncher #0] nodemanager.DefaultContainerExecutor(192): Exit code from task is : 127 2012-02-06 09:44:51,165 WARN [ContainersLauncher #0] launcher.ContainerLaunch(273): Container exited with a non-zero exit code 127
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12513394/HBASE-5317-v1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.replication.TestReplicationPeer
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/906//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/906//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/906//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513394/HBASE-5317-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplicationPeer org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/906//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/906//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/906//console This message is automatically generated.
        Hide
        Gregory Chanan added a comment -

        This patch gets past the "TestTable" errors. Now, the problem is the job never completes (you get an assertion of job.waitForCompletion). From digging through the mapreduce logs, I found the ClassNotFoundException listed above.

        Show
        Gregory Chanan added a comment - This patch gets past the "TestTable" errors. Now, the problem is the job never completes (you get an assertion of job.waitForCompletion). From digging through the mapreduce logs, I found the ClassNotFoundException listed above.
        Hide
        Gregory Chanan added a comment -

        @Ted:
        Yes, that is the problem that causes the "TestTable" error. I'll post a patch for that tomorrow.

        Show
        Gregory Chanan added a comment - @Ted: Yes, that is the problem that causes the "TestTable" error. I'll post a patch for that tomorrow.
        Hide
        Ted Yu added a comment -

        I found the following in test output (when using hadoop 0.23):

        2012-02-04 20:18:30,563 WARN  [main] conf.Configuration(326): mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address2012-02-04 20:18:30,578 WARN  [main] conf.Configuration(343): mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
        2012-02-04 20:18:30,579 INFO  [main] mapreduce.HFileOutputFormat(320): Writing partition information to /Users/zhihyu/trunk-hbase/target/test-data/67a9efc5-8b7c-4afd-b516-39dd454334b8/mapred-working-dir/partitions_864c11de-4d4f-45a3-ba94-26aac1c751e6
        2012-02-04 20:18:30,604 WARN  [IPC Server handler 4 on 63830] master.HMaster(1725): Failed getting all descriptors
        org.apache.hadoop.hbase.TableExistsException: No descriptor for target, hdfs://localhost:63821/user/zhihyu
          at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:164)
          at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:182)
          at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1723)
        

        When using hadoop 1.0, there is no such error.

        Show
        Ted Yu added a comment - I found the following in test output (when using hadoop 0.23): 2012-02-04 20:18:30,563 WARN [main] conf.Configuration(326): mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address2012-02-04 20:18:30,578 WARN [main] conf.Configuration(343): mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 2012-02-04 20:18:30,579 INFO [main] mapreduce.HFileOutputFormat(320): Writing partition information to /Users/zhihyu/trunk-hbase/target/test-data/67a9efc5-8b7c-4afd-b516-39dd454334b8/mapred-working-dir/partitions_864c11de-4d4f-45a3-ba94-26aac1c751e6 2012-02-04 20:18:30,604 WARN [IPC Server handler 4 on 63830] master.HMaster(1725): Failed getting all descriptors org.apache.hadoop.hbase.TableExistsException: No descriptor for target, hdfs: //localhost:63821/user/zhihyu at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:164) at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:182) at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1723) When using hadoop 1.0, there is no such error.
        Hide
        Ted Yu added a comment -

        The subject of this JIRA is for all of TestHFileOutputFormat to pass for 0.23
        I think we should try to achieve that goal.

        Show
        Ted Yu added a comment - The subject of this JIRA is for all of TestHFileOutputFormat to pass for 0.23 I think we should try to achieve that goal.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12513229/HBASE-5317-v0.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestImportTsv
        org.apache.hadoop.hbase.mapred.TestTableMapReduce
        org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/904//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/904//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/904//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513229/HBASE-5317-v0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/904//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/904//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/904//console This message is automatically generated.
        Hide
        Gregory Chanan added a comment -

        @Ted:
        your output is correct. There is another patch required to get past the TestTable error and to the "Error starting MRAppMaster." I didn't include it because it doesn't solve the issue, only lets you further along. I wanted to leave the patch in a "fresh" state that actually solves the issue addressed by the JIRA.

        Perhaps the best thing to do is narrow the scope of this JIRA (to just get test_TIMERANGE and testColumnFamilyCompression to pass) and then open a new JIRA to solve the remaining issues. Do you think that makes the most sense?

        Show
        Gregory Chanan added a comment - @Ted: your output is correct. There is another patch required to get past the TestTable error and to the "Error starting MRAppMaster." I didn't include it because it doesn't solve the issue, only lets you further along. I wanted to leave the patch in a "fresh" state that actually solves the issue addressed by the JIRA. Perhaps the best thing to do is narrow the scope of this JIRA (to just get test_TIMERANGE and testColumnFamilyCompression to pass) and then open a new JIRA to solve the remaining issues. Do you think that makes the most sense?
        Hide
        Ted Yu added a comment -

        Here was the command I used:

        mvn clean -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        
        Show
        Ted Yu added a comment - Here was the command I used: mvn clean -Dhadoop.profile=23 test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
        Hide
        Jimmy Xiang added a comment -

        @Ted, did you run it with Hadoop 0.23?

        Show
        Jimmy Xiang added a comment - @Ted, did you run it with Hadoop 0.23?
        Hide
        Ted Yu added a comment -

        Running TestHFileOutputFormat on MacBook, I don't see the 'Error starting MRAppMaster' message above.
        Three tests failed (after applying patch):

          testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable
          testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable
          testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable
        
        Show
        Ted Yu added a comment - Running TestHFileOutputFormat on MacBook, I don't see the 'Error starting MRAppMaster' message above. Three tests failed (after applying patch): testMRIncrementalLoad(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testMRIncrementalLoadWithSplit(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat): TestTable
        Hide
        Gregory Chanan added a comment -

        testMRIncrementalLoad (and probably the other 2) fail with TestTable; this is pretty easy to get past.

        But after that MR Job fails with:
        2012-02-03 17:02:37,982 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
        org.apache.hadoop.yarn.YarnException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.HFileOutputFormat
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:320)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init(MRAppMaster.java:213)

        Show
        Gregory Chanan added a comment - testMRIncrementalLoad (and probably the other 2) fail with TestTable; this is pretty easy to get past. But after that MR Job fails with: 2012-02-03 17:02:37,982 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster org.apache.hadoop.yarn.YarnException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.HFileOutputFormat at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:320) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.init(MRAppMaster.java:213)
        Hide
        Gregory Chanan added a comment -

        Patch v0 fixes test_TIMERANGE and testColumnFamilyCompression

        Other tests are a little more involved.

        Show
        Gregory Chanan added a comment - Patch v0 fixes test_TIMERANGE and testColumnFamilyCompression Other tests are a little more involved.

          People

          • Assignee:
            Gregory Chanan
            Reporter:
            Gregory Chanan
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development