HBase
  1. HBase
  2. HBASE-5514

Compile against hadoop 0.24-SNAPSHOT

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.92.0, 0.94.0
    • Fix Version/s: 0.94.0
    • Component/s: build, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Need to compile hbase against the latest hadoop trunk which just had NN HA merged in.

      1) add a hadoop 0.24 profile

      2) HBASE-5480

      3) HADOOP-8124 removed deprecated Syncable.sync(). It brings compile errors for hbase against hadoop trunk(0.24). TestHLogSplit and TestHLog still call the deprecated sync(). Need to replace it with hflush() so the compilation can pass.

      1. HBASE-5514-4.patch
        6 kB
        Mingjie Lai
      2. HBASE-5514-3.patch
        6 kB
        Mingjie Lai
      3. HBASE-5514-2.patch
        1 kB
        Mingjie Lai
      4. HBASE-5514.patch
        1 kB
        Mingjie Lai

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #28 (See https://builds.apache.org/job/HBase-0.94/28/)
          HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1299928)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/branches/0.94/pom.xml
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #28 (See https://builds.apache.org/job/HBase-0.94/28/ ) HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1299928) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/pom.xml /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          Hide
          Ted Yu added a comment -

          Integrated to 0.94 branch now that HBASE-5480 has been integrated to 0.94

          Show
          Ted Yu added a comment - Integrated to 0.94 branch now that HBASE-5480 has been integrated to 0.94
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #132 (See https://builds.apache.org/job/HBase-TRUNK-security/132/)
          HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1298488)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/pom.xml
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #132 (See https://builds.apache.org/job/HBase-TRUNK-security/132/ ) HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1298488) Result = FAILURE tedyu : Files : /hbase/trunk/pom.xml /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2674 (See https://builds.apache.org/job/HBase-TRUNK/2674/)
          HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1298488)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/pom.xml
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2674 (See https://builds.apache.org/job/HBase-TRUNK/2674/ ) HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) (Revision 1298488) Result = FAILURE tedyu : Files : /hbase/trunk/pom.xml /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
          Hide
          Ted Yu added a comment -

          TestSplitLogManager passed locally based on patch v4.

          Integrated to TRUNK.

          Thanks for the patch, Mingjie.

          Show
          Ted Yu added a comment - TestSplitLogManager passed locally based on patch v4. Integrated to TRUNK. Thanks for the patch, Mingjie.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12517055/HBASE-5514-4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -129 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 154 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestSplitLogManager
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestImportTsv

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1099//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1099//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1099//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517055/HBASE-5514-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 154 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1099//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1099//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1099//console This message is automatically generated.
          Hide
          Mingjie Lai added a comment -

          Per Ted's request: removed the redundant null check.

          > Can we extract the new code into a helper class or method in org.apache.hadoop.hbase.regionserver.wal package ?

          It only occurs for 2 test cases, and the code duplication is quite small. Do you really think we should have one method to cover them?

          Show
          Mingjie Lai added a comment - Per Ted's request: removed the redundant null check. > Can we extract the new code into a helper class or method in org.apache.hadoop.hbase.regionserver.wal package ? It only occurs for 2 test cases, and the code duplication is quite small. Do you really think we should have one method to cover them?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12516981/HBASE-5514-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 8 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -129 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 154 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestImportTsv

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1090//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1090//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1090//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516981/HBASE-5514-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -129 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 154 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1090//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1090//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1090//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Similar reflection code is introduced for both tests.
          Can we extract the new code into a helper class or method in org.apache.hadoop.hbase.regionserver.wal package ?

          +    if (syncMethod != null) {
          +      syncMethod.invoke(out, new Object[]{});
          +    }
          

          Is the above check needed ? If getMethod() calls fail, there should be exception thrown that would fail the test.

          Show
          Ted Yu added a comment - Similar reflection code is introduced for both tests. Can we extract the new code into a helper class or method in org.apache.hadoop.hbase.regionserver.wal package ? + if (syncMethod != null ) { + syncMethod.invoke(out, new Object []{}); + } Is the above check needed ? If getMethod() calls fail, there should be exception thrown that would fail the test.
          Hide
          Mingjie Lai added a comment -

          New patch:

          • add 0.24 profile to pom.xml
          • use reflection to determine which sync method to use in TestHLog and TestHLogSplit.
          Show
          Mingjie Lai added a comment - New patch: add 0.24 profile to pom.xml use reflection to determine which sync method to use in TestHLog and TestHLogSplit.
          Hide
          Mingjie Lai added a comment -

          @Zhihong, thanks for pointing it out. I thought hadoop hflush() was brought at 0.20, but actually 0.21.

          As Zhihong mentioned, the scope is broader than I originally expected.

          Rename the title from ``Minor fixes to compile against Hadoop 0.24-SNAPSHOT: replace deprecated Syncable.sync()'' -> ``Compile against hadoop 0.24-SNAPSHOT''.

          Show
          Mingjie Lai added a comment - @Zhihong, thanks for pointing it out. I thought hadoop hflush() was brought at 0.20, but actually 0.21. As Zhihong mentioned, the scope is broader than I originally expected. Rename the title from ``Minor fixes to compile against Hadoop 0.24-SNAPSHOT: replace deprecated Syncable.sync()'' -> ``Compile against hadoop 0.24-SNAPSHOT''.
          Hide
          Ted Yu added a comment -

          See the following example from src/main/java/org/apache/hadoop/hbase/util/FSUtils.java:

                try {
                  FSDataOutputStream.class.getMethod("hflush", new Class<?> []{});
                  append = true;
                } catch (NoSuchMethodException e) {
          
          Show
          Ted Yu added a comment - See the following example from src/main/java/org/apache/hadoop/hbase/util/FSUtils.java: try { FSDataOutputStream.class.getMethod( "hflush" , new Class <?> []{}); append = true ; } catch (NoSuchMethodException e) {
          Hide
          Ted Yu added a comment -

          @Mingjie:
          I am afraid you need to use reflection.
          Compiling against hadoop 1.0 I saw:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile (default-testCompile) on project hbase: Compilation failure: Compilation failure:
          [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java:[207,7] cannot find symbol
          [ERROR] symbol  : method hflush()
          [ERROR] location: class org.apache.hadoop.fs.FSDataOutputStream
          [ERROR] 
          [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java:[1269,9] cannot find symbol
          [ERROR] symbol  : method hflush()
          [ERROR] location: class org.apache.hadoop.fs.FSDataOutputStream
          

          This is because Syncable interface is different between hadoop 1.0 and hadoop 0.22/0.23

          Show
          Ted Yu added a comment - @Mingjie: I am afraid you need to use reflection. Compiling against hadoop 1.0 I saw: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:testCompile ( default -testCompile) on project hbase: Compilation failure: Compilation failure: [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java:[207,7] cannot find symbol [ERROR] symbol : method hflush() [ERROR] location: class org.apache.hadoop.fs.FSDataOutputStream [ERROR] [ERROR] /Users/zhihyu/trunk-hbase/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java:[1269,9] cannot find symbol [ERROR] symbol : method hflush() [ERROR] location: class org.apache.hadoop.fs.FSDataOutputStream This is because Syncable interface is different between hadoop 1.0 and hadoop 0.22/0.23
          Hide
          Mingjie Lai added a comment -

          patch generated with --no-prefix

          Show
          Mingjie Lai added a comment - patch generated with --no-prefix
          Hide
          Ted Yu added a comment -

          @Mingjie:
          This issue is related to HBASE-5480

          Can you generate patch with --no-prefix ?

          Show
          Ted Yu added a comment - @Mingjie: This issue is related to HBASE-5480 Can you generate patch with --no-prefix ?
          Hide
          Mingjie Lai added a comment -

          Patch for trunk, 0.92.

          Show
          Mingjie Lai added a comment - Patch for trunk, 0.92.

            People

            • Assignee:
              Mingjie Lai
              Reporter:
              Mingjie Lai
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development