Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.96.2, 0.98.1, 0.99.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HFileOutputFormat/KeyValueSortReducer use KeyValue. We should deprecate them and use Cell instead.

      1. hbase-10526_v1.1.patch
        96 kB
        Jimmy Xiang
      2. hbase-10526_v2.patch
        96 kB
        Jimmy Xiang
      3. hbase-10526_v3.patch
        96 kB
        Jimmy Xiang
      4. hbase-10526.patch
        96 kB
        Jimmy Xiang

        Activity

        Hide
        Jimmy Xiang added a comment -

        Attached a patch that added a new class HFileOutputFormat2. To share the code, most of the functions in HFileOutputFormat are moved to HFileOutputFormat2.

        As long as old client applications don't use the package protected methods/class in HFileOutputFormat, they will still work as before.

        It's recommended to use HFileOutputFormat2.configureIncrementalLoad to configure a corresponding Job. If users want to configure the Job themselves, they still need to specify the output value class to KeyValue (setOutputValueClass) because MR ensures the output value class is an actual class instead of interface/abstract class.

        Show
        Jimmy Xiang added a comment - Attached a patch that added a new class HFileOutputFormat2. To share the code, most of the functions in HFileOutputFormat are moved to HFileOutputFormat2. As long as old client applications don't use the package protected methods/class in HFileOutputFormat, they will still work as before. It's recommended to use HFileOutputFormat2.configureIncrementalLoad to configure a corresponding Job. If users want to configure the Job themselves, they still need to specify the output value class to KeyValue (setOutputValueClass) because MR ensures the output value class is an actual class instead of interface/abstract class.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12629154/hbase-10526.patch
        against trunk revision .
        ATTACHMENT ID: 12629154

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        -1 hadoop1.0. The patch failed to compile against the hadoop 1.0 profile.
        Here is snippet of errors:

        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hbase-server: Compilation failure: Compilation failure:
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java:[328,12] cannot find symbol
        [ERROR] symbol  : method file(org.apache.hadoop.fs.Path)
        [ERROR] location: class org.apache.hadoop.io.SequenceFile.Writer
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java:[328,41] cannot find symbol
        [ERROR] symbol  : method keyClass(java.lang.Class<org.apache.hadoop.hbase.io.ImmutableBytesWritable>)
        --
        org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hbase-server: Compilation failure
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
        	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
        --
        Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure
        	at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729)
        	at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:128)
        	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
        	... 19 more

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8711//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629154/hbase-10526.patch against trunk revision . ATTACHMENT ID: 12629154 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. -1 hadoop1.0 . The patch failed to compile against the hadoop 1.0 profile. Here is snippet of errors: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile ( default -compile) on project hbase-server: Compilation failure: Compilation failure: [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java:[328,12] cannot find symbol [ERROR] symbol : method file(org.apache.hadoop.fs.Path) [ERROR] location: class org.apache.hadoop.io.SequenceFile.Writer [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java:[328,41] cannot find symbol [ERROR] symbol : method keyClass(java.lang. Class <org.apache.hadoop.hbase.io.ImmutableBytesWritable>) -- org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile ( default -compile) on project hbase-server: Compilation failure at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) -- Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729) at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:128) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8711//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        Attached v1.1, using the deprecated method from SequenceFile.

        Show
        Jimmy Xiang added a comment - Attached v1.1, using the deprecated method from SequenceFile.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12629168/hbase-10526_v1.1.patch
        against trunk revision .
        ATTACHMENT ID: 12629168

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        -1 hadoop1.0. The patch failed to compile against the hadoop 1.0 profile.
        Here is snippet of errors:

        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hbase-server: Compilation failure: Compilation failure:
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java:[184,19] cannot find symbol
        [ERROR] symbol  : method getInstance(org.apache.hadoop.conf.Configuration)
        [ERROR] location: class org.apache.hadoop.mapreduce.Job
        [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java:[236,19] cannot find symbol
        [ERROR] symbol  : method getInstance(org.apache.hadoop.conf.Configuration)
        --
        org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hbase-server: Compilation failure
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
        	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
        --
        Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure
        	at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729)
        	at org.apache.maven.plugin.TestCompilerMojo.execute(TestCompilerMojo.java:161)
        	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
        	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
        	... 19 more

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8713//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629168/hbase-10526_v1.1.patch against trunk revision . ATTACHMENT ID: 12629168 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. -1 hadoop1.0 . The patch failed to compile against the hadoop 1.0 profile. Here is snippet of errors: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile ( default -testCompile) on project hbase-server: Compilation failure: Compilation failure: [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java:[184,19] cannot find symbol [ERROR] symbol : method getInstance(org.apache.hadoop.conf.Configuration) [ERROR] location: class org.apache.hadoop.mapreduce.Job [ERROR] /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java:[236,19] cannot find symbol [ERROR] symbol : method getInstance(org.apache.hadoop.conf.Configuration) -- org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile ( default -testCompile) on project hbase-server: Compilation failure at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) -- Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:729) at org.apache.maven.plugin.TestCompilerMojo.execute(TestCompilerMojo.java:161) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8713//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        Attached patch v2 that is good with hadoop 1.0.

        Show
        Jimmy Xiang added a comment - Attached patch v2 that is good with hadoop 1.0.
        Hide
        Nick Dimiduk added a comment -

        I'm not a fan of maintaining both versions of this thing – and especially not a fan of the new class's name because it confuses the file format version.

        Can we instead deprecate the KeyValue version in 0.96, 0.98 and replace it with the Cell version on trunk?

        Show
        Nick Dimiduk added a comment - I'm not a fan of maintaining both versions of this thing – and especially not a fan of the new class's name because it confuses the file format version. Can we instead deprecate the KeyValue version in 0.96, 0.98 and replace it with the Cell version on trunk?
        Hide
        Jimmy Xiang added a comment -

        It's better not to simply replace it, which may impact the upgrade path of some existing old applications. As to the class name, I am open for suggestions.

        Show
        Jimmy Xiang added a comment - It's better not to simply replace it, which may impact the upgrade path of some existing old applications. As to the class name, I am open for suggestions.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12629622/hbase-10526_v2.patch
        against trunk revision .
        ATTACHMENT ID: 12629622

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces the following lines longer than 100:
        + TaskAttemptContext context = hadoop.createTestTaskAttemptContext(job, "attempt_201402131733_0001_m_000000_0");
        + Map<String, Compression.Algorithm> familyToCompression = new HashMap<String, Compression.Algorithm>();
        + "(reader: " + reader + ")", hcd.getCompression(), reader.getFileContext().getCompression());

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.util.TestHBaseFsck

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629622/hbase-10526_v2.patch against trunk revision . ATTACHMENT ID: 12629622 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + TaskAttemptContext context = hadoop.createTestTaskAttemptContext(job, "attempt_201402131733_0001_m_000000_0"); + Map<String, Compression.Algorithm> familyToCompression = new HashMap<String, Compression.Algorithm>(); + "(reader: " + reader + ")", hcd.getCompression(), reader.getFileContext().getCompression()); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.util.TestHBaseFsck Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8738//console This message is automatically generated.
        Hide
        Jonathan Hsieh added a comment -

        Nick D

        I'm not a fan of maintaining both versions of this thing – and especially not a fan of the new class's name because it confuses the file format version.

        Can we instead deprecate the KeyValue version in 0.96, 0.98 and replace it with the Cell version on trunk?

        jimmy

        It's better not to simply replace it, which may impact the upgrade path of some existing old applications. As to the class name, I am open for suggestions.

        v2 has HFIleOutputFormat (KV version) deprecated and pointing to HFileOutputFormat2 (Cell version). I'd be fine if 0.98 and 0.96 had both with the old KV version deprecated. (I'd argue this is early enough in both branch's lives and this is a compat problem from 0.94.) I'd also prefer it if trunk could remove the old version and only have the Cell version.

        I think HFileOutputFormat2 is as good as any. We are still writing the same files out – just providing a api cleanup.

        Show
        Jonathan Hsieh added a comment - Nick D I'm not a fan of maintaining both versions of this thing – and especially not a fan of the new class's name because it confuses the file format version. Can we instead deprecate the KeyValue version in 0.96, 0.98 and replace it with the Cell version on trunk? jimmy It's better not to simply replace it, which may impact the upgrade path of some existing old applications. As to the class name, I am open for suggestions. v2 has HFIleOutputFormat (KV version) deprecated and pointing to HFileOutputFormat2 (Cell version). I'd be fine if 0.98 and 0.96 had both with the old KV version deprecated. (I'd argue this is early enough in both branch's lives and this is a compat problem from 0.94.) I'd also prefer it if trunk could remove the old version and only have the Cell version. I think HFileOutputFormat2 is as good as any. We are still writing the same files out – just providing a api cleanup.
        Hide
        Jonathan Hsieh added a comment -

        I'm +1 on this assuming that it was mostly a copy and paste job and after the qabot cleanup.

        Show
        Jonathan Hsieh added a comment - I'm +1 on this assuming that it was mostly a copy and paste job and after the qabot cleanup.
        Hide
        Jimmy Xiang added a comment -

        Attached patch v3 that fixed the qabot warnings. Ok, will remove the old version in trunk, and keep both the new one and the older one in 0.96 and 0.98.

        Show
        Jimmy Xiang added a comment - Attached patch v3 that fixed the qabot warnings. Ok, will remove the old version in trunk, and keep both the new one and the older one in 0.96 and 0.98.
        Hide
        Nick Dimiduk added a comment -

        Appending a version number to a class name stinks of interface miss-management. However, given the generic type restrictions, I lack a better suggestion.

        Show
        Nick Dimiduk added a comment - Appending a version number to a class name stinks of interface miss-management. However, given the generic type restrictions, I lack a better suggestion.
        Hide
        Jonathan Hsieh added a comment - - edited

        append a # seems reasonable to me – given your argument should we rename metrics2 and hfilev2 as well?

        Show
        Jonathan Hsieh added a comment - - edited append a # seems reasonable to me – given your argument should we rename metrics2 and hfilev2 as well?
        Hide
        Nick Dimiduk added a comment -

        append a # seems reasonable to me – given your argument should we rename metrics2 and hfilev2 as well?

        I can't speak to metrics as I don't know well enough what changed from the first implementation to the second. Not so on the HFile because that's a different serialization format. Having versioning there is acceptable, I think, though were it a public API I would argue for managing API changes as we do with other public APIs. Even that is done relatively well, given the presence of AbstractHFileReader, etc.

        In this case, we're talking about a public API which itself has not changed (a blanket copy-paste, right?), only the detail of which class is instantiated in the template. That's not a serialization format change and it's not a significant API change (though it is one). As I said, there's no way for us to work around this in Java, so I guess that's that.

        Show
        Nick Dimiduk added a comment - append a # seems reasonable to me – given your argument should we rename metrics2 and hfilev2 as well? I can't speak to metrics as I don't know well enough what changed from the first implementation to the second. Not so on the HFile because that's a different serialization format. Having versioning there is acceptable, I think, though were it a public API I would argue for managing API changes as we do with other public APIs. Even that is done relatively well, given the presence of AbstractHFileReader, etc. In this case, we're talking about a public API which itself has not changed (a blanket copy-paste, right?), only the detail of which class is instantiated in the template. That's not a serialization format change and it's not a significant API change (though it is one). As I said, there's no way for us to work around this in Java, so I guess that's that.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12629874/hbase-10526_v3.patch
        against trunk revision .
        ATTACHMENT ID: 12629874

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12629874/hbase-10526_v3.patch against trunk revision . ATTACHMENT ID: 12629874 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8752//console This message is automatically generated.
        Hide
        Jimmy Xiang added a comment -

        Agree. The class name is not perfect. It seems to be the best we can have for now. If no objection, I will commit the patch tomorrow. Thanks.

        Show
        Jimmy Xiang added a comment - Agree. The class name is not perfect. It seems to be the best we can have for now. If no objection, I will commit the patch tomorrow. Thanks.
        Hide
        Nick Dimiduk added a comment -

        No further objection.

        Show
        Nick Dimiduk added a comment - No further objection.
        Hide
        stack added a comment -

        I would argue we keep both versions in trunk. 0.98 has shipped. We need a class deprecated for a whole major release before we can remove it. Folks moving from 0.98 to 1.0 shouldn't have to redo their MR jobs IMO. Jimmy Xiang You might open new issue for discussion of whether we should remove the old version in 1.0 so you can commit the patch here.

        Show
        stack added a comment - I would argue we keep both versions in trunk. 0.98 has shipped. We need a class deprecated for a whole major release before we can remove it. Folks moving from 0.98 to 1.0 shouldn't have to redo their MR jobs IMO. Jimmy Xiang You might open new issue for discussion of whether we should remove the old version in 1.0 so you can commit the patch here.
        Hide
        Jimmy Xiang added a comment -

        Good idea. Filed sub-task HBASE-10588 to clean up all related in one issue.

        Show
        Jimmy Xiang added a comment - Good idea. Filed sub-task HBASE-10588 to clean up all related in one issue.
        Hide
        Jimmy Xiang added a comment -

        Integrated into 0.96, 0.98, and trunk. Thanks.

        Show
        Jimmy Xiang added a comment - Integrated into 0.96, 0.98, and trunk. Thanks.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4942 (See https://builds.apache.org/job/HBase-TRUNK/4942/)
        HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570702)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4942 (See https://builds.apache.org/job/HBase-TRUNK/4942/ ) HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570702) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98 #176 (See https://builds.apache.org/job/HBase-0.98/176/)
        HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570688)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98 #176 (See https://builds.apache.org/job/HBase-0.98/176/ ) HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570688) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96 #307 (See https://builds.apache.org/job/hbase-0.96/307/)
        HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570714)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96 #307 (See https://builds.apache.org/job/hbase-0.96/307/ ) HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570714) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in hbase-0.96-hadoop2 #211 (See https://builds.apache.org/job/hbase-0.96-hadoop2/211/)
        HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570714)

        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
        • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Show
        Hudson added a comment - SUCCESS: Integrated in hbase-0.96-hadoop2 #211 (See https://builds.apache.org/job/hbase-0.96-hadoop2/211/ ) HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570714) /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-1.1 #96 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/96/)
        HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570702)

        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-1.1 #96 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/96/ ) HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570702) /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #164 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/164/)
        HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570688)

        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
        • /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
        • /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Show
        Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #164 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/164/ ) HBASE-10526 Using Cell instead of KeyValue in HFileOutputFormat (jxiang: rev 1570688) /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
        Hide
        Enis Soztutar added a comment -

        Closing this issue after 0.99.0 release.

        Show
        Enis Soztutar added a comment - Closing this issue after 0.99.0 release.

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            Jimmy Xiang
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development