Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4205

[Hive] metastore and ql to use the refactored SerDe library

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.19.0
    • None
    • None
    • Reviewed
    • Improved Hive metastore and ql to use the refactored SerDe library.

    Description

      Metastore and ql should use the refactored SerDe library.

      Attachments

        1. HADOOP-4205.1.patch
          1.57 MB
          Zheng Shao
        2. HADOOP-4205.2.2.patch
          2.13 MB
          Zheng Shao
        3. HADOOP-4205.2.patch
          2.13 MB
          Zheng Shao
        4. HADOOP-4205.3.patch
          2.14 MB
          Zheng Shao

        Activity

          zshao Zheng Shao added a comment -

          I was able to remove 15 classes because the functionality is moved to (be shared with) serde and ObjectInspector.

          List of files added:
          A src/java/org/apache/hadoop/hive/ql/io/HiveKey.java

          List of classes removed (15 classes + 2 tests):
          D src/test/org/apache/hadoop/hive/ql/exec/TestCompositeHiveObject.java
          D src/test/org/apache/hadoop/hive/ql/exec/TestWritables.java
          D src/java/org/apache/hadoop/hive/ql/exec/PrimitiveHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/exec/NullHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/exec/HiveObject.java
          D src/java/org/apache/hadoop/hive/ql/exec/TableHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/exec/CompositeHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/exec/LabeledCompositeHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/io/NaiiveJSONSerializer.java
          D src/java/org/apache/hadoop/hive/ql/io/WritableComparableHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/io/NoTagHiveObjectComparator.java
          D src/java/org/apache/hadoop/hive/ql/io/NaiiveSerializer.java
          D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/io/HiveObjectComparator.java
          D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableComparableHiveObject.java
          D src/java/org/apache/hadoop/hive/ql/io/HiveObjectSerializer.java
          D src/java/org/apache/hadoop/hive/ql/io/WritableHiveObject.java

          However the map/reduce boundary part is not optimized. I will optimize that together with DynamicSerDe.

          zshao Zheng Shao added a comment - I was able to remove 15 classes because the functionality is moved to (be shared with) serde and ObjectInspector. List of files added: A src/java/org/apache/hadoop/hive/ql/io/HiveKey.java List of classes removed (15 classes + 2 tests): D src/test/org/apache/hadoop/hive/ql/exec/TestCompositeHiveObject.java D src/test/org/apache/hadoop/hive/ql/exec/TestWritables.java D src/java/org/apache/hadoop/hive/ql/exec/PrimitiveHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/NullHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/HiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/TableHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/CompositeHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/LabeledCompositeHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/NaiiveJSONSerializer.java D src/java/org/apache/hadoop/hive/ql/io/WritableComparableHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/NoTagHiveObjectComparator.java D src/java/org/apache/hadoop/hive/ql/io/NaiiveSerializer.java D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/HiveObjectComparator.java D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableComparableHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/HiveObjectSerializer.java D src/java/org/apache/hadoop/hive/ql/io/WritableHiveObject.java However the map/reduce boundary part is not optimized. I will optimize that together with DynamicSerDe.
          zshao Zheng Shao added a comment -

          This patch includes changes for this issue as well as:

          http://issues.apache.org/jira/browse/HADOOP-4169 (Joydeep)
          http://issues.apache.org/jira/browse/HADOOP-4175 (Prasad)
          Namit's changes from svn 122459.
          Prasad's changes from svn 121894.

          zshao Zheng Shao added a comment - This patch includes changes for this issue as well as: http://issues.apache.org/jira/browse/HADOOP-4169 (Joydeep) http://issues.apache.org/jira/browse/HADOOP-4175 (Prasad) Namit's changes from svn 122459. Prasad's changes from svn 121894.
          zshao Zheng Shao added a comment -

          rename for hadoop QA to pick it up.

          zshao Zheng Shao added a comment - rename for hadoop QA to pick it up.
          zshao Zheng Shao added a comment -

          resolved conflict for the commit for CREATE FUNCTION .

          zshao Zheng Shao added a comment - resolved conflict for the commit for CREATE FUNCTION .
          zshao Zheng Shao added a comment -

          > ant -Dforrest.home=/XXX/forrest -Djava5.home=/usr/local/jdk1.5.0_07 -Dfindbugs.home=/XXX/findbugs -Dpatch.file=/home/zshao/HADOOP-4205.3.patch test-patch

          ...
          [exec] +1 overall.

          [exec] +1 @author. The patch does not contain any @author tags.

          [exec] +1 tests included. The patch appears to include 700 new or modified tests.

          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.

          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

          [exec] ======================================================================
          [exec] ======================================================================
          [exec] Finished build.
          [exec] ======================================================================
          [exec] ======================================================================

          BUILD SUCCESSFUL
          Total time: 8 minutes 27 seconds

          zshao Zheng Shao added a comment - > ant -Dforrest.home=/XXX/forrest -Djava5.home=/usr/local/jdk1.5.0_07 -Dfindbugs.home=/XXX/findbugs -Dpatch.file=/home/zshao/ HADOOP-4205 .3.patch test-patch ... [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 700 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== BUILD SUCCESSFUL Total time: 8 minutes 27 seconds
          zshao Zheng Shao added a comment -

          Committed. svn version: 697291.

          zshao Zheng Shao added a comment - Committed. svn version: 697291.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12390545/HADOOP-4205.3.patch
          against trunk revision 697241.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 700 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390545/HADOOP-4205.3.patch against trunk revision 697241. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 700 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/console This message is automatically generated.
          hudson Hudson added a comment -
          hudson Hudson added a comment - Integrated in Hadoop-trunk #611 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/ )

          People

            zshao Zheng Shao
            zshao Zheng Shao
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: