Hadoop Common
  1. Hadoop Common
  2. HADOOP-4205

[Hive] metastore and ql to use the refactored SerDe library

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Improved Hive metastore and ql to use the refactored SerDe library.

      Description

      Metastore and ql should use the refactored SerDe library.

      1. HADOOP-4205.1.patch
        1.57 MB
        Zheng Shao
      2. HADOOP-4205.2.patch
        2.13 MB
        Zheng Shao
      3. HADOOP-4205.2.2.patch
        2.13 MB
        Zheng Shao
      4. HADOOP-4205.3.patch
        2.14 MB
        Zheng Shao

        Activity

        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #611 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/ )
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390545/HADOOP-4205.3.patch
        against trunk revision 697241.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 700 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390545/HADOOP-4205.3.patch against trunk revision 697241. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 700 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/console This message is automatically generated.
        Hide
        Zheng Shao added a comment -

        Committed. svn version: 697291.

        Show
        Zheng Shao added a comment - Committed. svn version: 697291.
        Hide
        Zheng Shao added a comment -

        > ant -Dforrest.home=/XXX/forrest -Djava5.home=/usr/local/jdk1.5.0_07 -Dfindbugs.home=/XXX/findbugs -Dpatch.file=/home/zshao/HADOOP-4205.3.patch test-patch

        ...
        [exec] +1 overall.

        [exec] +1 @author. The patch does not contain any @author tags.

        [exec] +1 tests included. The patch appears to include 700 new or modified tests.

        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.

        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

        [exec] ======================================================================
        [exec] ======================================================================
        [exec] Finished build.
        [exec] ======================================================================
        [exec] ======================================================================

        BUILD SUCCESSFUL
        Total time: 8 minutes 27 seconds

        Show
        Zheng Shao added a comment - > ant -Dforrest.home=/XXX/forrest -Djava5.home=/usr/local/jdk1.5.0_07 -Dfindbugs.home=/XXX/findbugs -Dpatch.file=/home/zshao/ HADOOP-4205 .3.patch test-patch ... [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 700 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== BUILD SUCCESSFUL Total time: 8 minutes 27 seconds
        Hide
        Zheng Shao added a comment -

        resolved conflict for the commit for CREATE FUNCTION .

        Show
        Zheng Shao added a comment - resolved conflict for the commit for CREATE FUNCTION .
        Hide
        Zheng Shao added a comment -

        rename for hadoop QA to pick it up.

        Show
        Zheng Shao added a comment - rename for hadoop QA to pick it up.
        Hide
        Zheng Shao added a comment -

        This patch includes changes for this issue as well as:

        http://issues.apache.org/jira/browse/HADOOP-4169 (Joydeep)
        http://issues.apache.org/jira/browse/HADOOP-4175 (Prasad)
        Namit's changes from svn 122459.
        Prasad's changes from svn 121894.

        Show
        Zheng Shao added a comment - This patch includes changes for this issue as well as: http://issues.apache.org/jira/browse/HADOOP-4169 (Joydeep) http://issues.apache.org/jira/browse/HADOOP-4175 (Prasad) Namit's changes from svn 122459. Prasad's changes from svn 121894.
        Hide
        Zheng Shao added a comment -

        I was able to remove 15 classes because the functionality is moved to (be shared with) serde and ObjectInspector.

        List of files added:
        A src/java/org/apache/hadoop/hive/ql/io/HiveKey.java

        List of classes removed (15 classes + 2 tests):
        D src/test/org/apache/hadoop/hive/ql/exec/TestCompositeHiveObject.java
        D src/test/org/apache/hadoop/hive/ql/exec/TestWritables.java
        D src/java/org/apache/hadoop/hive/ql/exec/PrimitiveHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/exec/NullHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/exec/HiveObject.java
        D src/java/org/apache/hadoop/hive/ql/exec/TableHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/exec/CompositeHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/exec/LabeledCompositeHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/io/NaiiveJSONSerializer.java
        D src/java/org/apache/hadoop/hive/ql/io/WritableComparableHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/io/NoTagHiveObjectComparator.java
        D src/java/org/apache/hadoop/hive/ql/io/NaiiveSerializer.java
        D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/io/HiveObjectComparator.java
        D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableComparableHiveObject.java
        D src/java/org/apache/hadoop/hive/ql/io/HiveObjectSerializer.java
        D src/java/org/apache/hadoop/hive/ql/io/WritableHiveObject.java

        However the map/reduce boundary part is not optimized. I will optimize that together with DynamicSerDe.

        Show
        Zheng Shao added a comment - I was able to remove 15 classes because the functionality is moved to (be shared with) serde and ObjectInspector. List of files added: A src/java/org/apache/hadoop/hive/ql/io/HiveKey.java List of classes removed (15 classes + 2 tests): D src/test/org/apache/hadoop/hive/ql/exec/TestCompositeHiveObject.java D src/test/org/apache/hadoop/hive/ql/exec/TestWritables.java D src/java/org/apache/hadoop/hive/ql/exec/PrimitiveHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/NullHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/HiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/TableHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/CompositeHiveObject.java D src/java/org/apache/hadoop/hive/ql/exec/LabeledCompositeHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/NaiiveJSONSerializer.java D src/java/org/apache/hadoop/hive/ql/io/WritableComparableHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/NoTagHiveObjectComparator.java D src/java/org/apache/hadoop/hive/ql/io/NaiiveSerializer.java D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/HiveObjectComparator.java D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableComparableHiveObject.java D src/java/org/apache/hadoop/hive/ql/io/HiveObjectSerializer.java D src/java/org/apache/hadoop/hive/ql/io/WritableHiveObject.java However the map/reduce boundary part is not optimized. I will optimize that together with DynamicSerDe.

          People

          • Assignee:
            Zheng Shao
            Reporter:
            Zheng Shao
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development