[#HADOOP-4205] [Hive] metastore and ql to use the refactored SerDe library

Hadoop Common

[Hive] metastore and ql to use the refactored SerDe library

Created: 18/Sep/08 08:58 PM Updated: 08/Jul/09 05:06 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

0.19.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
HADOOP-4205.1.patch	2008-09-18 11:02 PM	Zheng Shao	1.57 MB
HADOOP-4205.2.2.patch	2008-09-19 06:54 PM	Zheng Shao	2.13 MB
HADOOP-4205.2.patch	2008-09-19 03:01 AM	Zheng Shao	2.13 MB
HADOOP-4205.3.patch	2008-09-19 10:17 PM	Zheng Shao	2.14 MB

Hadoop Flags:	Reviewed
Release Note:	Improved Hive metastore and ql to use the refactored SerDe library.
Resolution Date:	19/Sep/08 11:58 PM

Description

« Hide

Metastore and ql should use the refactored SerDe library.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Zheng Shao added a comment - 18/Sep/08 10:11 PM

I was able to remove 15 classes because the functionality is moved to (be shared with) serde and ObjectInspector.

List of files added:
A src/java/org/apache/hadoop/hive/ql/io/HiveKey.java

List of classes removed (15 classes + 2 tests):
D src/test/org/apache/hadoop/hive/ql/exec/TestCompositeHiveObject.java
D src/test/org/apache/hadoop/hive/ql/exec/TestWritables.java
D src/java/org/apache/hadoop/hive/ql/exec/PrimitiveHiveObject.java
D src/java/org/apache/hadoop/hive/ql/exec/NullHiveObject.java
D src/java/org/apache/hadoop/hive/ql/exec/HiveObject.java
D src/java/org/apache/hadoop/hive/ql/exec/TableHiveObject.java
D src/java/org/apache/hadoop/hive/ql/exec/CompositeHiveObject.java
D src/java/org/apache/hadoop/hive/ql/exec/LabeledCompositeHiveObject.java
D src/java/org/apache/hadoop/hive/ql/io/NaiiveJSONSerializer.java
D src/java/org/apache/hadoop/hive/ql/io/WritableComparableHiveObject.java
D src/java/org/apache/hadoop/hive/ql/io/NoTagHiveObjectComparator.java
D src/java/org/apache/hadoop/hive/ql/io/NaiiveSerializer.java
D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableHiveObject.java
D src/java/org/apache/hadoop/hive/ql/io/HiveObjectComparator.java
D src/java/org/apache/hadoop/hive/ql/io/NoTagWritableComparableHiveObject.java
D src/java/org/apache/hadoop/hive/ql/io/HiveObjectSerializer.java
D src/java/org/apache/hadoop/hive/ql/io/WritableHiveObject.java

However the map/reduce boundary part is not optimized. I will optimize that together with DynamicSerDe.

[ Permlink | « Hide ]

Zheng Shao added a comment - 19/Sep/08 03:01 AM

This patch includes changes for this issue as well as:

http://issues.apache.org/jira/browse/HADOOP-4169 (Joydeep)
http://issues.apache.org/jira/browse/HADOOP-4175 (Prasad)
Namit's changes from svn 122459.
Prasad's changes from svn 121894.

[ Permlink | « Hide ]

Zheng Shao added a comment - 19/Sep/08 06:54 PM

rename for hadoop QA to pick it up.

[ Permlink | « Hide ]

Zheng Shao added a comment - 19/Sep/08 10:17 PM

resolved conflict for the commit for CREATE FUNCTION .

[ Permlink | « Hide ]

Zheng Shao added a comment - 19/Sep/08 10:35 PM

> ant -Dforrest.home=/XXX/forrest -Djava5.home=/usr/local/jdk1.5.0_07 -Dfindbugs.home=/XXX/findbugs -Dpatch.file=/home/zshao/~~HADOOP-4205~~.3.patch test-patch

...
[exec] +1 overall.

[exec] +1 @author. The patch does not contain any @author tags.

[exec] +1 tests included. The patch appears to include 700 new or modified tests.

[exec] +1 javadoc. The javadoc tool did not generate any warning messages.

[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================

BUILD SUCCESSFUL
Total time: 8 minutes 27 seconds

[ Permlink | « Hide ]

Zheng Shao added a comment - 19/Sep/08 11:58 PM

Committed. svn version: 697291.

[ Permlink | « Hide ]

Hadoop QA added a comment - 20/Sep/08 12:37 AM

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390545/HADOOP-4205.3.patch
against trunk revision 697241.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 700 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3335/console

This message is automatically generated.

[ Permlink | « Hide ]

Hudson added a comment - 22/Sep/08 03:18 PM

Integrated in Hadoop-trunk #611 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/)