[HIVE-11095] SerDeUtils another bug ,when Text is reused - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 0.14.0, 1.0.0, 1.2.0
Fix Version/s: 1.3.0, 2.0.0
Component/s: API, CLI
Labels:
None
Environment:

Hadoop 2.3.0-cdh5.0.0
Hive 0.14

External issue URL:
https://issues.apache.org/jira/browse/HIVE-10983
External issue ID:
10983

Description

The method transformTextFromUTF8 have a  error bug, It invoke a bad method of Text,getBytes()!
The method getBytes of Text returns the raw bytes; however, only data up to Text.length is valid.A better way is  use copyBytes()  if you need the returned array to be precisely the length of the data.
But the copyBytes is added behind hadoop1.

How I found this bug？
When i query data from a lzo table ， I found in results ： the length of the current row is always largr than the previous row， and sometimes，the current row contains the contents of the previous row。 For example ，i execute a sql ,

select * from web_searchhub where logdate=2015061003

the result of sql see blow.Notice that ,the second row content contains the first row content.

INFO [03:00:05.589] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42098,session=3151,thread=254 2015061003
INFO [03:00:05.594] <18941e66-9962-44ad-81bc-3519f47ba274> session=901,thread=223ession=3151,thread=254 2015061003

The content of origin lzo file content see below ,just 2 rows.

INFO [03:00:05.635] <b88e0473-7530-494c-82d8-e2d2ebd2666c_forweb> session=3148,thread=285
INFO [03:00:05.635] HttpFrontServer::FrontSH msgRecv:Remote=/10.13.193.68:42095,session=3148,thread=285

I think this error is caused by the Text reuse,and I found the solutions .
Addicational, table create sql is :

CREATE EXTERNAL TABLE `web_searchhub`(
`line` string)
PARTITIONED BY (
`logdate` string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '
U0000'
WITH SERDEPROPERTIES (
'serialization.encoding'='GBK')
STORED AS INPUTFORMAT "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat";
LOCATION
'viewfs://nsX/user/hive/warehouse/raw.db/web/web_searchhub'