[HCATALOG-436] JSON SerDe column misnaming on CTAS

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5, 0.4.1
• Labels: json serde
• Assignee: Sushanth Sowmyan
• Reporter: Sushanth Sowmyan
• Created: 26/Jun/12 21:17
• Updated: 15/Feb/13 21:32
• Resolved: 30/Jul/12 19:02

Description

Given an origin table as follows:

–
hive -e 'describe extended ttf'
OK
sterm string
count bigint

Detailed Table Information Table(tableName:ttf, dbName:default, owner:hive, createTime:1339518715, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:sterm, type:string, comment:null), FieldSchema(name:count, type:bigint, comment:null)], location:hdfs://localhost:54310/user/hive/warehouse/ttf, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:

{serialization.format=1}

), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[], parameters:

{numPartitions=0, numFiles=1, transient_lastDdlTime=1339518715, totalSize=2155, numRows=0, rawDataSize=0}

, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
–

On doing a CTAS, such as:

–
hive -e "create table ttf_json row format serde 'org.apache.hcatalog.data.JsonSerDe' as select * from ttf;"
–

We get a resultant table ttf_json with schema similar to ttf, but on looking at the data present in the json file itself, we'd notice data like this:

–

{"_col0":"S8.66045288732867","_col1":103} {"_col0":"S8.66322678828148","_col1":95}

–

This will then result in this table not being readable.

This is behaviour similar to the one fixed in HCATALOG-275, but we've obviously not fixed all the possibilities of that problem.

Issue Links

• incorporates: HIVE-3221 HiveConf.getPositionFromInternalName does not support more than sinle digit column numbers
• is related to: HCATALOG-275 JSON SerDe issues
• is related to: HCATALOG-492 Document CTAS workaround for Hive with JSON serde