[HIVE-7855] Invalid partition values specified when writing to static partitioned tables via HCatalog

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: HCatalog
• Assignee: David Chen
• Reporter: David Chen
• Created: 22/Aug/14 20:14
• Updated: 22/Aug/14 20:15

Description

While working on my fix for HIVE-4329, I found that on trunk, writing to static partitioned tables via HCatalog causes the following exception to be thrown:

org.apache.hive.hcatalog.common.HCatException : 2010 : Invalid partition values specified : Unable to configure dynamic partitioning for storage handler, mismatch between number of partition values obtained[0] and number of partition values required[2]
	at org.apache.hive.hcatalog.mapreduce.HCatBaseOutputFormat.configureOutputStorageHandler(HCatBaseOutputFormat.java:143)
	at org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.configureDynamicStorageHandler(DynamicPartitionFileRecordWriterContainer.java:209)
	at org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:132)
	at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:117)
	at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.hive.hcatalog.mapreduce.HCatMapReduceTest$MapCreate.map(HCatMapReduceTest.java:249)
	at org.apache.hive.hcatalog.mapreduce.HCatMapReduceTest$MapCreate.map(HCatMapReduceTest.java:240)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

While the tests for static partitioned tables (TestHCatPartitioned, TestHCatExternalPartitioned, TestHCatMutablePartitioned) do not fail, this exception is thrown (and then caught and ignored) during the write path.

This exception is thrown because the partition values list is empty while the partition keys list is not. It seems to me that partition columns for static partitioned tables are currently not handled correctly in HCatalog and the implementation only works incidentally.

However, this bug actually manifests in failures with my fix for HIVE-4329. In FileRecordWriterContainer.write, the partition columns are removed from the record being written. However, since the partition values list is empty, jobinfo.getPosOfPartCols ends up returning an empty list, resulting in the SerDe throwing an exception when it finds that the number of columns in the record does not match those of the schema.

Issue Links

• is related to: HIVE-4329 HCatalog should use getHiveRecordWriter rather than getRecordWriter