[HIVE-21987] Hive is unable to read Parquet int32 annotated with decimal - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 4.0.0-alpha-1
Component/s: None
Labels:
None

Description

When I tried to read a Parquet file from a Hive (with Tez execution engine) table with a small decimal column, I got the following exception:

Caused by: java.lang.UnsupportedOperationException: org.apache.hadoop.hive.ql.io.parquet.convert.ETypeConverter$8$1
	at org.apache.parquet.io.api.PrimitiveConverter.addInt(PrimitiveConverter.java:98)
	at org.apache.parquet.column.impl.ColumnReaderImpl$2$3.writeValue(ColumnReaderImpl.java:248)
	at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
	at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
	... 28 more

Steps to reproduce:

Create a Hive table with a single decimal(4, 2) column
Create a Parquet file with int32 column annotated with decimal(4, 2) logical type, put it into the previously created table location (or use the attached parquet file, in this case the column should be named as 'd', to match the Hive schema with the Parquet schema in the file)
Execute a select * on this table

Also, I'm afraid that similar problems can happen with int64 decimals too. Parquet specification allows both of these cases.