[PARQUET-647] Null Pointer Exception in Hive upon reading Parquet

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Not A Problem
• Affects Version/s: 1.6.0
• Component/s: parquet-format, parquet-mr
• Labels: hadoop hive nullpointerexception parquet spark

Environment

Hadoop 2.6
Hive 0.14
Parquet 1.6
SPARK 1.6.1
Scala 2.11

• Assignee: Unassigned
• Reporter: Mahadevan Sudarsanan
• Created: 28/Jun/16 23:53
• Updated: 23/Jun/24 03:28
• Resolved: 08/Sep/16 16:36

Description

When I write Parquet files from Spark Job, and try to read it in Hive as an External Table , I get Null Pointer Exception. After further analysis , I found I had some Null values in my transformation(used Dataset and DataFrame API's) before saving to parquet. These 2 fields which contains NULL are float data types. When I removed these two columns from the parquet datasets, I was able to read it in hive. Contrastingly , with all NULL columns I was able to read it Hive when I write my job to ORC format.
When a datatype is anything other than String , which is completely empty(NULL) written in parquet is not been able to read by Hive and throws NP Exception.