[HIVE-9303] Parquet files are written with incorrect definition levels

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.1
• Fix Version/s: 1.2.0
• Assignee: Sergio Peña
• Reporter: Skye Wanderman-Milne
• Created: 08/Jan/15 01:36
• Updated: 18/May/15 19:49
• Resolved: 03/Feb/15 01:56

Description

The definition level, which determines which level of nesting is NULL, appears to always be n or n-1, where n is the maximum definition level. This means that only the innermost level of nesting can be NULL. This is only relevant for Parquet files. For example:

CREATE TABLE text_tbl (a STRUCT<b:STRUCT<c:INT>>)
STORED AS TEXTFILE;

INSERT OVERWRITE TABLE text_tbl
SELECT IF(false, named_struct("b", named_struct("c", 1)), NULL)
FROM tbl LIMIT 1;

CREATE TABLE parq_tbl
STORED AS PARQUET
AS SELECT * FROM text_tbl;

SELECT * FROM text_tbl;
=> NULL # right

SELECT * FROM parq_tbl;
=> {"b":{"c":null}} # wrong