[HIVE-14310] ORC schema evolution should not completely disable PPD

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.2.0
• Fix Version/s: 2.1.1, 2.2.0
• Target Version/s: 2.1.0, 2.2.0
• Assignee: Prasanth Jayachandran
• Reporter: Prasanth Jayachandran
• Created: 21/Jul/16 22:52
• Updated: 08/Dec/16 14:15
• Resolved: 28/Jul/16 19:07

Description

Follow up for HIVE-14214 which completely shuts off PPD when there is any schema evolution. Some evolutions are safer for PPD like

byte -> short -> int -> long
float -> double (This is unsafe, see comments in SchemaEvolution.java in the patch)
varchar <-> string (string to char, varchar to char and vice versa is also unsafe conversion as Orc stores internal index with padded spaces for char)

For all other conversions we can disable PPD for that specific column that has evolved by returning TruthValue.YES_NO

Issue Links

• blocks: HIVE-14355 Schema evolution for ORC in llap is broken for int to string conversion
• is blocked by: HIVE-14324 ORC PPD for floats is broken
• is blocked by: HIVE-14333 ORC schema evolution from float to double changes precision and breaks filters
• is related to: HIVE-14214 ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
• links to: RB