[HIVE-6052] metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0, 0.13.0
• Fix Version/s: 0.13.0

Release Note

JDO pushdown for integers in metastore didn't work correctly for some legacy data in partition columns in Hive 0.12. In 0.13, hive.metastore.integral.jdo.pushdown setting is added to enable it, and it's disabled by default. Enabling it improves metastore perf for integral columns, especially if there's a large number of partitions. However, it doesn't work correctly with integral values that are not normalized (e.g. have leading zeroes, like 0012). If metastore direct SQL is enabled and works, this optimization is also irrelevant.

• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 18/Dec/13 01:28
• Updated: 18/May/14 22:53
• Resolved: 19/Dec/13 22:49

Description

If integer partition columns have values stores in non-canonical form, for example with leading zeroes, the integer filter doesn't work. That is because JDO pushdown uses substrings to compare for equality, and SQL pushdown is intentionally crippled to do the same to produce same results.
Probably, since both SQL pushdown and integers pushdown are just perf optimizations, we can remove it for JDO (or make configurable and disable by default), and uncripple SQL.

Issue Links

• is related to: HIVE-6070 document HIVE-6052