Hive
  1. Hive
  2. HIVE-6052

metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0, 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      JDO pushdown for integers in metastore didn't work correctly for some legacy data in partition columns in Hive 0.12. In 0.13, hive.metastore.integral.jdo.pushdown setting is added to enable it, and it's disabled by default. Enabling it improves metastore perf for integral columns, especially if there's a large number of partitions. However, it doesn't work correctly with integral values that are not normalized (e.g. have leading zeroes, like 0012). If metastore direct SQL is enabled and works, this optimization is also irrelevant.
      Show
      JDO pushdown for integers in metastore didn't work correctly for some legacy data in partition columns in Hive 0.12. In 0.13, hive.metastore.integral.jdo.pushdown setting is added to enable it, and it's disabled by default. Enabling it improves metastore perf for integral columns, especially if there's a large number of partitions. However, it doesn't work correctly with integral values that are not normalized (e.g. have leading zeroes, like 0012). If metastore direct SQL is enabled and works, this optimization is also irrelevant.

      Description

      If integer partition columns have values stores in non-canonical form, for example with leading zeroes, the integer filter doesn't work. That is because JDO pushdown uses substrings to compare for equality, and SQL pushdown is intentionally crippled to do the same to produce same results.
      Probably, since both SQL pushdown and integers pushdown are just perf optimizations, we can remove it for JDO (or make configurable and disable by default), and uncripple SQL.

      1. HIVE-6052.01.patch
        68 kB
        Sergey Shelukhin
      2. HIVE-6052.02.patch
        55 kB
        Sergey Shelukhin
      3. HIVE-6052.patch
        9 kB
        Sergey Shelukhin

        Issue Links

          Activity

            People

            • Assignee:
              Sergey Shelukhin
              Reporter:
              Sergey Shelukhin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development