Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-18552

Split hive.strict.checks.large.query into two configs

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: Hive
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      This change splits {{hive.strict.checks.large.query}} into two new configs: {{hive.strict.checks.no.partition.filter}} and {{hive.strict.checks.orderby.no.limit}}

      Previously, {{hive.strict.checks.large.query}} controlled whether or not (1) order bys were allowed without limits, and (2) scans on partitioned tables were allowed without filters.

      After this change, {{hive.strict.checks.no.partition.filter}} controls the partitioned table check and {{hive.strict.checks.orderby.no.limit}} controls the order by check.
      Show
      This change splits {{hive.strict.checks.large.query}} into two new configs: {{hive.strict.checks.no.partition.filter}} and {{hive.strict.checks.orderby.no.limit}} Previously, {{hive.strict.checks.large.query}} controlled whether or not (1) order bys were allowed without limits, and (2) scans on partitioned tables were allowed without filters. After this change, {{hive.strict.checks.no.partition.filter}} controls the partitioned table check and {{hive.strict.checks.orderby.no.limit}} controls the order by check.

      Description

      hive.strict.checks.large.query controls the strict checks for restricting order bys with no limits, and scans of a partitioned table without a filter on the partition table.

      While both checks prevent "large" queries from being run, they both control very different behavior. It would be better if users could control these restrictions separately.

      Furthermore, many users make the mistake of abusing partitioned tables and often end up in a situation where they are running queries that are doing full-table scans of partitioned tables. This can lead to lots of issues for Hive - e.g. OOM issues because so many partitions are loaded in memory. So it would be good if we enabled this restriction by default.

        Attachments

        1. HIVE-18552.1.patch
          11 kB
          Sahil Takiar
        2. HIVE-18552.2.patch
          23 kB
          Sahil Takiar
        3. HIVE-18552.3.patch
          23 kB
          Sahil Takiar
        4. HIVE-18552.4.patch
          25 kB
          Sahil Takiar

          Issue Links

            Activity

              People

              • Assignee:
                stakiar Sahil Takiar
                Reporter:
                stakiar Sahil Takiar
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: