Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13884

Disallow queries in HMS fetching more than a configured number of partitions

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.3.0
    • None
    • None

    Description

      Currently the PartitionPruner requests either all partitions or partitions based on filter expression. In either scenarios, if the number of partitions accessed is large there can be significant memory pressure at the HMS server end.

      We already have a config hive.limit.query.max.table.partition that enforces limits on number of partitions that may be scanned per operator. But this check happens after the PartitionPruner has already fetched all partitions.

      We should add an option at PartitionPruner level to disallow queries that attempt to access number of partitions beyond a configurable limit.

      Note that hive.mapred.mode=strict disallow queries without a partition filter in PartitionPruner, but this check accepts any query with a pruning condition, even if partitions fetched are large. In multi-tenant environments, admins could use more control w.r.t. number of partitions allowed based on HMS memory capacity.

      One option is to have PartitionPruner first fetch the partition names (instead of partition specs) and throw an exception if number of partitions exceeds the configured value. Otherwise, fetch the partition specs.

      Looks like the existing listPartitionNames call could be used if extended to take partition filter expressions like getPartitionsByExpr call does.

      Attachments

        1. HIVE-13884.9.patch
          21 kB
          Sergio Peña
        2. HIVE-13884.8.patch
          18 kB
          Sergio Peña
        3. HIVE-13884.7.patch
          18 kB
          Sergio Peña
        4. HIVE-13884.6.patch
          17 kB
          Sergio Peña
        5. HIVE-13884.5.patch
          17 kB
          Sergio Peña
        6. HIVE-13884.4.patch
          16 kB
          Sergio Peña
        7. HIVE-13884.3.patch
          15 kB
          Sergio Peña
        8. HIVE-13884.2.patch
          15 kB
          Sergio Peña
        9. HIVE-13884.10.patch
          22 kB
          Sergio Peña
        10. HIVE-13884.1.patch
          16 kB
          Sergio Peña

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            spena Sergio Peña Assign to me
            mohitsabharwal Mohit Sabharwal
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment