Hive
  1. Hive
  2. HIVE-2847

Add support for query rewrite from the metadata

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      A simple query like:

      select count(1) from T

      should be able to get the results from the metadata instead of scanning the complete input table T.

      2. the metadata only queries today does not scale to a very large number of partitions - that needs to be fixed.

        Activity

        Namit Jain created issue -
        Hide
        Namit Jain added a comment -

        Few parameters,like hive.stats.reliable, have been introduced to figure out if stats are accurate.
        The idea behind this jira is to use the stats from the metadata to rewrite some queries.

        For eg: for queries like

        select count(1) from T where ds > 10;

        Say, partitions T@ds=11 and T@ds=12 satisfy the query.

        If number of rows are present in the metadata for T@ds=11, T@ds=12, there is no need to scan the data to get the results.

        Show
        Namit Jain added a comment - Few parameters,like hive.stats.reliable, have been introduced to figure out if stats are accurate. The idea behind this jira is to use the stats from the metadata to rewrite some queries. For eg: for queries like select count(1) from T where ds > 10; Say, partitions T@ds=11 and T@ds=12 satisfy the query. If number of rows are present in the metadata for T@ds=11, T@ds=12, there is no need to scan the data to get the results.
        Hide
        Namit Jain added a comment -

        This will require a number of issues to be fixed:

        1. Metadata needs to be maintained at the partition level whether the stats for that partition are accurate or not.
        2. As more column level statistics are being added, the existing infrastructure should be used to support queries like:

        select max from T

        where column level statistics are stored for 'x'

        This jira can either be extended to support column level stats, or can propose a format to store column level stats, and then use them for
        query rewrite.

        Show
        Namit Jain added a comment - This will require a number of issues to be fixed: 1. Metadata needs to be maintained at the partition level whether the stats for that partition are accurate or not. 2. As more column level statistics are being added, the existing infrastructure should be used to support queries like: select max from T where column level statistics are stored for 'x' This jira can either be extended to support column level stats, or can propose a format to store column level stats, and then use them for query rewrite.

          People

          • Assignee:
            Unassigned
            Reporter:
            Namit Jain
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development