Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15250

Reuse partitions info generated in MoveTask to its subscribers (StatsTask)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.2.0
    • 2.2.0
    • Metastore
    • None

    Description

      When dynamic partitions are enabled, StatsTask loads partition information by querying metastore. In cases like insert overwrite table, this can be expensive operation depending on the number of partitions involved (for e.g, in tpcds populating web_returns table would incur 2184 DB calls just on this function).

      It would be good to pass on the partition information generated in MoveTask to its subscribers to reduce the number of DB calls.

      Attachments

        1. HIVE-15250.1.patch
          3 kB
          Rajesh Balamohan
        2. HIVE-15250.2.patch
          3 kB
          Rajesh Balamohan
        3. HIVE-15250.3.patch
          3 kB
          Rajesh Balamohan

        Issue Links

          causes

          Sub-task - The sub-task of the issue HIVE-25918 Invalid stats after multi inserting into the same partition

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-13925 ETL optimizations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: