Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15250

Reuse partitions info generated in MoveTask to its subscribers (StatsTask)

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.0
    • Component/s: Metastore
    • Labels:
      None
    • Target Version/s:

      Description

      When dynamic partitions are enabled, StatsTask loads partition information by querying metastore. In cases like insert overwrite table, this can be expensive operation depending on the number of partitions involved (for e.g, in tpcds populating web_returns table would incur 2184 DB calls just on this function).

      It would be good to pass on the partition information generated in MoveTask to its subscribers to reduce the number of DB calls.

        Attachments

        1. HIVE-15250.3.patch
          3 kB
          Rajesh Balamohan
        2. HIVE-15250.2.patch
          3 kB
          Rajesh Balamohan
        3. HIVE-15250.1.patch
          3 kB
          Rajesh Balamohan

          Issue Links

            Activity

              People

              • Assignee:
                rajesh.balamohan Rajesh Balamohan
                Reporter:
                rajesh.balamohan Rajesh Balamohan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: