Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14803

S3: Stats gathering for insert queries can be expensive for partitioned dataset

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.1.0
    • 2.3.0
    • Metastore
    • None

    Description

      StatsTask's aggregateStats populates stats details for all partitions by checking the file sizes which turns out to be expensive when larger number of partitions are inserted.

      Attachments

        1. HIVE-14803.7.patch
          5 kB
          Rajesh Balamohan
        2. HIVE-14803.6.patch
          5 kB
          Rajesh Balamohan
        3. HIVE-14803.5.patch
          10 kB
          Rajesh Balamohan
        4. HIVE-14803.4.patch
          10 kB
          Rajesh Balamohan
        5. HIVE-14803.3.patch
          10 kB
          Rajesh Balamohan
        6. HIVE-14803.2.patch
          10 kB
          Rajesh Balamohan
        7. HIVE-14803.1.patch
          9 kB
          Rajesh Balamohan

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HIVE-13925 ETL optimizations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              rajesh.balamohan Rajesh Balamohan
              rajesh.balamohan Rajesh Balamohan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: