Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14535 add insert-only ACID tables to Hive
  3. HIVE-15368

consider optimizing Utilities::handleMmTableFinalPath

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.3.0
    • None
    • None
    • None

    Description

      Branch: hive-14535

      https://github.com/apache/hive/blob/hive-14535/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L4049

      When running "insert overwrite...on partitioned table" with 2000+ partitions, good amount of time (~245 seconds) was spent in iterating every mmDirectory entry and checking its file listings in S3. Creating this jira to consider optimizing this codepath, as information from getMmDirectoryCandidates could be used in terms of reducing the number of times S3 needs to be contacted.

      Attachments

        1. HIVE-15368.branch.14535.1.patch
          11 kB
          Rajesh Balamohan

        Activity

          People

            Unassigned Unassigned
            rajesh.balamohan Rajesh Balamohan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: