Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13646

make hive.optimize.sort.dynamic.partition compatible with ACID tables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 1.0.0
    • 1.3.0, 2.1.0
    • Transactions
    • None

    Description

      HIVE-8875 disabled hive.optimize.sort.dynamic.partition for ACID queries.
      dynamic inserts are common in ACID and this leaves users with few options if they are seeing OutOfMemory errors due to too many writers.

      hive.optimize.sort.dynamic.partition sorts data by partition col/bucket col/sort col to ensure each reducer only needs 1 writer.

      Acid requires data in each bucket file to be sorted by ROW__ID and thus doesn't allow end user to determine sorting.

      So we should be able to support hive.optimize.sort.dynamic.partition with
      sort on partition col/bucket col/ROW__ID

      Attachments

        1. HIVE-13646.2.patch
          12 kB
          Eugene Koifman
        2. HIVE-13646.patch
          0.8 kB
          Eugene Koifman

        Issue Links

          Activity

            People

              ekoifman Eugene Koifman
              ekoifman Eugene Koifman
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 168h
                  168h
                  Remaining:
                  Remaining Estimate - 168h
                  168h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified