Details

      Description

      Similar to HIVE-4248, Parquet tries to write large very large "row groups". This causes Hive to run out of memory during dynamic partitions when a reducer may have many Parquet files open at a given time.

      As such, we should implement a memory manager which ensures that we don't run out of memory due to writing too many row groups within a single JVM.

        Attachments

        1. HIVE-7685.patch.ready
          3 kB
          Dong Chen
        2. HIVE-7685.patch
          2 kB
          Dong Chen
        3. HIVE-7685.1.patch.ready
          3 kB
          Dong Chen
        4. HIVE-7685.1.patch
          1 kB
          Dong Chen

          Issue Links

            Activity

              People

              • Assignee:
                dongc Dong Chen
                Reporter:
                brocknoland Brock Noland
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: