Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7685

Parquet memory manager

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Similar to HIVE-4248, Parquet tries to write large very large "row groups". This causes Hive to run out of memory during dynamic partitions when a reducer may have many Parquet files open at a given time.

      As such, we should implement a memory manager which ensures that we don't run out of memory due to writing too many row groups within a single JVM.

      Attachments

        1. HIVE-7685.1.patch
          1 kB
          Dong Chen
        2. HIVE-7685.patch
          2 kB
          Dong Chen
        3. HIVE-7685.1.patch.ready
          3 kB
          Dong Chen
        4. HIVE-7685.patch.ready
          3 kB
          Dong Chen

        Issue Links

        blocks

        Improvement - An improvement or enhancement to an existing feature or task. HIVE-8120 Umbrella JIRA tracking Parquet improvements

        • Major - Major loss of function.
        • Open
        is related to

        Task - A task that needs to be done. HIVE-11598 Document Configuration for Parquet Files

        • Major - Major loss of function.
        • Open
        relates to

        Bug - A problem which impairs or prevents the functions of the product. PARQUET-164 Warn when parquet memory manager kicks in

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HIVE-10149 Shuffle Hive data before storing in Parquet

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. PARQUET-108 Parquet Memory Management in Java

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. PARQUET-177 MemoryManager ensure minimum Column Chunk size

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        links to

        Web Link Parquet Documentation

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            dongc Dong Chen Assign to me
            brocknoland Brock Noland
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment