Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8120 Umbrella JIRA tracking Parquet improvements
  3. HIVE-9658

Reduce parquet memory usage by bypassing java primitive objects on ETypeConverter

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      The ETypeConverter class passes Writable objects to the collection converters in order to be read later by the map/reduce functions. These objects are all wrapped in a unique ArrayWritable object.

      We can save some memory by returning the java primitive objects instead in order to prevent memory allocation. The only writable object needed by map/reduce is ArrayWritable. If we create another writable class where to store primitive objects (Object), then we can stop using all primitive wirtables.

      Attachments

        1. HIVE-9658.1.patch
          74 kB
          Sergio Peña
        2. HIVE-9658.2.patch
          98 kB
          Sergio Peña
        3. HIVE-9658.3.patch
          109 kB
          Sergio Peña
        4. HIVE-9658.4.patch
          98 kB
          Sergio Peña
        5. HIVE-9658.5.patch
          100 kB
          Sergio Peña
        6. HIVE-9658.6.patch
          103 kB
          Sergio Peña
        7. HIVE-9658.7.patch
          115 kB
          Sergio Peña

        Issue Links

          Activity

            People

              spena Sergio Peña
              spena Sergio Peña
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: