[HIVE-9658] Reduce parquet memory usage by bypassing java primitive objects on ETypeConverter - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Bulk Copy Attachments

Bulk Move Attachments

Add vote

Voters

Watch issue

Watchers

Convert to Issue

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Patch Available
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
None

Description

The ETypeConverter class passes Writable objects to the collection converters in order to be read later by the map/reduce functions. These objects are all wrapped in a unique ArrayWritable object.

We can save some memory by returning the java primitive objects instead in order to prevent memory allocation. The only writable object needed by map/reduce is ArrayWritable. If we create another writable class where to store primitive objects (Object), then we can stop using all primitive wirtables.