[HIVE-9605] Remove parquet nested objects from wrapper writable objects

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.3.0, 2.0.0
• Assignee: Sergio Peña
• Reporter: Sergio Peña
• Created: 06/Feb/15 23:29
• Updated: 16/Feb/16 23:50
• Resolved: 27/May/15 19:29

Description

Parquet nested types are using an extra wrapper object (ArrayWritable) as a wrapper of map and list elements. This extra object is not needed and causing unnecessary memory allocations.

An example of code is on HiveCollectionConverter.java:

public void end() {
    parent.set(index, wrapList(new ArrayWritable(
        Writable.class, list.toArray(new Writable[list.size()]))));
}

This object is later unwrapped on AbstractParquetMapInspector, i.e.:

final Writable[] mapContainer = ((ArrayWritable) data).get();
final Writable[] mapArray = ((ArrayWritable) mapContainer[0]).get();
for (final Writable obj : mapArray) {
  ...
}

We should get rid of this wrapper object to save time and memory.

Issue Links

• links to: Review Board