[PARQUET-78] Provide a ByteBuffer based write path

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Parth Chandra
• Created: 22/Aug/14 23:18
• Updated: 23/Jun/24 03:26

Description

Similar to the effort to use a ByteBuffer in the read path, we should provide a path to use a ByteBuffer in the write path.

The proposed idea is to provide an allocator in the writer that is passed down to CapacityByteArrayOutputStream and is used to allocate a ByteBuffer instead of the byte array that it currently uses.

I've attached a patch based on the 1.5 release that does that. There are two implementations of the allocator - one for Heap memory and one for Direct memory. In addition, the allocator interface allows for a 'release' method so that reference counted memory allocators like Netty's ByteBuf can be used to provide an allocator.

Issue Links

• is depended upon by: DRILL-1410 Move off of Parquet fork