[HBASE-10353] Alternative I/O Engine

• Type: Umbrella
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Assignee: Unassigned
• Reporter: Avik Dey
• Created: 15/Jan/14 23:05
• Updated: 17/Jun/22 04:59
• Resolved: 11/Jun/22 18:24

Description

Increasingly HBase is being asked to operate in server environments where the Java Virtual Machine is still working to take optimum advantage of the server resources, for example those backed by solid-state storage, or deployed on large memory systems. Many HBase users also struggle in environments with demanding and tight request latency SLAs. The Java sandbox does not yet have facilities for taking advantage of OS level features for high throughput low latency I/O, nor efficient access to interconnects for high performance computing. Our test results show that even using G1 GC, Java server processes that must operate within tight latency bounds must keep their heaps small relative to the kind of RAM we can pack into a server today.

We would like this issue to serve as an umbrella for brainstorming and proposals for an incremental evolution of HBase, through use of existing plug points and careful introduction of new ones, to a complete architecture for plugging in alternative I/O engines. As a consequence we will be able to explore improvements within and outside the JVM. Such an alternative engine would be expected to provide lower latency and higher throughput than what is possible within the constraints of today's implementation. Other JIRAs already cover moving allocations for the I/O path off heap. Once we have I/O code paths built on direct buffers that conveniently provides substrate for efficient boundary crossings from Java into native code, we can open subtasks for proposing and discussing where those boundaries can be drawn. We will also open subtasks for proposing the shape and scope of the first experimental alternative I/O engine.

This work should maintain API and wire compatibility as well as preserve data format consistency with today’s HBase, so HBase users can take advantage of this work without any need to change client code or massage existing data.

Issue Links

• is related to: HBASE-10191 Move large arena storage off heap