[HBASE-1849] HTable doesn't work well at the core of a multi-threaded server; e.g. webserver

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Incomplete
• Component/s: Performance
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 17/Sep/09 05:44
• Updated: 11/Jun/22 21:03
• Resolved: 19/Jul/14 00:46

Description

HTable must do the following:

+ Sit in a shell or simple client – e.g. Map or Reduce task – and feed and read from HBase single-threadedly. It does this job OK.
+ Sit at core of a multithreaded server (100s of threads) – a webserver or thrift gateway – and keep the throughput high. Its currently not good at this job.

In the way of our achieving the second in the list above are the following:

+ HTable must seekout and cache region locations. It keeps cache down in HConnectionManager. One is shared by all HTable instances if the HTable instance was made with same HBaseConfiguration instance. Lookups of regions is inside a synchronize block; if the region wanted is in the cache, the lock is held a short time. Otherwise, must wait till trip to server completed (may require retries). Meantime all other work is blocked even if we're using HTablePool.
+ Regardless of the identity of the HBaseConfiguration, Hadoop RPC has ONE Connection open to a server at a time; request and response are multiplexed over this single connection.

Broken stuff:

+ Puts are synchronized to protect the write buffer so only one thread at a time appends but flushcommit is open for any thread to call it. Once the write buffer is full, all Puts block until its freed again. This looks like hang if hundreds of threads and each write is to a random region in a big table and each write has to have its region looked-up (There may be some other brokenness in here because this bottleneck seems to last longer than it should even if hundreds of threads).

Ideas:

+ Query of the cache does not block all access to the cache. We only block access if wanted region is being looked up so other reads and writes to regions we know the location of can go ahead.
+ nio'd client and server

Issue Links

• is related to: HBASE-1845 MultiGet, MultiDelete, and MultiPut - batched to the appropriate region servers
• is related to: HBASE-2939 Allow Client-Side Connection Pooling
• is related to: HBASE-2182 rpc/ipc refactor; x-version compatibility, nio, async, enveloping, sane timeouts, etc.