[HBASE-11052] Sending random data crashes thrift service

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.98.1, 1.0.0, 0.94.18
• Fix Version/s: 0.99.0, 0.94.21, 0.98.4
• Component/s: Thrift
• Hadoop Flags: Reviewed

Release Note

Thrift servers should use framed/compact protocol to protect against buffer overflow (default disabled as they are breaking old clients)
- hbase.regionserver.thrift.framed = true
- hbase.regionserver.thrift.compact = true

• Assignee: Adrian Muraru
• Reporter: Adrian Muraru
• Created: 23/Apr/14 08:18
• Updated: 08/Nov/14 12:18
• Resolved: 18/Jun/14 16:09

Description

Upstream thrift library has a know issue (THRIFT-601) causing the thrift server to crash with an Out-of-Memory Error when bogus requests are sent.

This reproduces when a very large request size is sent in the request header, making the thrift server to allocate a large memory segment leading to OOM.

LoadBalancer health checks are the first "candidate" for bogus requests
Thrift developers admit this is a known issue with TBinaryProtocol and their recommandation is to use TCompactProtocol/TFramedTransport but this requires all thrift clients to be updated (might not be feasible atm)

So we need a fix similar to CASSANDRA-475.

Issue Links

• is duplicated by: HBASE-11420 ThriftServer (version 1) may crash on OOME