[HBASE-18023] Log multi-* requests for more than threshold number of rows

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.4.0, 1.3.2, 2.0.0
• Component/s: regionserver
• Hadoop Flags: Reviewed

Release Note

HBASE-18023 introduces a warning message in the RegionServer log when an RPC is received from a client that has more than 5000 "actions" (where an "action" is a collection of mutations for a specific row) in a single RPC. Misbehaving clients who send large RPCs to RegionServers can be malicious, causing temporary pauses via garbage collection or denial of service via crashes. The threshold of 5000 actions per RPC is defined by the property "hbase.rpc.rows.warning.threshold" in hbase-site.xml.

• Assignee: David Harju
• Reporter: Clay B.
• Created: 10/May/17 21:25
• Updated: 06/Apr/18 04:58
• Resolved: 25/Jul/17 22:45

Description

Today, if a user happens to do something like a large multi-put, they can get through request throttling (e.g. it is one request) but still crash a region server with a garbage storm. We have seen regionservers hit this issue and it is silent and deadly. The RS will report nothing more than a mysterious garbage collection and exit out.

Ideally, we could report a large multi-* request before starting it, in case it happens to be deadly. Knowing the client, user and how many rows are affected would be a good start to tracking down painful users.

Sub-Tasks

• Replication sink frequently triggers HBASE-18023 warnings: Resolved