Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-18023

Log multi-* requests for more than threshold number of rows

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 1.4.0, 1.3.2, 2.0.0
    • regionserver
    • None
    • Reviewed
    • Hide
      HBASE-18023 introduces a warning message in the RegionServer log when an RPC is received from a client that has more than 5000 "actions" (where an "action" is a collection of mutations for a specific row) in a single RPC. Misbehaving clients who send large RPCs to RegionServers can be malicious, causing temporary pauses via garbage collection or denial of service via crashes. The threshold of 5000 actions per RPC is defined by the property "hbase.rpc.rows.warning.threshold" in hbase-site.xml.
      Show
      HBASE-18023 introduces a warning message in the RegionServer log when an RPC is received from a client that has more than 5000 "actions" (where an "action" is a collection of mutations for a specific row) in a single RPC. Misbehaving clients who send large RPCs to RegionServers can be malicious, causing temporary pauses via garbage collection or denial of service via crashes. The threshold of 5000 actions per RPC is defined by the property "hbase.rpc.rows.warning.threshold" in hbase-site.xml.

    Description

      Today, if a user happens to do something like a large multi-put, they can get through request throttling (e.g. it is one request) but still crash a region server with a garbage storm. We have seen regionservers hit this issue and it is silent and deadly. The RS will report nothing more than a mysterious garbage collection and exit out.

      Ideally, we could report a large multi-* request before starting it, in case it happens to be deadly. Knowing the client, user and how many rows are affected would be a good start to tracking down painful users.

      Attachments

        1. HBASE-18023.addendum.patch
          2 kB
          Josh Elser
        2. HBASE-18023.master.001.patch
          9 kB
          David Harju
        3. HBASE-18023.master.002.patch
          9 kB
          David Harju
        4. HBASE-18023.master.003.patch
          13 kB
          David Harju
        5. HBASE-18023.master.004.patch
          11 kB
          David Harju
        6. HBASE-18023-branch-1.3.patch
          11 kB
          Andrew Kyle Purtell
        7. HBASE-18023-branch-1.patch
          11 kB
          Andrew Kyle Purtell

        Activity

          People

            dharju David Harju
            clayb Clay B.
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: