[HBASE-14443] Add request parameter to the TooSlow/TooLarge warn message of RpcServer

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Abandoned
• Affects Version/s: 1.2.1
• Component/s: IPC/RPC, Operability
• Assignee: Unassigned
• Reporter: Jianwei Cui
• Created: 16/Sep/15 12:34
• Updated: 01/Jul/22 21:09
• Resolved: 11/Jun/22 20:01

Description

The RpcServer will log a warn message for TooSlow or TooLarge request as:

        logResponse(new Object[]{param},
            md.getName(), md.getName() + "(" + param.getClass().getName() + ")",
            (tooLarge ? "TooLarge" : "TooSlow"),
            status.getClient(), startTime, processingTime, qTime,
            responseSize);

The RpcServer#logResponse will create the warn message as:

if (params.length == 2 && server instanceof HRegionServer &&
        params[0] instanceof byte[] &&
        params[1] instanceof Operation) {
      ...
      responseInfo.putAll(((Operation) params[1]).toMap());
      ...
    } else if (params.length == 1 && server instanceof HRegionServer &&
        params[0] instanceof Operation) {
      ...
      responseInfo.putAll(((Operation) params[0]).toMap());
      ...
    } else {
      ...
    }

Because the parameter is always a protobuf message, not an instance of Operation, the request parameter will not be added into the warn message. The parameter is helpful to find out the problem, for example, knowing the startRow/endRow is useful for a TooSlow scan. To improve the warn message, we can transform the protobuf request message to corresponding Operation subclass object by ProtobufUtil, so that it can be added the warn message. Suggestion and discussion are welcomed.

Issue Links

• supercedes: HBASE-14333 responseTooLarge/Slow logs are not actionable when the rpc method is ExecService