[YARN-9327] Improve synchronisation in ProtoUtils#convertToProtoFormat block

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.2.1, 3.1.3
• Assignee: Bibin Chundatt
• Reporter: Bibin Chundatt
• Created: 22/Feb/19 12:51
• Updated: 26/Dec/19 04:54
• Resolved: 02/Jul/19 06:46

Description

  public static synchronized ResourceProto convertToProtoFormat(Resource r) {
    return ResourcePBImpl.getProto(r);
  }

"IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0 tid=0x00007f181de72800 nid=0x222 waiting for monitor entry [0x00007ef153dad000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
	- waiting to lock <0x00007ef2d8bcf6d8> (a java.lang.Class for org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
	at org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
	at org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
	at org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToProto(NodeReportPBImpl.java:289)
	at org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.getProto(NodeReportPBImpl.java:228)
	at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.convertToProtoFormat(AllocateResponsePBImpl.java:844)
	- locked <0x00007f0fed968a30> (a org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
	at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.access$500(AllocateResponsePBImpl.java:72)
	at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:810)
	- locked <0x00007f0fed96f500> (a org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1)
	at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:799)
	at com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
	at com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
	at org.apache.hadoop.yarn.proto.YarnServiceProtos$AllocateResponseProto$Builder.addAllUpdatedNodes(YarnServiceProtos.java:13810)
	at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToBuilder(AllocateResponsePBImpl.java:158)
	- locked <0x00007f0fed968a30> (a org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
	at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToProto(AllocateResponsePBImpl.java:198)
	- eliminated <0x00007f0fed968a30> (a org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
	at org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.getProto(AllocateResponsePBImpl.java:103)
	- locked <0x00007f0fed968a30> (a org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:61)
	at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684)

Seems synchronization is not required here.

Issue Links

• relates to: YARN-9871 Miscellaneous scalability improvement