[MAPREDUCE-3053] YARN Protobuf RPC Failures in RM

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.0
• Component/s: mrv2, resourcemanager

Environment

URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1173401
Node Kind: directory
Schedule: normal
Last Changed Author: vinodkv
Last Changed Rev: 1173130
Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)

• Assignee: Vinod Kumar Vavilapalli
• Reporter: Chris Riccomini
• Created: 20/Sep/11 22:32
• Updated: 15/Nov/11 00:49
• Resolved: 25/Sep/11 10:43

Description

When I try to register my ApplicationMaster with YARN's RM, it fails.

In my ApplicationMaster's logs:

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)

at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
at $Proxy6.registerApplicationMaster(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
... 3 more
Caused by: java.lang.NullPointerException: java.lang.NullPointerException
at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)

at org.apache.hadoop.ipc.Client.call(Client.java:1084)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
... 5 more

In the ResourceManager's logs:

2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
java.lang.NullPointerException
at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)

My registration code:

val appId = args(0).toInt
val attemptId = args(1).toInt
val timestamp = args(2).toLong

// these are our application master's parameters
val streamerClass = args(3)
val tasks = args(4).toInt

// TODO log params here

// start the application master helper
val conf = new Configuration
val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
.registerWithResourceManager

.....

val rpc = YarnRPC.create(conf)
val appId = Records.newRecord(classOf[ApplicationId])
val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
var requestId = 0

appId.setClusterTimestamp(lTimestamp)
appId.setId(iAppId)
appAttemptId.setApplicationId(appId)
appAttemptId.setAttemptId(iAppAttemptId)

def registerWithResourceManager(): ApplicationMasterHelper =

{ val req = Records.newRecord(classOf[RegisterApplicationMasterRequest]) req.setApplicationAttemptId(appAttemptId) // TODO not sure why these are blank- This is how spark does it req.setHost("") req.setRpcPort(1) req.setTrackingUrl("") resourceManager.registerApplicationMaster(req) this }

My params are receiving the proper app/attempt/cluster timestamps:

app - 1
attempt - 1
timestamp - 1316556657998