[TEZ-1004] AM relocalization doesn't handle conflicting resources correctly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0, 0.4.1
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 31/Mar/14 21:36
• Updated: 06/Sep/14 01:35
• Resolved: 04/Apr/14 00:09

Description

If you add a jar that tez already has, it throws despite having the same jar, probably due to ts. Might need to compare SHA-s.

org.apache.hadoop.ipc.RemoteException(org.apache.tez.dag.api.TezUncheckedException): Cannot add different additional resources with the same name : protobuf-java-2.5.0.jar, Existing: [resource { scheme: "hdfs" host: "hor18n31.gq1.ygridcore.net" port: 8020 file: "/apps/tez/lib/protobuf-java-2.5.0.jar" } size: 533455 timestamp: 1395764131176 type: FILE visibility: PUBLIC], New: [resource { scheme: "hdfs" port: -1 file: "/user/hrt_qa/.hiveJars/protobuf-java-2.5.0.jar" } size: 533455 timestamp: 1395961799302 type: FILE visibility: APPLICATION]
   at org.apache.tez.dag.app.DAGAppMaster.getAdditionalLocalResourceDiff(DAGAppMaster.java:1108)
   at org.apache.tez.dag.app.DAGAppMaster.startDAG(DAGAppMaster.java:1821)
   at org.apache.tez.dag.app.DAGAppMaster.submitDAGToAppMaster(DAGAppMaster.java:955)
   at org.apache.tez.dag.app.DAGAppMaster$DAGClientHandler.submitDAG(DAGAppMaster.java:1058)
   at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:131)
   at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:8292)
   at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2007)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2003)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2001)
   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
   at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)