[AMBARI-10129] Pig Client needs Tez Client on same host, ambari web need to update HostRoles it sends

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: ambari-web
• Assignee: Andrii Tkach
• Reporter: Andrii Tkach
• Created: 19/Mar/15 13:24
• Updated: 19/Mar/15 19:04
• Resolved: 19/Mar/15 15:54

Description

Pig service check fails if the Tez client is not installed on the same host that contains the Pig client.

If during the Install Wizard, Pig and Tez are added at the same time, then this guarantees that whatever hosts are selected to contain the clients will have both services, which works.

However, if Tez is added first (plus it dependencies on MR2, Yarn) on say host 3, and then Pig service is added, and the client is only installed on host 1, then the Pig service check will fail.

I tried adding this to PIG's metainfo.xml

<dependencies>
  <dependency>
    <name>TEZ/TEZ_CLIENT</name>
    <scope>host</scope>
    <auto-deploy>
      <enabled>true</enabled>
    </auto-deploy>
  </dependency>
  <dependency>
    <name>MAPREDUCE2/MAPREDUCE2_CLIENT</name>
    <scope>host</scope>
    <auto-deploy>
      <enabled>true</enabled>
    </auto-deploy>
  </dependency>
</dependencies>

However, it was not sufficient for Tez Client to be installed on the same hosts.
When I installed Pig, I saw that ambari-web only sent the request as,

POST http://c6401.ambari.apache.org:8080/api/v1/clusters/c1/hosts
{"RequestInfo":{"query":"Hosts/host_name:
c6401.ambari.apache.org"},"Body":{"host_components":[{"HostRoles":{"component_name":"PIG"}}]}}

Notice that the HostRoles only contains PIG; it should also be adding TEZ_CLIENT and MAPREDUCE2_CLIENT, as per the dependencies shown above

This is the failure that Pig's service check generates,

15/02/11 20:17:28 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType

...
2015-02-11 20:17:35,966 [PigTezLauncher-0] ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG
org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration
	at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:162)
	at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:721)
	at org.apache.tez.client.TezClient.start(TezClient.java:298)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:95)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159)
	at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)