[HIVE-5547] webhcat pig job submission should ship hive tar if -usehcatalog is specified

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.13.0
• Component/s: WebHCat
• Assignee: Eugene Koifman Assign to me
• Reporter: Eugene Koifman
• Created: 15/Oct/13 16:50
• Updated: 30/May/14 19:20
• Resolved: 01/Nov/13 22:24

Description

Currently when when a Pig job is submitted through WebHCat and the Pig script uses HCatalog, that means that Hive should be installed on the node in the cluster which ends up executing the job. For large clusters is this a manageability issue so we should use DistributedCache to ship the Hive tar file to the target node as part of job submission

TestPig_11 in hcatalog/src/test/e2e/templeton/tests/jobsubmission.conf has the test case for this

Issue Links

• depends upon: HIVE-5627 Document 'usehcatalog' parameter on WebHCat calls
• depends upon: PIG-3547 Pig11 doesn't work with WebHCat 0.13 in some cases
• depends upon: PIG-3543 Pig 11 doesn't work with Hive/HCat 13
• is blocked by: HIVE-5511 percentComplete returned by job status from WebHCat is null
• is related to: HIVE-5188 MR job launched through WebHCat fails to find additional jars in classpath
• is related to: HIVE-5133 webhcat jobs that need to access metastore fails in secure mode
• relates to: AMBARI-5976 User should be able choose in UI whether hive and pig is shipped by webhcat