[HIVE-6521] WebHCat cannot fetch correct percentComplete for Hive jobs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.14.0
• Component/s: WebHCat
• Assignee: Deepesh Khandelwal
• Reporter: Deepesh Khandelwal
• Created: 28/Feb/14 07:56
• Updated: 13/Nov/14 19:40
• Resolved: 26/Mar/14 02:19

Description

WebHCat E2E test TestHive_7 failed because percentComplete wasn't returned as expected.

check_job_percent_complete failed. got percentComplete "map 0% reduce 0%",  expected  "map 100% reduce 100%"

So, there are two problems here.

1. The log parsing is broken for status of percentComplete. In the stderr of the job we see:

   Launching Job 1 out of 1
   Number of reduce tasks is set to 0 since there's no reduce operator
   Starting Job = job_1393486488858_0691, Tracking URL = http://ambari-sec-1393480847-others-2-4.cs1cloud.internal:8088/proxy/application_1393486488858_0691/
   Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1393486488858_0691
   Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
   2014-02-27 18:40:50,166 Stage-1 map = 0%,  reduce = 0%
   2014-02-27 18:40:56,599 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.87 sec
   2014-02-27 18:40:57,656 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.87 sec
   2014-02-27 18:40:58,706 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.87 sec
   MapReduce Total cumulative CPU time: 870 msec
   Ended Job = job_1393486488858_0691
   MapReduce Jobs Launched: 
   Job 0: Map: 1   Cumulative CPU: 0.87 sec   HDFS Read: 305 HDFS Write: 0 SUCCESS
   Total MapReduce CPU Time Spent: 870 msec
   

   The assumption in the code is that the line containing the percent status will end after "reduce = \d+%" but that fails with the above.

2. The last status from Hive job is "map = 100%, reduce = 0%" instead of expected "map = 100%, reduce = 100%".