[HIVE-15662] check startTime in SparkTask to make sure startTime is not less than submitTime

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Assignee: Zhihai Xu
• Reporter: Zhihai Xu
• Created: 18/Jan/17 23:35
• Updated: 21/Jul/17 18:36
• Resolved: 24/Jan/17 18:04

Description

Check startTime in SparkTask to make sure startTime is not less than submitTime. We saw a corner case when the sparkTask is finished in less than 1 second, the startTime may not be set because RemoteSparkJobMonitor will sleep for 1 second then check the state, in this case, right after sleep for one second, the spark job is already completed. one example query with 3 spark tasks, the second one finished quickly for around 1 second:

SparkTask1:
"finishTime":1484638391978
"submitTime":1484638385933
"startTime": 1484638386973
SparkTask2:
"finishTime":1484638393019
"submitTime":1484638391979
"startTime": 1484638386973
SparkTask3:
"finishTime":1484638432123
"submitTime":1484638393020
"startTime": 1484638394057