[HIVE-20591] hive query hung during compilation if same previous query is unable to invalidate the QueryResultsCache entry

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: HiveServer2

Environment

Hive-3,java-8

• Assignee: Unassigned
• Reporter: Rajkumar Singh
• Created: 19/Sep/18 01:49
• Updated: 19/Sep/18 01:50

Description

I believe this is the sequence of event to reproduce this issue.
1. query failed with some env issue while setting up the Tez session.
2. hiveserver2 tries do query cleanup, it invokes queryresultscache cleanup.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java#L235
3. for some reason eighter following 2 event never happen and query falls into the endless loop of checking the valid status.
i: unable to set the invalid status and return the old status
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java#L260
ii: or this condition never reached.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java#L245

I don't have complete jstack so its tough to say who is waiting on what, the stuck thread stack snipped look like

 java.lang.Thread.State: TIMED_WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  at org.apache.hadoop.hive.ql.cache.results.QueryResultsCache$CacheEntry.waitForValidStatus(QueryResultsCache.java:325)
  - locked <0x00000000b32661c0> (a org.apache.hadoop.hive.ql.cache.results.QueryResultsCache$CacheEntry)
  at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.checkResultsCache(SemanticAnalyzer.java:14860)
  at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12200)

will add more details after reproducing the issue again.