[HAWQ-1326] Cancel the query earlier if one of the segments for the query crashes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0.0-incubating
• Assignee: Paul Guo
• Reporter: Paul Guo
• Created: 14/Feb/17 03:12
• Updated: 18/Feb/17 05:27
• Resolved: 14/Feb/17 09:14

Description

QD thread could hang in the loop of poll() since: 1) The alive segments could wait at the interconnect for the dead segment until interconnect timeout (by default 1 hour). 2) In the QD thread poll() will not sense the system-down until kernel tcp keepalive messaging is triggered, however the keepalive timeout is a bit long (2 hours by default on rhel6.x) and it could be configured via procfs only.

A proper solution would be using the RM heartbeat mechanism:

RM maintains a global ID lists (stable cross node adding or removing) for all nodes and keeps updating the health state via userspace heartbeat mechanism, thus we could maintain a bitmap in shared memory which keeps the latest node healthy info updated then we could use it in QD code, i.e. Cancel the query if finding the segment node, which handles part of the query, is down.

Issue Links

• links to: GitHub Pull Request #1118