[TRAFODION-2310] DTM Lead Logic on very busy system resulted in trafodion crash

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1-incubating
• Fix Version/s: 2.1-incubating
• Component/s: foundation
• Flags: Important
• Assignee: Zalo Correa
• Reporter: Zalo Correa
• Due: 28/Oct/16
• Created: 25/Oct/16 20:04
• Updated: 23/Jun/17 00:25
• Resolved: 14/Feb/17 17:18

Description

The root cause of this problem is that the monitor in node 0 was starved out of CPU cycles and the watchdog timer expired. Consequently, the node was brought down by the SQWatchdog process.

This caused a sequence events where the TM leader was still in node 0 as far as all the remote monitors were concerned, but the $TM0 process no longer existed. The TM processes on the other nodes got the death message before the node was marked down which caused them to send a TM Leader request to their local monitor which checks to make sure the process exists, if not it aborts (this really should re-drive the selection of a new TM leader). However, the node down processing is what currently select a new TM leader, but there should be logic that reassigns a new TM leader when the leader dies. This is a bug that needs fixing.

Issue Links

• links to: GitHub Pull Request #782