[KUDU-373] pstack_watcher will hang if output is large

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: M4.5
• Component/s: test, util
• Target Version/s: M4.5
• Code Review: http://gerrit.ent.cloudera.com:8080/3280
• Assignee: Mike Percy
• Reporter: Todd Lipcon
• Created: 25/Jun/14 17:50
• Updated: 26/Jun/14 02:57
• Resolved: 26/Jun/14 02:57

Description

I found a hung Jenkins build which is stuck in a very interesting scenario:

• a test had timed out and triggered the pstack watcher
• it had execed pstack, which had execed gdb and frozen the original process
• pstack is stuck in a "write()" call trying to write its output back to its stdout (a pipe which leads back to the unit test)
• the unit test process is frozen in utrace_stop (ie kill -STOP) because ptrace does that
• therefore, the unit test isn't calling read() on the pipe, and the kernel's pipe buffer is full. thus pstack's write() is hung, and it will never exit