[TEZ-4527] Add generic and pluggable hooks for DAGs and task attempts

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.5
• Assignee: Shohei Okumiya
• Reporter: Shohei Okumiya
• Created: 18/Dec/23 12:29
• Updated: 22/Dec/24 12:49
• Resolved: 22/Dec/24 11:10

Description

TEZ-4344 added a capability to periodically dump thread information to Tez. I think the basic architecture, injecting an additional process into DAGAppMaster or TezChild, is potentially extensible.

 

For example, we maintain a very similar patch to collect stack traces.

• https://speakerdeck.com/okumin/hive-distributed-profiling-system-in-treasure-data-english-version-number-tdtechtalk?slide=26
• https://api-docs.treasuredata.com/blog/hive-distributed-profiling/

 

I would propose to make the injection step pluggable and maintain TezThreadDumpHelper as one of the plugins.

Issue Links

• relates to: TEZ-4344 Collect jstack periodically from all containers
• links to: GitHub Pull Request #324