[PIG-4714] Improve logging across multiple components with callerId

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0, 0.17.0
• Component/s: impl
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 27/Oct/15 04:57
• Updated: 08/Jun/16 20:48
• Resolved: 24/May/16 18:19

Description

The idea is to add callerId to every component, so we can track the chain of application which cause the underlining operation. A typical chain is Oozie->Pig->Tez->Hdfs. With proper callerId logging, we can trace Hdfs operation back to Oozie workflow which triggers it.

The protocol we decided is every component log its immediate callerId.

For Pig, this includes passing Pig script ID to underlining components as callerId using component specific API, log callerId of Pig and store it on ATS. More specific, it includes:
1. Generate a CallerId for each Pig script, pass it to Hdfs/Yarn/MapReduce/Tez which Pig invokes
2. Pig caller passes "pig.log.trace.id" to Pig, Pig will publish it to ATS

Issue Links

• is blocked by: TEZ-2851 Support a way for upstream applications to pass in a caller context to Tez
• is blocked by: HDFS-9184 Logging HDFS operation's caller context into audit logs