Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4714

Improve logging across multiple components with callerId

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.16.0, 0.17.0
    • impl
    • None
    • Reviewed

    Description

      The idea is to add callerId to every component, so we can track the chain of application which cause the underlining operation. A typical chain is Oozie->Pig->Tez->Hdfs. With proper callerId logging, we can trace Hdfs operation back to Oozie workflow which triggers it.

      The protocol we decided is every component log its immediate callerId.

      For Pig, this includes passing Pig script ID to underlining components as callerId using component specific API, log callerId of Pig and store it on ATS. More specific, it includes:
      1. Generate a CallerId for each Pig script, pass it to Hdfs/Yarn/MapReduce/Tez which Pig invokes
      2. Pig caller passes "pig.log.trace.id" to Pig, Pig will publish it to ATS

      Attachments

        1. PIG-4714-1.patch
          14 kB
          Daniel Dai
        2. PIG-4714-2.patch
          15 kB
          Daniel Dai
        3. PIG-4714-3.patch
          15 kB
          Daniel Dai
        4. PIG-4714-4.patch
          15 kB
          Daniel Dai
        5. PIG-4714-5.patch
          15 kB
          Daniel Dai

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            daijy Daniel Dai
            daijy Daniel Dai
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment