Pig
  1. Pig
  2. PIG-2916

Add script file name for logging with log4j MDC

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.11
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      We often execute several scripts in parallel and log the messages into one log file.
      If got error , we can not identify which pig script is causing it. So add a MDC with name "pig_script_file_name" and add %X

      {pig_script_file_name}

      into log4j configurations, we will surely identify the errorous pig script.

      the patch is attached.

      1. mdc.patch
        1 kB
        Haitao Yao

        Activity

        Hide
        Haitao Yao added a comment -

        But i think the MDC is better, because we may log more information into the log. MDC will filter the specific value you want.

        Show
        Haitao Yao added a comment - But i think the MDC is better, because we may log more information into the log. MDC will filter the specific value you want.
        Haitao Yao made changes -
        Patch Info Patch Available [ 10042 ]
        Hide
        Rohini Palaniswamy added a comment -

        MDC requires log4j configuration being configured with the exact key and if there are too many it becomes difficult to maintain. If NDC was used, it would be more easy to add more context information without having to go re-configure the log4j configuration every time. Also it would be better to log "file" instead of localFileRet.file.getName(). If the script was actually in s3 or hdfs, logging "file" would give the full url of the original file.

        Show
        Rohini Palaniswamy added a comment - MDC requires log4j configuration being configured with the exact key and if there are too many it becomes difficult to maintain. If NDC was used, it would be more easy to add more context information without having to go re-configure the log4j configuration every time. Also it would be better to log "file" instead of localFileRet.file.getName(). If the script was actually in s3 or hdfs, logging "file" would give the full url of the original file.
        Haitao Yao made changes -
        Field Original Value New Value
        Attachment mdc.patch [ 12544758 ]
        Hide
        Haitao Yao added a comment -

        just add MDC when file mode is enable.

        Show
        Haitao Yao added a comment - just add MDC when file mode is enable.
        Haitao Yao created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Haitao Yao
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development