Pig
  1. Pig
  2. PIG-3035

With latest version of hadoop23 pig does not return the correct exception stack trace from backend

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11, 0.10.1, 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      With latest version of hadoop, the job failure shown to user is always Container killed by the ApplicationMaster. For eg:

      "Backend error : Unable to recreate exception from backed error: AttemptID:attempt_1352163563357_0002_m_000000_1 Info:Container killed by the ApplicationMaster"

      Steps to Reproduce:
      Change hadoop version from 2.0.0-alpha to 0.23.5-SNAPSHOT in ivy/libraries.properties. Tests TestScalarAliases.testScalarErrMultipleRowsInInput and TestEvalPipeline2.testNonStandardData will fail when run with -Dhadoopversion=23.

      1. PIG-3035.patch
        0.9 kB
        Rohini Palaniswamy

        Activity

        Hide
        Rohini Palaniswamy added a comment -

        In Launcher.java getErrorMessages(), the TaskReport reports[] gives the first element as the actual stacktrace and the second one as "Container killed by the ApplicationMaster". The parsing logic to construct StackTrace out of the error message has an issue, so it fails to construct the Exception and the message from the last TaskReport which is Container killed is thrown as error. Fixing the logic which constructs the Exception fixes the issue.

        if(items.length > 0) {
                    fileName = items[0];
                    lineNumber = Integer.parseInt(items[1]);
                }
        

        items.length is 1 in case of "at java.security.AccessController.doPrivileged(Native Method)" in the stack trace which does not have a line number.

        Show
        Rohini Palaniswamy added a comment - In Launcher.java getErrorMessages(), the TaskReport reports[] gives the first element as the actual stacktrace and the second one as "Container killed by the ApplicationMaster". The parsing logic to construct StackTrace out of the error message has an issue, so it fails to construct the Exception and the message from the last TaskReport which is Container killed is thrown as error. Fixing the logic which constructs the Exception fixes the issue. if (items.length > 0) { fileName = items[0]; lineNumber = Integer .parseInt(items[1]); } items.length is 1 in case of "at java.security.AccessController.doPrivileged(Native Method)" in the stack trace which does not have a line number.
        Hide
        Rohini Palaniswamy added a comment -

        The attached patch fixes the ArrayIndexOutOfBoundsException for elements in stacktrace that do not have a line number by setting it to -1.

        According to javadoc,
        lineNumber - the line number of the source line containing the execution point represented by this stack trace element, or a negative number if this information is unavailable. A value of -2 indicates that the method containing the execution point is a native method

        Show
        Rohini Palaniswamy added a comment - The attached patch fixes the ArrayIndexOutOfBoundsException for elements in stacktrace that do not have a line number by setting it to -1. According to javadoc, lineNumber - the line number of the source line containing the execution point represented by this stack trace element, or a negative number if this information is unavailable. A value of -2 indicates that the method containing the execution point is a native method
        Hide
        Daniel Dai added a comment -

        +1

        Show
        Daniel Dai added a comment - +1
        Hide
        Rohini Palaniswamy added a comment -

        Thanks Daniel. Committed to 0.10.1, 0.11 and trunk

        Show
        Rohini Palaniswamy added a comment - Thanks Daniel. Committed to 0.10.1, 0.11 and trunk

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Rohini Palaniswamy
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development