Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0
    • Fix Version/s: 0.20.204.0
    • Component/s: tasktracker
    • Labels:
      None

      Description

      Observing a lot of jvmValidate exceptions in TT logs for grid mix run

      ************************
      2011-04-28 02:00:37,578 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 46121, call
      statusUpdate(attempt_201104270735_5993_m_003305_0, org.apache.hadoop.mapred.MapTaskStatus@1840a9c,
      org.apache.hadoop.mapred.JvmContext@1d4ab6b) from 127.0.0.1:50864: error: java.io.IOException: JvmValidate Failed.
      Ignoring request from task: attempt_201104270735_5993_m_003305_0, with JvmId:
      jvm_201104270735_5993_m_103399012gsbl20430: java.io.IOException: JvmValidate Failed. Ignoring request from task:
      attempt_201104270735_5993_m_003305_0, with JvmId: jvm_201104270735_5993_m_103399012gsbl20430: –
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1386)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1384)

      *********************

      1. MAPREDUCE-2555-20security-v2.patch
        1 kB
        Thomas Graves
      2. MAPREDUCE-2555-20security.patch
        1 kB
        Thomas Graves

        Issue Links

          Activity

          Hide
          Thomas Graves added a comment -

          The exceptions are generated by status updates sent by the child after it reports 'done' to the TaskTracker. Ideally
          this should not happen.

          The fix for MAPREDUCE-2429 is meant to ignore such calls (since the child jvm is effectively not supposed to exist after the done call is processed).

          In previous versions - the statusUpdate RPC would return success but would ignore the request.
          This exception is effectively replacing another message from previous version (Ignoring status-update since task is
          'done'). With the current code - the statusUpdate RPC fails - which causes the Child to retry - effectively generating way more messages in the logs.

          Show
          Thomas Graves added a comment - The exceptions are generated by status updates sent by the child after it reports 'done' to the TaskTracker. Ideally this should not happen. The fix for MAPREDUCE-2429 is meant to ignore such calls (since the child jvm is effectively not supposed to exist after the done call is processed). In previous versions - the statusUpdate RPC would return success but would ignore the request. This exception is effectively replacing another message from previous version (Ignoring status-update since task is 'done'). With the current code - the statusUpdate RPC fails - which causes the Child to retry - effectively generating way more messages in the logs.
          Hide
          Thomas Graves added a comment -

          I think all we want to do is get the child to exit and not generate all the log messages. To do that we can have the TaskTracker::statusUpdate return false if the validateJVM throws. The only catch to that is the TaskTracker::commitPending call doesn't return boolean so will have to have it still throw.

          Show
          Thomas Graves added a comment - I think all we want to do is get the child to exit and not generate all the log messages. To do that we can have the TaskTracker::statusUpdate return false if the validateJVM throws. The only catch to that is the TaskTracker::commitPending call doesn't return boolean so will have to have it still throw.
          Hide
          Thomas Graves added a comment -

          This is the patch for the branch-0.20-security.

          Show
          Thomas Graves added a comment - This is the patch for the branch-0.20-security.
          Hide
          Thomas Graves added a comment -

          Only adding patch for branch-0.20-security since MAPREDUCE-2429 not in trunk yet.

          Show
          Thomas Graves added a comment - Only adding patch for branch-0.20-security since MAPREDUCE-2429 not in trunk yet.
          Hide
          Todd Lipcon added a comment -

          Can you please add a comment to MAPREDUCE-2429 as well so that we don't miss this in trunk?

          Is Sidd still working on the forward-port?

          Show
          Todd Lipcon added a comment - Can you please add a comment to MAPREDUCE-2429 as well so that we don't miss this in trunk? Is Sidd still working on the forward-port?
          Hide
          Thomas Graves added a comment -

          fix a couple spacing issues.

          Show
          Thomas Graves added a comment - fix a couple spacing issues.
          Hide
          Thomas Graves added a comment -

          Tested manually via changing code as this is hard to reproduce.

          Show
          Thomas Graves added a comment - Tested manually via changing code as this is hard to reproduce.
          Hide
          Thomas Graves added a comment -

          Added comment to MAPREDUCE-2429. I'll ask Sidd about that forward-port.

          Show
          Thomas Graves added a comment - Added comment to MAPREDUCE-2429 . I'll ask Sidd about that forward-port.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481267/MAPREDUCE-2555-20security-v2.patch
          against trunk revision 1130554.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/336//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481267/MAPREDUCE-2555-20security-v2.patch against trunk revision 1130554. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/336//console This message is automatically generated.
          Hide
          Thomas Graves added a comment -

          Note this failed since patch is for branch-0.20-security.

          Show
          Thomas Graves added a comment - Note this failed since patch is for branch-0.20-security.
          Hide
          Siddharth Seth added a comment -

          +1 lgtm to get rid of the extra logging and to kill the child process.

          Show
          Siddharth Seth added a comment - +1 lgtm to get rid of the extra logging and to kill the child process.
          Hide
          Siddharth Seth added a comment -

          @Todd, I am still working on the 2429 forward port.

          Show
          Siddharth Seth added a comment - @Todd, I am still working on the 2429 forward port.
          Hide
          Chris Douglas added a comment -

          Noting dependency in trunk

          Show
          Chris Douglas added a comment - Noting dependency in trunk
          Hide
          Chris Douglas added a comment -

          I committed this to the 20-security branch. I'll leave it open to track the forward-port

          Show
          Chris Douglas added a comment - I committed this to the 20-security branch. I'll leave it open to track the forward-port
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was just released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.

            People

            • Assignee:
              Thomas Graves
              Reporter:
              Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development