Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0, 0.23.0
    • Fix Version/s: 0.20.204.0, 0.23.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Added 2 new config parameters:

      mapreduce.reduce.shuffle.catch.exception.stack.regex
      mapreduce.reduce.shuffle.catch.exception.message.regex
      Show
      Added 2 new config parameters: mapreduce.reduce.shuffle.catch.exception.stack.regex mapreduce.reduce.shuffle.catch.exception.message.regex

      Description

      We are seeing many instances of the Jetty-1342 (http://jira.codehaus.org/browse/JETTY-1342). The bug doesn't cause Jetty to stop responding altogether, some fetches go through but a lot of them throw exceptions and eventually fail. The only way we have found to get the TT out of this state is to restart the TT. This jira is to catch this particular exception (or perhaps a configurable regex) and handle it in an automated way to either blacklist or shutdown the TT after seeing it a configurable number of them.

      1. jetty1342-20security.patch
        16 kB
        Thomas Graves
      2. mapred2529-trunk.patch
        18 kB
        Thomas Graves
      3. M2529-1-20s.patch
        14 kB
        Chris Douglas
      4. M2529-1.patch
        16 kB
        Chris Douglas

        Activity

        Thomas Graves created issue -
        Thomas Graves made changes -
        Field Original Value New Value
        Attachment jetty1342-20security.patch [ 12480605 ]
        Thomas Graves made changes -
        Attachment mapred2529-trunk.patch [ 12480969 ]
        Thomas Graves made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note Added 2 new config parameters:

        mapreduce.reduce.shuffle.catch.exception.stack.regex
        mapreduce.reduce.shuffle.catch.exception.message.regex
        Fix Version/s 0.20.205.0 [ 12316391 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Thomas Graves made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Thomas Graves made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Chris Douglas made changes -
        Attachment M2529-1-20s.patch [ 12481432 ]
        Attachment M2529-1.patch [ 12481433 ]
        Chris Douglas made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Owen O'Malley made changes -
        Fix Version/s 0.20.204.0 [ 12316318 ]
        Owen O'Malley made changes -
        Fix Version/s 0.20.205.0 [ 12316391 ]
        Owen O'Malley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Thomas Graves
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development