Hadoop Common
  1. Hadoop Common
  2. HADOOP-1158

JobTracker should collect statistics of failed map output fetches, and take decisions to reexecute map tasks and/or restart the (possibly faulty) Jetty server on the TaskTracker

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.2
    • Fix Version/s: 0.15.0
    • Component/s: None
    • Labels:
      None

      Description

      The JobTracker should keep a track (with feedback from Reducers) of how many times a fetch for a particular map output failed. If this exceeds a certain threshold, then that map should be declared as lost, and should be reexecuted elsewhere. Based on the number of such complaints from Reducers, the JobTracker can blacklist the TaskTracker. This will make the framework reliable - it will take care of (faulty) TaskTrackers that sometimes always fail to serve up map outputs (for which exceptions are not properly raised/handled, for e.g., if the exception/problem happens in the Jetty server).

      1. HADOOP-1158_20070702_1.patch
        48 kB
        Arun C Murthy
      2. HADOOP-1158_2_20070808.patch
        50 kB
        Arun C Murthy
      3. HADOOP-1158_3_20070809.patch
        51 kB
        Arun C Murthy
      4. HADOOP-1158_4_20070817.patch
        51 kB
        Arun C Murthy
      5. HADOOP-1158_5_20070823.patch
        51 kB
        Arun C Murthy

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          I'd propose a minor variant where the reduce/fetcher tries a few (2? 3?) times before complaining to the JobTracker to cut down on the noise.

          I think that restarting Jetty wouldn't be that useful and would potentially cause more trouble.

          Show
          Owen O'Malley added a comment - I'd propose a minor variant where the reduce/fetcher tries a few (2? 3?) times before complaining to the JobTracker to cut down on the noise. I think that restarting Jetty wouldn't be that useful and would potentially cause more trouble.
          Hide
          Devaraj Das added a comment -

          Yes, the reduce/fetcher should try a few times (it could be configurable?) before complaining to the JobTracker. The JobTracker can take a decision on whether to reexecute a Map based on the % of complaints (>50% ?) from fetching reduces. For example, if there are 10 reduces currently fetching, and if at least 5 of them complained about a fetch failing for a particular Map, then the JobTracker should reexecute that Map. Makes sense?

          Show
          Devaraj Das added a comment - Yes, the reduce/fetcher should try a few times (it could be configurable?) before complaining to the JobTracker. The JobTracker can take a decision on whether to reexecute a Map based on the % of complaints (>50% ?) from fetching reduces. For example, if there are 10 reduces currently fetching, and if at least 5 of them complained about a fetch failing for a particular Map, then the JobTracker should reexecute that Map. Makes sense?
          Hide
          Devaraj Das added a comment -

          Linking this issue with HADOOP-1454 to record that someone requested for this feature explicitly.

          Show
          Devaraj Das added a comment - Linking this issue with HADOOP-1454 to record that someone requested for this feature explicitly.
          Hide
          Arun C Murthy added a comment -

          Some early thoughts...

          Bottomline: we don't want the reducer and hence the job to get stuck forever.

          The main issue is that when a reducer is stuck in shuffle it's hard to accurately say whether the fault lies at the map (jetty acting weird) or at the reduce or both. Having said that it's pertinent to keep in mind that normally maps are cheaper to re-execute.

          Given the above I'd like to propose something along these lines:

          a) The reduce maintains a per-map count of fetch failures.

          b) Given sufficient fetch-failures per-map (say 3 or 4), the reducer then complains to the JobTracker via a new rpc:

          JobTracker.java
          public synchronized void notifyFailedFetch(String reduceTaskId, String mapTaskId) {
            // ...
          }
          

          c) The JobTracker maintains a per-map count of failed-fetch notfications, and given a sufficient no. of them (say 2/3?) from any reducer (even multiple times from the same reducer) fails the map and re-schedules it elsewhere.

          This handles 2 cases: a) Faulty maps are re-executed and b) Corner case where only the last reducer is stuck on a given map and hence the map will have to be re-executed.

          d) To counter the case of faulty reduces, we could implement a scheme where the reducer kills itself when it notifies the JobTracker of more than, say 5 unique, faulty fetches. This will ensure that a faulty reducer will not result in the JobTracker spawning maps willy-nilly...

          Thoughts?

          Show
          Arun C Murthy added a comment - Some early thoughts... Bottomline: we don't want the reducer and hence the job to get stuck forever. The main issue is that when a reducer is stuck in shuffle it's hard to accurately say whether the fault lies at the map (jetty acting weird) or at the reduce or both. Having said that it's pertinent to keep in mind that normally maps are cheaper to re-execute. Given the above I'd like to propose something along these lines: a) The reduce maintains a per-map count of fetch failures. b) Given sufficient fetch-failures per-map (say 3 or 4), the reducer then complains to the JobTracker via a new rpc: JobTracker.java public synchronized void notifyFailedFetch( String reduceTaskId, String mapTaskId) { // ... } c) The JobTracker maintains a per-map count of failed-fetch notfications, and given a sufficient no. of them (say 2/3?) from any reducer (even multiple times from the same reducer) fails the map and re-schedules it elsewhere. This handles 2 cases: a) Faulty maps are re-executed and b) Corner case where only the last reducer is stuck on a given map and hence the map will have to be re-executed. d) To counter the case of faulty reduces, we could implement a scheme where the reducer kills itself when it notifies the JobTracker of more than, say 5 unique, faulty fetches. This will ensure that a faulty reducer will not result in the JobTracker spawning maps willy-nilly... Thoughts?
          Hide
          Doug Cutting added a comment -

          This sounds like a good design to me.

          Show
          Doug Cutting added a comment - This sounds like a good design to me.
          Hide
          Owen O'Malley added a comment -

          It looks good to me too.

          Show
          Owen O'Malley added a comment - It looks good to me too.
          Hide
          Devaraj Das added a comment -

          Given the fact that in general losing reduces is detrimental, I'd propose a minor variant to the logic behind killing reduces. The reduce should kill itself when it fails to fetch the map output from even the new location, i.e., the unique 5 faulty fetches should have at least 1 retrial (i.e., we don't kill a reduce too early).

          Also, does it make sense to have the logic behind killing/reexecuting reduces in the JobTracker. Two reasons:
          1) since the JobTracker knows very well how many times a reduce complained, and, for which maps it complained, etc.,
          2) consistent behavior - jobtracker handles the reexecution of maps and it might handle the reexecution of reduces as well.

          Show
          Devaraj Das added a comment - Given the fact that in general losing reduces is detrimental, I'd propose a minor variant to the logic behind killing reduces. The reduce should kill itself when it fails to fetch the map output from even the new location, i.e., the unique 5 faulty fetches should have at least 1 retrial (i.e., we don't kill a reduce too early). Also, does it make sense to have the logic behind killing/reexecuting reduces in the JobTracker. Two reasons: 1) since the JobTracker knows very well how many times a reduce complained, and, for which maps it complained, etc., 2) consistent behavior - jobtracker handles the reexecution of maps and it might handle the reexecution of reduces as well.
          Hide
          Arun C Murthy added a comment -

          The reduce should kill itself when it fails to fetch the map output from even the new location, i.e., the unique 5 faulty fetches should have at least 1 retrial (i.e., we don't kill a reduce too early).

          Though it makes sense in the long-term I'd vote we keep it simple for now... to implement this would entail more complex code and more state to be maintained. 5 notifications anyway mean that the reducer has seen 20 attempts to fetch on 5 different maps fail. I'd say, for now, it's a sufficient reason to kill the reducer.

          Also, does it make sense to have the logic behind killing/reexecuting reduces in the JobTracker. Two reasons:

          1) since the JobTracker knows very well how many times a reduce complained, and, for which maps it complained, etc.,

          If the reducer kills itself, the JobTracker need not maintain information of which reduces failed to fetch which maps, it could just do with a per-taskid count of failed fetches (for the maps, as notified by reducers) - again leads to simpler code for a first-shot.

          2) consistent behavior - jobtracker handles the reexecution of maps and it might handle the reexecution of reduces as well.

          I agree with the general sentiment, but given that this leads to more complex code and the reducer already knows it has failed to fetch from 5 different maps it doesn't make sense for it to wait for the JobTracker to fail the task. Also, there is an existing precedent for this behaviour in TaskTracker.fsError (task is marked as 'failed' by the TaskTracker itself on an FSError).

          Thoughts?

          Show
          Arun C Murthy added a comment - The reduce should kill itself when it fails to fetch the map output from even the new location, i.e., the unique 5 faulty fetches should have at least 1 retrial (i.e., we don't kill a reduce too early). Though it makes sense in the long-term I'd vote we keep it simple for now... to implement this would entail more complex code and more state to be maintained. 5 notifications anyway mean that the reducer has seen 20 attempts to fetch on 5 different maps fail. I'd say, for now, it's a sufficient reason to kill the reducer. Also, does it make sense to have the logic behind killing/reexecuting reduces in the JobTracker. Two reasons: 1) since the JobTracker knows very well how many times a reduce complained, and, for which maps it complained, etc., If the reducer kills itself, the JobTracker need not maintain information of which reduces failed to fetch which maps, it could just do with a per-taskid count of failed fetches (for the maps, as notified by reducers) - again leads to simpler code for a first-shot. 2) consistent behavior - jobtracker handles the reexecution of maps and it might handle the reexecution of reduces as well. I agree with the general sentiment, but given that this leads to more complex code and the reducer already knows it has failed to fetch from 5 different maps it doesn't make sense for it to wait for the JobTracker to fail the task. Also, there is an existing precedent for this behaviour in TaskTracker.fsError (task is marked as 'failed' by the TaskTracker itself on an FSError). Thoughts?
          Hide
          Arun C Murthy added a comment - - edited

          b) Given sufficient fetch-failures per-map (say 3 or 4), the reducer then complains to the JobTracker via a new rpc:

          I take that back.. I propose we augument TaskStatus itself to let the JobTracker know about the failed-fetches i.e. map taskids.

          We could just add an new RPC to TaskUmbilicalProtocol for the reduce-task to let the TaskTracker know about the failed fetch...

          TaskUmbilical.java
          void fetchError(String taskId, String failedFetchMapTaskId);
          

          Even better, a tad more involved, is to rework

          TaskUmbilical.java
            void progress(String taskid, float progress, String state, 
                                      TaskStatus.Phase phase, Counters counters)
             throws IOException, InterruptedException;
          

          as

          TaskUmbilical.java
            void progress(String taskid, TaskStatus taskStatus}
             throws IOException, InterruptedException;
          

          This simplies the flow so that the child-vm itself computes it's TaskStatus (which will be augumented to contain the failed-fetch-mapIds) and sends it along the TaskTracker which just forwards it to the JobTracker, thereby relieving it of some of the responsibilities vis-a-vis computing the TaskStatus. Clearly this could be linked to the the reporting re-design at HADOOP-1462 ...

          Thoughts?

          Show
          Arun C Murthy added a comment - - edited b) Given sufficient fetch-failures per-map (say 3 or 4), the reducer then complains to the JobTracker via a new rpc: I take that back.. I propose we augument TaskStatus itself to let the JobTracker know about the failed-fetches i.e. map taskids. We could just add an new RPC to TaskUmbilicalProtocol for the reduce-task to let the TaskTracker know about the failed fetch... TaskUmbilical.java void fetchError( String taskId, String failedFetchMapTaskId); Even better, a tad more involved, is to rework TaskUmbilical.java void progress( String taskid, float progress, String state, TaskStatus.Phase phase, Counters counters) throws IOException, InterruptedException; as TaskUmbilical.java void progress( String taskid, TaskStatus taskStatus} throws IOException, InterruptedException; This simplies the flow so that the child-vm itself computes it's TaskStatus (which will be augumented to contain the failed-fetch-mapIds) and sends it along the TaskTracker which just forwards it to the JobTracker , thereby relieving it of some of the responsibilities vis-a-vis computing the TaskStatus . Clearly this could be linked to the the reporting re-design at HADOOP-1462 ... Thoughts?
          Hide
          Arun C Murthy added a comment -

          Oh, it might be worth considering a separate MapTaskStatus and ReduceTaskStatus classes since there are varied pieces of un-related information for Map and Reduce tasks (i.e. shuffle/sort-merge related info, fetch failures etc.) ... we could stick it up in the appropriate 'Task' class too (which the child-vm could then compute and send to the TaskTracker) - perhaps as a separate issue?

          Show
          Arun C Murthy added a comment - Oh, it might be worth considering a separate MapTaskStatus and ReduceTaskStatus classes since there are varied pieces of un-related information for Map and Reduce tasks (i.e. shuffle/sort-merge related info, fetch failures etc.) ... we could stick it up in the appropriate 'Task' class too (which the child-vm could then compute and send to the TaskTracker) - perhaps as a separate issue?
          Hide
          Doug Cutting added a comment -

          Putting all status into a single object sounds like a good approach. Perhaps the method should be renamed 'status' rather than 'progress'?

          Show
          Doug Cutting added a comment - Putting all status into a single object sounds like a good approach. Perhaps the method should be renamed 'status' rather than 'progress'?
          Hide
          Devaraj Das added a comment -

          Arun, I think this issue is kind of a slightly longer term solution a problem and we have time enough with us to work towards that. I'd still argue that the best place to have the logic behind killing the reduces is in the JobTracker. Exceptions like the disk exception, ping exception, are very local cases where a task decides to kill itself, but in this issue there is a certain element of globalness involved (like dependency on maps), and the JobTracker is the only guy who has a global picture of jobs. I don't see how we will lose simpilicity by having the logic in the JobTracker. I understand that it will have to maintain a few bytes more per task, but that's not unreasonable.

          Show
          Devaraj Das added a comment - Arun, I think this issue is kind of a slightly longer term solution a problem and we have time enough with us to work towards that. I'd still argue that the best place to have the logic behind killing the reduces is in the JobTracker. Exceptions like the disk exception, ping exception, are very local cases where a task decides to kill itself, but in this issue there is a certain element of globalness involved (like dependency on maps), and the JobTracker is the only guy who has a global picture of jobs. I don't see how we will lose simpilicity by having the logic in the JobTracker. I understand that it will have to maintain a few bytes more per task, but that's not unreasonable.
          Hide
          Arun C Murthy added a comment -

          Early patch while I continue testing it further...

          Show
          Arun C Murthy added a comment - Early patch while I continue testing it further...
          Hide
          Devaraj Das added a comment -

          One comment - for cases where a reduce fails to fetch a map output for a number of times, it informs the jobtracker, and the jobtracker reexecutes the map. The reduce task also increments a counter associated with the number of failed fetches (which in the patch is used to kill itself when it exceeds a certain number). Now if the reduce can fetch the map output correctly from the new map location, it should decrement that counter.

          Show
          Devaraj Das added a comment - One comment - for cases where a reduce fails to fetch a map output for a number of times, it informs the jobtracker, and the jobtracker reexecutes the map. The reduce task also increments a counter associated with the number of failed fetches (which in the patch is used to kill itself when it exceeds a certain number). Now if the reduce can fetch the map output correctly from the new map location, it should decrement that counter.
          Hide
          Arun C Murthy added a comment -

          Finally a new patch incorporating Devaraj's comments and well-tested too...

          Show
          Arun C Murthy added a comment - Finally a new patch incorporating Devaraj's comments and well-tested too...
          Hide
          Hadoop QA added a comment -
          Show
          Hadoop QA added a comment - +0, new Findbugs warnings http://issues.apache.org/jira/secure/attachment/12363431/HADOOP-1158_2_20070808.patch applied and successfully tested against trunk revision r563649, but there appear to be new Findbugs warnings introduced by this patch. New Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/530/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/530/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/530/console
          Hide
          Arun C Murthy added a comment -

          Need to fix the findbugs warnings...

          Show
          Arun C Murthy added a comment - Need to fix the findbugs warnings...
          Hide
          Arun C Murthy added a comment -

          Patch which fixes findbugs' warnings...

          Show
          Arun C Murthy added a comment - Patch which fixes findbugs' warnings...
          Hide
          Hadoop QA added a comment -

          -1, build or testing failed

          2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12363458/HADOOP-1158_3_20070809.patch against trunk revision r564012.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/535/testReport/
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/535/console

          Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

          Show
          Hadoop QA added a comment - -1, build or testing failed 2 attempts failed to build and test the latest attachment http://issues.apache.org/jira/secure/attachment/12363458/HADOOP-1158_3_20070809.patch against trunk revision r564012. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/535/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/535/console Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
          Hide
          Arun C Murthy added a comment -

          TestDFSUpgrade failed with:

          junit.framework.AssertionFailedError: expected:<1790222743> but was:<3731403302>
          	at org.apache.hadoop.dfs.TestDFSUpgrade.checkResult(TestDFSUpgrade.java:74)
          	at org.apache.hadoop.dfs.TestDFSUpgrade.testUpgrade(TestDFSUpgrade.java:142
          

          as noted by Nicholas here: http://issues.apache.org/jira/browse/HADOOP-1696#action_12518584

          I'm not sure why this failed... passes on my local-box.

          Show
          Arun C Murthy added a comment - TestDFSUpgrade failed with: junit.framework.AssertionFailedError: expected:<1790222743> but was:<3731403302> at org.apache.hadoop.dfs.TestDFSUpgrade.checkResult(TestDFSUpgrade.java:74) at org.apache.hadoop.dfs.TestDFSUpgrade.testUpgrade(TestDFSUpgrade.java:142 as noted by Nicholas here: http://issues.apache.org/jira/browse/HADOOP-1696#action_12518584 I'm not sure why this failed... passes on my local-box.
          Hide
          Doug Cutting added a comment -

          This generates a new compiler warning for me:

              [javac] /home/cutting/src/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTaskStatus.java:48: warning: [unchecked] unchecked cast
              [javac] found   : java.lang.Object
              [javac] required: java.util.List<java.lang.String>
              [javac]       (List<String>)(((ArrayList<String>)failedFetchTasks).clone());
              [javac]                     ^
          
          Show
          Doug Cutting added a comment - This generates a new compiler warning for me: [javac] /home/cutting/src/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTaskStatus.java:48: warning: [unchecked] unchecked cast [javac] found : java.lang.Object [javac] required: java.util.List<java.lang.String> [javac] (List<String>)(((ArrayList<String>)failedFetchTasks).clone()); [javac] ^
          Hide
          Arun C Murthy added a comment -

          Fixed the warning and updated to reflect changes to trunk...

          Show
          Arun C Murthy added a comment - Fixed the warning and updated to reflect changes to trunk...
          Hide
          Hadoop QA added a comment -

          -1, new javadoc warnings

          The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12363993/HADOOP-1158_4_20070817.patch against trunk revision r567308.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/573/testReport/
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/573/console

          Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

          Show
          Hadoop QA added a comment - -1, new javadoc warnings The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12363993/HADOOP-1158_4_20070817.patch against trunk revision r567308. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/573/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/573/console Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
          Hide
          Arun C Murthy added a comment -

          Looks like the javadoc warnings were caused by removal of the ant jars and fixed by HADOOP-1726.

          I'll check if this patch needs to be updated to reflect changes to trunk and submit a new one...

          Show
          Arun C Murthy added a comment - Looks like the javadoc warnings were caused by removal of the ant jars and fixed by HADOOP-1726 . I'll check if this patch needs to be updated to reflect changes to trunk and submit a new one...
          Hide
          Arun C Murthy added a comment -

          Re-submitting to check if the warnings were indeed fixed by HADOOP-1726 .

          Show
          Arun C Murthy added a comment - Re-submitting to check if the warnings were indeed fixed by HADOOP-1726 .
          Hide
          Hadoop QA added a comment -

          -1, new javadoc warnings

          The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12363993/HADOOP-1158_4_20070817.patch against trunk revision r568706.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/596/testReport/
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/596/console

          Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

          Show
          Hadoop QA added a comment - -1, new javadoc warnings The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12363993/HADOOP-1158_4_20070817.patch against trunk revision r568706. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/596/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/596/console Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
          Hide
          Arun C Murthy added a comment -

          Fix javadoc warnings...

          Show
          Arun C Murthy added a comment - Fix javadoc warnings...
          Show
          Hadoop QA added a comment - +1 http://issues.apache.org/jira/secure/attachment/12364381/HADOOP-1158_5_20070823.patch applied and successfully tested against trunk revision r568809. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/602/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/602/console
          Hide
          Enis Soztutar added a comment -

          The patch looks good, but i would like to mention another major issue here.
          There are some cases when TaskTracker send heartbeats, but the jetty server cannot serve the outputs. Recently we have seen the jetty servers failing to allocate new threads from the thread pool on some of the tasktrackers, emiting logs:

            2007-08-23 09:31:46,378 INFO org.mortbay.http.SocketListener: LOW ON THREADS ((40-40+0)<1) on SocketListener0@0.0.0.0:50060
            2007-08-23 09:31:46,379 WARN org.mortbay.http.SocketListener: OUT OF THREADS: SocketListener0@0.0.0.0:50060
          

          Moreover, HADOOP-1179 mentions OOM exceptions related to Jetty. We will try to find and eliminate the sources of jetty related leaks and bugs, but it is not likely that all of them will be resolved. There will be cases such as above that RPC responds but http may not, so taking a "computer engineering approach" by solving the problem by restarting seems appropriate.

          long story short, i think it would be great to do some bookkeeping in JT about failed fetches per TT and send reinit action to TT above some threshold.

          Show
          Enis Soztutar added a comment - The patch looks good, but i would like to mention another major issue here. There are some cases when TaskTracker send heartbeats, but the jetty server cannot serve the outputs. Recently we have seen the jetty servers failing to allocate new threads from the thread pool on some of the tasktrackers, emiting logs: 2007-08-23 09:31:46,378 INFO org.mortbay.http.SocketListener: LOW ON THREADS ((40-40+0)<1) on SocketListener0@0.0.0.0:50060 2007-08-23 09:31:46,379 WARN org.mortbay.http.SocketListener: OUT OF THREADS: SocketListener0@0.0.0.0:50060 Moreover, HADOOP-1179 mentions OOM exceptions related to Jetty. We will try to find and eliminate the sources of jetty related leaks and bugs, but it is not likely that all of them will be resolved. There will be cases such as above that RPC responds but http may not, so taking a "computer engineering approach" by solving the problem by restarting seems appropriate. long story short, i think it would be great to do some bookkeeping in JT about failed fetches per TT and send reinit action to TT above some threshold.
          Hide
          Enis Soztutar added a comment -

          Here is the code in jetty to print the above warning.

           log.info("LOW ON THREADS (("+getMaxThreads()+"-"+getThreads()+"+"+getIdleThreads()+")<"+getMinThreads()+") on "+ this); 
          

          it seems jetty is configured with max threads = 40, isn't it insufficient?

          Show
          Enis Soztutar added a comment - Here is the code in jetty to print the above warning. log.info( "LOW ON THREADS ((" +getMaxThreads()+ "-" +getThreads()+ "+" +getIdleThreads()+ ")<" +getMinThreads()+ ") on " + this ); it seems jetty is configured with max threads = 40, isn't it insufficient?
          Hide
          Doug Cutting added a comment -

          I just committed this. Thanks, Arun!

          Show
          Doug Cutting added a comment - I just committed this. Thanks, Arun!
          Hide
          Arun C Murthy added a comment -

          Thanks for the review Enis.

          So, here is how we solve issues emanating from Jetty: if there are sufficient failures for a given map (say due to Jetty), we just fail the map and re-run it elsewhere, there-by the reducer isn't stuck. Now given sufficient no. of maps fail on the same TaskTracker (say Jetty again) then it gets blacklisted and hence no tasks are assigned to it... does that make sense?

          Please feel free to open further issues if you have other thoughts help improve things...

          Show
          Arun C Murthy added a comment - Thanks for the review Enis. So, here is how we solve issues emanating from Jetty: if there are sufficient failures for a given map (say due to Jetty), we just fail the map and re-run it elsewhere, there-by the reducer isn't stuck. Now given sufficient no. of maps fail on the same TaskTracker (say Jetty again) then it gets blacklisted and hence no tasks are assigned to it... does that make sense? Please feel free to open further issues if you have other thoughts help improve things...
          Hide
          Enis Soztutar added a comment -

          Yes the TT will be blacklisted and it is sometimes unlikely that the TT will continue its normal computation. But restarting the tt automatically has a chance to recover its state. I think it will improve self managing aspects of the cluster.

          Show
          Enis Soztutar added a comment - Yes the TT will be blacklisted and it is sometimes unlikely that the TT will continue its normal computation. But restarting the tt automatically has a chance to recover its state. I think it will improve self managing aspects of the cluster.

            People

            • Assignee:
              Arun C Murthy
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development